Statictea Dependencies Graphs

Statictea has two types of dependency graphs, the module dependencies and the nim system module dependencies. You can learn a lot about a project from its dependencies.

Below is the link to the statictea’s source code documentation. The module dependency graph is at the top. If you scroll down to the bottom you see the nim system module dependency graph. If you are viewing on a phone, you can pinch zoom to see more detail.

statictea’s source code docs

Module Dependencies

A graph node represents a nim source code module (file). The lines connecting the nodes show the dependencies. For example the parseCommandLine module on the left depends on the cmdline module.

From the module dependency graph you can see:

  • which modules you can share with other projects
  • the frequently used modules
  • the relative module file sizes
  • the project entry point
  • the stand alone modules

It’s easier to edit and test modular code because of well defined and simple interfaces.

The green nodes do not depend on any module. These are the modules you can easily share with other projects.

The red dependency line tells you that the module has only one dependency. I try to rework the code to remove the dependency if feasible. The opresult and matches modules are example of this.

If you put all your code in one module, it wouldn’t have any dependencies! However it would be to bloated and not granular enough for sharing with other projects. You can tell the relative size of a module by the size of its node. runCommand is the biggest module.

The comparelines, jsondocraw and stfrunner modules are stand alone commands so they are not connected in this graph.

Nim Module Graph

The nim module graph at the bottom of the page tells you the nim language APIs you need. It shows which nim modules are used by which statictea modules.

The green nim modules are only used by one statictea module. I strive for green modules, that way all related functions are grouped together.

Often I wrap a nim module. The statictea wrapper module exposes a limited and simple interface tailored for statictea needs. This was done for the re module. The graph tells you that statictea only uses the regular expression functions in the regexes module and not the multitude of functions in the re module. The readjson module is another example of this.

Building Graphs

You build the dependency graph with the nimble task dotsrc and the nim module graph with dotsys. Search for createDependencyGraph and createDependencyGraph2 in the nimble file.

Each graph is an svg file. The graphviz program produces the file based on a text file defining the dependencies. The text file is created by the nimble task starting from the data generated by nim’s genDepend command.

* https://graphviz.org/

Uniform Function Call Syntax

The Uniform Function Call Syntax generalizes calling a function and calling a member function.

This relatively small feature has a big impact how you think about objects how you reference code.

You can call a procedure two equivalent ways, by passing all the arguments to it or by treating the first argument like you would for an object when calling its methods (dot notation). For example calling procName which take three parameters can be done like:

procName(a, b, c)

or

a.procName(b, c)


You can omit the parentheses when they are empty. For example when no parameters:

procName2()

or

procName2

Or when there is one parameter:

procName3(a)

or

a.procName3()

or

a.procName3


The first parameter can be any type, it is not restricted to objects and you can use variables or literals.

For an add procedure with two integer parameters you can call it shown below. The last line shows chaining by adding 1 + 2 + 3.

v1 = add(1, 2)
v2 = 1.add(2)
v3 = 1.add(2).add(3)


You can extend an object defined in another module by defining a procedure with the object as the first parameter.

newMethodName(obj, a, b, c)

And you call it like the other methods of the object:

obj.newMethodName(a, b, c)

When you omit the parentheses it appears like you are accessing a member variable. These are called object getters in some languages. For example:

“str”.len

The Nim tutorial talks about how to do “setters”.

* https://nim-lang.org/docs/tut2.html

See the wikipeadia page for more information.

* https://en.wikipedia.org/wiki/Uniform_Function_Call_Syntax

Posted in Nim

Nim Macros

The Nim programming language provides powerful meta-programming capabilities with nim macros. Macros run at compile time and they operate on nim’s abstract syntax tree (AST) directly. You write macros using the regular nim language.

This post tells how to write nim macros with lots of simple examples.

You can write a macro using a string of code or by creating AST structures. I call these two styles of macros:

  • text style macro
  • AST style macro

You can also categorize nim macros by how you invoke them.

1. Expression macro

You invoke an expression macro like a procedure call and it will generate new code at that point in the program. The macro generates AST from scratch and it is inserted at the point it is called.

2. Pragma macro

You invoke a pragma macro when a procedure is compiled by tagging the procedure with a pragma and naming the macro as the pragma. The procedure AST is passed to the macro for modification.

3. Block macro

You invoke a block macro when a block of code is compiled by naming the macro as the block. The block AST is passed to the block macro for modification.

You define each type of macro the same except the pragma and block macros have a hidden last parameter for the AST. In all cases macros return an AST.

Simple Example

Here is a simple nim program stored in the file t.nim. We will use it to investigate nim macros.

proc hello() =
  echo "hi"

hello()

The program defines the hello procedure then calls it.

Here is the output when compiling and running the program. All the Hint lines have been removed from the output for simplicity.

nim c -r t
hi

Text Style Expression Macro

Let’s write a text style expression macro to generate the hello proc above.

import macros

macro gen_hello(): typed =
  let source = """
proc hello() =
  echo "hi"
"""
  result = parseStmt(source)

gen_hello()
hello()

Here is the output when compiling and running:


nim c -r t
hi

The macro is defined like a procedure except you use “macro” instead of “proc”. The result is the AST you want to insert at the point the macro is called. The parseStmt converts the string to AST. The “hello()” call calls the hello procedure generated by the macro.

AST Style Expression Macro

Now lets write the same expression macro in AST style.

Before we do that we need to know what the AST looks like. You could consult the macro module docs. But it is easier run a couple of macros in the macros module to dump out the code so you can see the AST.

For example here is code to dump out our simple hello program using the dumpTree macro.

import macros
dumpTree:
  proc hello() =
    echo "hi"

When running it you get a list of AST nodes indented to show the hierarchy. At the root is a StmtList (statement list) and it contains one node called ProcDef for the definition of the procedure named hello.

StmtList
  ProcDef
    Ident !"hello"
    Empty
    Empty
    FormalParams
      Empty
    Empty
    Empty
    StmtList
      Command
        Ident !"echo"
        StrLit hi

You can also use the dumpAstGen macro. It will generate the textual code needed to build the AST.

import macros
dumpAstGen:
  proc hello() =
    echo "hi"

When running it you get:

nnkStmtList.newTree(
  nnkProcDef.newTree(
    newIdentNode(!"hello"),
    newEmptyNode(),
    newEmptyNode(),
    nnkFormalParams.newTree(
      newEmptyNode()
    ),
    newEmptyNode(),
    newEmptyNode(),
    nnkStmtList.newTree(
      nnkCommand.newTree(
        newIdentNode(!"echo"),
        newLit("hi")
      )
    )
  )
)

Now that we know the required AST we can write our AST style expression macro. We take the dumpAstGen output and assign it to result.

import macros
macro gen_hello(): typed =
  result = nnkStmtList.newTree(
    nnkProcDef.newTree(
      newIdentNode(!"hello"),
      newEmptyNode(),
      newEmptyNode(),
      nnkFormalParams.newTree(
        newEmptyNode()
      ),
      newEmptyNode(),
      newEmptyNode(),
      nnkStmtList.newTree(
        nnkCommand.newTree(
          newIdentNode(!"echo"),
          newLit("hi")
        )
      )
    )
  )
gen_hello()
hello()

Here is the output when compiling and running:

nim c -r t
hi

Pragma Macro

A pragma macro has the same name as a nim pragma. A pragma is specified with curly bracks like: {.pragma echoName.}. You add pragmas to procedures.

Let’s write a pragma macro to display the procedure’s name when the procedure is called. For our hello procedure the macro would transform it to:

proc hello():
  echo "hello"
  echo "hi"

Looking back at the output from dumpAstGen we see the AST structure we need to generate a command that echos “hello”.

    nnkStmtList.newTree(
      nnkCommand.newTree(
        newIdentNode(!"echo"),
        newLit("hello")
      ),

But we do not want to show “hello” for all procedures but instead show the procedure’s name. The name comes from the top of the tree in the IdentNode.

nnkStmtList.newTree(
  nnkProcDef.newTree(
    newIdentNode(!"hello"),

Here is our starting attempt at writing the pragma macro. The pragma and macro are called echoName. The macro is passed the AST of the procedure, in this case the procedure is main. The main procedure is annotated with the pragma. Notice it goes at the end of the procedure definition. The line “let msg = name(x)” gets the procedure name and the next line displays it.

import macros

macro echoName(x: untyped): untyped =
  let msg = name(x)
  echo msg

proc main (p: int): string {.echoName.} =
  result = "test"

Here is the output when compiling and running. During the macro processing step it outputs “main”. You can debug your macro with echo statements.

nim c -r t
Hint: used config file '/usr/local/Cellar/nim/0.17.2/nim/config/nim.cfg' [Conf]
Hint: system [Processing]
Hint: t [Processing]
Hint: macros [Processing]
main
Hint:  [Link]

The “name” procedure is defined in the macro module. It returns the name of the procedure given a procedure AST node.

The meta-type “untyped” matches anything. It is lazy evaluated so you can pass undefined symbols to it.

There are two other meta-types, typed and typedesc. They are not lazy evaluated.

Here is a working pragma macro that echoes the procedure name when it is called. The “let name = $name(x)” line gets the name of the procedure as a string. The next line creates a new node that echoes the name. The insert adds the node to the body of the procedure as the first statement. You can use treeRepr for debugging.

Our pragma macro is invoked at compile time for each proc tagged with the {.echoName.} pragma.

import macros

macro echoName(x: untyped): untyped =
  let name = $name(x)
  let node = nnkCommand.newTree(newIdentNode(!"echo"), newLit(name))
  insert(body(x), 0, node)
  # echo "treeRepr = ", treeRepr(x)
  result = x

proc add(p: int): int {.echoName.} =
  result = p + 1

proc process(p: int) {.echoName.} =
  echo "ans for ", p, " is ", add(p)

process(5)
process(8)

Here is the output when compiling and running:

nim c -r t

process
add
ans for 5 is 6
process
add
ans for 8 is 9

Now we enhance the macro to show how you pass parameters to pragmas. In this example we pass a custom message string. When invoking the pragma you add the parameter after a colon as shown below.

By default all arguments are AST expressions. The msg string is passed to the macro as a StrLit node, which happens to be what the newIdentNode procedure requires.

The ! operator in the macro module creates an identifier node from a string.

import macros

macro echoName(msg: untyped, x: untyped): untyped =
  let node = nnkCommand.newTree(newIdentNode(!"echo"), msg)
  insert(body(x), 0, node)
  result = x

proc add(p: int): int {.echoName: "calling add proc".} =
  result = p + 1

proc process(p: int) {.echoName: "calling process".} =
  echo "ans for ", p, " is ", add(p)

process(5)
process(8)

Here is the output when compiling and running:

calling process
calling add proc
ans for 5 is 6
calling process
calling add proc
ans for 8 is 9

Pass Normal Parameters

You can pass normal types to macros with the “static” syntax. Here is an example of passing an int. The macro echoes the name concatenated with the number.

import macros

macro echoName(value: static[int], x: untyped): untyped =
  let node = nnkCommand.newTree(newIdentNode(!"echo"), newLit($name(x) & $value))
  insert(body(x), 0, node)
  result = x

proc add(p: int): int {.echoName: 42} =
  result = p + 1

proc process(p: int) {.echoName: 43} =
  echo "ans for ", p, " is ", add(p)

process(5)
process(8)

output:

process43
add42
ans for 5 is 6
process43
add42
ans for 8 is 9

Multiple Macro Parameters

You can pass one parameter to a pragma macro. If you want to pass more values, you can use a tuple. Here is an example of passing a number and a string to the macro.

import macros

type
  Parameters = tuple[value: int, ending: string]

macro echoName(p: static[Parameters], x: untyped): untyped =
  # echo "x = ", treeRepr(x)
  let node = nnkCommand.newTree(newIdentNode(!"echo"),
               newLit($name(x) & $p.value & p.ending))
  insert(body(x), 0, node)
  result = x

proc add(p: int): int {.echoName: (42, "p1").} =
  result = p + 1

proc process(p: int) {.echoName: (43, "p2").} =
  echo "ans for ", p, " is ", add(p)

process(5)
process(8)

Here is the output when compiling and running:

process43p2
add42p1
ans for 5 is 6
process43p2
add42p1
ans for 8 is 9

Block Macro

If you name a block with the name of a macro, the macro is invoked when the block is compiled. The AST of the block is passed to the macro as the last parameter.

Here is an example block macro that prints out the AST past to it.

import macros

macro echoName(x: untyped): untyped =
  echo "x = ", treeRepr(x)
  result = x

echoName:
  proc add(p: int): int =
    result = p + 1

  proc process(p: int) =
    echo "ans for ", p, " is ", add(p)

process(5)
process(8)

Here is the results when compiling and running.

x = StmtList
  ProcDef
    Ident !"add"
    Empty
    Empty
    FormalParams
      Ident !"int"
      IdentDefs
        Ident !"p"
        Ident !"int"
        Empty
    Empty
    Empty
    StmtList
      Asgn
        Ident !"result"
        Infix
          Ident !"+"
          Ident !"p"
          IntLit 1
  ProcDef
    Ident !"process"
    Empty
    Empty
    FormalParams
      Empty
      IdentDefs
        Ident !"p"
        Ident !"int"
        Empty
    Empty
    Empty
    StmtList
      Command
        Ident !"echo"
        StrLit ans for
        Ident !"p"
        StrLit  is
        Call
          Ident !"add"
          Ident !"p"

To write the macro so it prints out the name of the procedures when called, we need to find the procedure nodes in the AST and add the echo as before.

You can use the children procedure to loop through the child nodes of the AST statements. You find the proc’s by checking for the node type nnkProcDef. You add nnk prefix to the names output by treeRepr. Notice we added echo statements to the block that we need to skip over.

import macros

macro echoName(x: untyped): untyped =
  for child in x.children():
    if child.kind == nnkProcDef:
      let node = nnkCommand.newTree(newIdentNode(!"echo"),
                   newLit($name(child)))
      insert(body(child), 0, node)
  result = x

echoName:
  echo "an echo statement"
  proc add(p: int): int =
    result = p + 1
  echo "another echo statement"
  proc process(p: int) =
    echo "ans for ", p, " is ", add(p)

process(5)
process(8)

Here is the output:

an echo statement
another echo statement
process
add
ans for 5 is 6
process
add
ans for 8 is 9

The following shows how to pass parameters to your block macros. In this example we pass the string “called” .

import macros

macro echoName(msg: static[string], x: untyped): untyped =
  for child in x.children():
    if child.kind == nnkProcDef:
      let node = nnkCommand.newTree(newIdentNode(!"echo"),
                   newLit($name(child) & "-" & $msg))
      insert(body(child), 0, node)
  result = x

echoName("called"):
  echo "an echo statement"
  proc add(p: int): int =
    result = p + 1
  echo "another echo statement"
  proc process(p: int) =
    echo "ans for ", p, " is ", add(p)

process(5)
process(8)

Here is the output:

an echo statement
another echo statement
process-called
add-called
ans for 5 is 6
process-called
add-called
ans for 8 is 9

More Information

See the macro module documentation for the complete AST syntax and other useful procedures and operators.

https://nim-lang.org/docs/macros.html

Posted in Nim