Nim Macros

The Nim programming language provides powerful meta-programming capabilities with nim macros. Macros run at compile time and they operate on nim’s abstract syntax tree (AST) directly. You write macros using the regular nim language.

This post tells how to write nim macros with lots of simple examples.

You can write a macro using a string of code or by creating AST structures. I call these two styles of macros:

  • text style macro
  • AST style macro

You can also categorize nim macros by how you invoke them.

1. Expression macro

You invoke an expression macro like a procedure call and it will generate new code at that point in the program. The macro generates AST from scratch and it is inserted at the point it is called.

2. Pragma macro

You invoke a pragma macro when a procedure is compiled by tagging the procedure with a pragma and naming the macro as the pragma. The procedure AST is passed to the macro for modification.

3. Block macro

You invoke a block macro when a block of code is compiled by naming the macro as the block. The block AST is passed to the block macro for modification.

You define each type of macro the same except the pragma and block macros have a hidden last parameter for the AST. In all cases macros return an AST.

Simple Example

Here is a simple nim program stored in the file t.nim. We will use it to investigate nim macros.

proc hello() =
  echo "hi"

hello()

The program defines the hello procedure then calls it.

Here is the output when compiling and running the program. All the Hint lines have been removed from the output for simplicity.

nim c -r t
hi

Text Style Expression Macro

Let’s write a text style expression macro to generate the hello proc above.

import macros

macro gen_hello(): typed =
  let source = """
proc hello() =
  echo "hi"
"""
  result = parseStmt(source)

gen_hello()
hello()

Here is the output when compiling and running:


nim c -r t
hi

The macro is defined like a procedure except you use “macro” instead of “proc”. The result is the AST you want to insert at the point the macro is called. The parseStmt converts the string to AST. The “hello()” call calls the hello procedure generated by the macro.

AST Style Expression Macro

Now lets write the same expression macro in AST style.

Before we do that we need to know what the AST looks like. You could consult the macro module docs. But it is easier run a couple of macros in the macros module to dump out the code so you can see the AST.

For example here is code to dump out our simple hello program using the dumpTree macro.

import macros
dumpTree:
  proc hello() =
    echo "hi"

When running it you get a list of AST nodes indented to show the hierarchy. At the root is a StmtList (statement list) and it contains one node called ProcDef for the definition of the procedure named hello.

StmtList
  ProcDef
    Ident !"hello"
    Empty
    Empty
    FormalParams
      Empty
    Empty
    Empty
    StmtList
      Command
        Ident !"echo"
        StrLit hi

You can also use the dumpAstGen macro. It will generate the textual code needed to build the AST.

import macros
dumpAstGen:
  proc hello() =
    echo "hi"

When running it you get:

nnkStmtList.newTree(
  nnkProcDef.newTree(
    newIdentNode(!"hello"),
    newEmptyNode(),
    newEmptyNode(),
    nnkFormalParams.newTree(
      newEmptyNode()
    ),
    newEmptyNode(),
    newEmptyNode(),
    nnkStmtList.newTree(
      nnkCommand.newTree(
        newIdentNode(!"echo"),
        newLit("hi")
      )
    )
  )
)

Now that we know the required AST we can write our AST style expression macro. We take the dumpAstGen output and assign it to result.

import macros
macro gen_hello(): typed =
  result = nnkStmtList.newTree(
    nnkProcDef.newTree(
      newIdentNode(!"hello"),
      newEmptyNode(),
      newEmptyNode(),
      nnkFormalParams.newTree(
        newEmptyNode()
      ),
      newEmptyNode(),
      newEmptyNode(),
      nnkStmtList.newTree(
        nnkCommand.newTree(
          newIdentNode(!"echo"),
          newLit("hi")
        )
      )
    )
  )
gen_hello()
hello()

Here is the output when compiling and running:

nim c -r t
hi

Pragma Macro

A pragma macro has the same name as a nim pragma. A pragma is specified with curly bracks like: {.pragma echoName.}. You add pragmas to procedures.

Let’s write a pragma macro to display the procedure’s name when the procedure is called. For our hello procedure the macro would transform it to:

proc hello():
  echo "hello"
  echo "hi"

Looking back at the output from dumpAstGen we see the AST structure we need to generate a command that echos “hello”.

    nnkStmtList.newTree(
      nnkCommand.newTree(
        newIdentNode(!"echo"),
        newLit("hello")
      ),

But we do not want to show “hello” for all procedures but instead show the procedure’s name. The name comes from the top of the tree in the IdentNode.

nnkStmtList.newTree(
  nnkProcDef.newTree(
    newIdentNode(!"hello"),

Here is our starting attempt at writing the pragma macro. The pragma and macro are called echoName. The macro is passed the AST of the procedure, in this case the procedure is main. The main procedure is annotated with the pragma. Notice it goes at the end of the procedure definition. The line “let msg = name(x)” gets the procedure name and the next line displays it.

import macros

macro echoName(x: untyped): untyped =
  let msg = name(x)
  echo msg

proc main (p: int): string {.echoName.} =
  result = "test"

Here is the output when compiling and running. During the macro processing step it outputs “main”. You can debug your macro with echo statements.

nim c -r t
Hint: used config file '/usr/local/Cellar/nim/0.17.2/nim/config/nim.cfg' [Conf]
Hint: system [Processing]
Hint: t [Processing]
Hint: macros [Processing]
main
Hint:  [Link]

The “name” procedure is defined in the macro module. It returns the name of the procedure given a procedure AST node.

The meta-type “untyped” matches anything. It is lazy evaluated so you can pass undefined symbols to it.

There are two other meta-types, typed and typedesc. They are not lazy evaluated.

Here is a working pragma macro that echoes the procedure name when it is called. The “let name = $name(x)” line gets the name of the procedure as a string. The next line creates a new node that echoes the name. The insert adds the node to the body of the procedure as the first statement. You can use treeRepr for debugging.

Our pragma macro is invoked at compile time for each proc tagged with the {.echoName.} pragma.

import macros

macro echoName(x: untyped): untyped =
  let name = $name(x)
  let node = nnkCommand.newTree(newIdentNode(!"echo"), newLit(name))
  insert(body(x), 0, node)
  # echo "treeRepr = ", treeRepr(x)
  result = x

proc add(p: int): int {.echoName.} =
  result = p + 1

proc process(p: int) {.echoName.} =
  echo "ans for ", p, " is ", add(p)

process(5)
process(8)

Here is the output when compiling and running:

nim c -r t

process
add
ans for 5 is 6
process
add
ans for 8 is 9

Now we enhance the macro to show how you pass parameters to pragmas. In this example we pass a custom message string. When invoking the pragma you add the parameter after a colon as shown below.

By default all arguments are AST expressions. The msg string is passed to the macro as a StrLit node, which happens to be what the newIdentNode procedure requires.

The ! operator in the macro module creates an identifier node from a string.

import macros

macro echoName(msg: untyped, x: untyped): untyped =
  let node = nnkCommand.newTree(newIdentNode(!"echo"), msg)
  insert(body(x), 0, node)
  result = x

proc add(p: int): int {.echoName: "calling add proc".} =
  result = p + 1

proc process(p: int) {.echoName: "calling process".} =
  echo "ans for ", p, " is ", add(p)

process(5)
process(8)

Here is the output when compiling and running:

calling process
calling add proc
ans for 5 is 6
calling process
calling add proc
ans for 8 is 9

Pass Normal Parameters

You can pass normal types to macros with the “static” syntax. Here is an example of passing an int. The macro echoes the name concatenated with the number.

import macros

macro echoName(value: static[int], x: untyped): untyped =
  let node = nnkCommand.newTree(newIdentNode(!"echo"), newLit($name(x) & $value))
  insert(body(x), 0, node)
  result = x

proc add(p: int): int {.echoName: 42} =
  result = p + 1

proc process(p: int) {.echoName: 43} =
  echo "ans for ", p, " is ", add(p)

process(5)
process(8)

output:

process43
add42
ans for 5 is 6
process43
add42
ans for 8 is 9

Multiple Macro Parameters

You can pass one parameter to a pragma macro. If you want to pass more values, you can use a tuple. Here is an example of passing a number and a string to the macro.

import macros

type
  Parameters = tuple[value: int, ending: string]

macro echoName(p: static[Parameters], x: untyped): untyped =
  # echo "x = ", treeRepr(x)
  let node = nnkCommand.newTree(newIdentNode(!"echo"),
               newLit($name(x) & $p.value & p.ending))
  insert(body(x), 0, node)
  result = x

proc add(p: int): int {.echoName: (42, "p1").} =
  result = p + 1

proc process(p: int) {.echoName: (43, "p2").} =
  echo "ans for ", p, " is ", add(p)

process(5)
process(8)

Here is the output when compiling and running:

process43p2
add42p1
ans for 5 is 6
process43p2
add42p1
ans for 8 is 9

Block Macro

If you name a block with the name of a macro, the macro is invoked when the block is compiled. The AST of the block is passed to the macro as the last parameter.

Here is an example block macro that prints out the AST past to it.

import macros

macro echoName(x: untyped): untyped =
  echo "x = ", treeRepr(x)
  result = x

echoName:
  proc add(p: int): int =
    result = p + 1

  proc process(p: int) =
    echo "ans for ", p, " is ", add(p)

process(5)
process(8)

Here is the results when compiling and running.

x = StmtList
  ProcDef
    Ident !"add"
    Empty
    Empty
    FormalParams
      Ident !"int"
      IdentDefs
        Ident !"p"
        Ident !"int"
        Empty
    Empty
    Empty
    StmtList
      Asgn
        Ident !"result"
        Infix
          Ident !"+"
          Ident !"p"
          IntLit 1
  ProcDef
    Ident !"process"
    Empty
    Empty
    FormalParams
      Empty
      IdentDefs
        Ident !"p"
        Ident !"int"
        Empty
    Empty
    Empty
    StmtList
      Command
        Ident !"echo"
        StrLit ans for
        Ident !"p"
        StrLit  is
        Call
          Ident !"add"
          Ident !"p"

To write the macro so it prints out the name of the procedures when called, we need to find the procedure nodes in the AST and add the echo as before.

You can use the children procedure to loop through the child nodes of the AST statements. You find the proc’s by checking for the node type nnkProcDef. You add nnk prefix to the names output by treeRepr. Notice we added echo statements to the block that we need to skip over.

import macros

macro echoName(x: untyped): untyped =
  for child in x.children():
    if child.kind == nnkProcDef:
      let node = nnkCommand.newTree(newIdentNode(!"echo"),
                   newLit($name(child)))
      insert(body(child), 0, node)
  result = x

echoName:
  echo "an echo statement"
  proc add(p: int): int =
    result = p + 1
  echo "another echo statement"
  proc process(p: int) =
    echo "ans for ", p, " is ", add(p)

process(5)
process(8)

Here is the output:

an echo statement
another echo statement
process
add
ans for 5 is 6
process
add
ans for 8 is 9

The following shows how to pass parameters to your block macros. In this example we pass the string “called” .

import macros

macro echoName(msg: static[string], x: untyped): untyped =
  for child in x.children():
    if child.kind == nnkProcDef:
      let node = nnkCommand.newTree(newIdentNode(!"echo"),
                   newLit($name(child) & "-" & $msg))
      insert(body(child), 0, node)
  result = x

echoName("called"):
  echo "an echo statement"
  proc add(p: int): int =
    result = p + 1
  echo "another echo statement"
  proc process(p: int) =
    echo "ans for ", p, " is ", add(p)

process(5)
process(8)

Here is the output:

an echo statement
another echo statement
process-called
add-called
ans for 5 is 6
process-called
add-called
ans for 8 is 9

More Information

See the macro module documentation for the complete AST syntax and other useful procedures and operators.

https://nim-lang.org/docs/macros.html

Processing DNG

How is a JPEG generated from a DNG image file?

DNG is the standard raw format. I have been using it for many years.

Even though it adds a conversion step, I think it is worthwhile.

I load my camera raw photos onto my machine, convert them to DNG using Adobe’s raw converter, then I delete the original raws.

Like the old 35mm negative film I save DNG files forever so I can make new images and get the best quality output. In the old days you would go to your darkroom or drugstore to have the negative made into a print. This process of enlarging, cropping and color adjustment is called processing or developing the image.

In the digital world JPEG is the like the old print. It is the final result for photos. There are other formats but none come close to JPEGs.

The DNG to JPEG processing step is mostly ignored by the articles on raw and DNG files.

I haven’t thought much about the full ramifications of the processing step until recently even though I have been using DNG for over a decade.

My raw files contain a small preview, I set it to 1024 x 768 to save storage space.

I do not store JPEGs for my raw files. I don’t want to maintain two files for the same image. When I edit the image, it would be a pain to keep the JPEG synced. I would have to come up with a scheme so two duplicate images do not show when viewing among other things.

Instead the raw files are processed on the fly when looking at them. I use an old version of Adobe Bridge as my viewer. The processing takes about two seconds on my machine the first time. After the first time it is fast because the processed data is cached.

I make JPEGs whenever I want to share my images. When posting to my or other websites or when transferring to friends.

In the old days you would write down cropping, sizing and exposure information for the developer to follow.

The equivalent with the digital negative is the embedded raw metadata. It holds the developer information.

The raw pixel data does not change when you edit in Camera Raw or Adobe Lightroom, just the embedded metadata changes. It contains data for the exposure, cropping etc. as text and numbers, for example:

Exposure .90
Cropping 1,2,3,4

There is no standard way to process the DNG to make a JPEG. The software to read the raw pixel data and apply the metadata to process it is proprietary. Adobe does this in their apps like Bridge and Lightroom seamlessly. It is easy to miss this important processing step.

You do notice it if you don’t have any Adobe applications. Try viewing without. For long term storage this is an issue.

There is a great open source command line program called dcraw that can process raw. It is used internally by a lot of apps that say they handle DNG.

Irfan viewer a popular free Windows image viewer uses dcraw. So does ImageMagik, a popular open source image library. Raw editing apps on Linux like Ufraw use it too.

The required processing metadata is documented by the DNG specification but how to interpret it is not specified.

The JPEG you get from dcraw does not look like the JPEG generated by Bridge. The exposure, cropping and rotation specified in the metadata is not honored. There are a lot of parameters to control dcraw, but none that control these options.

ImageMagik supports rotation and cropping, so if you were writing the processing code yourself you could figure out how to do this. At least these two seem straight forward. But what about exposure and shadow adjustment, color and the others? You need to be an imaging processing expert.

Adobe provides an SDK for dealing with DNG . I used it about seven years ago to produce JPEGs. It did not support cropping and rotation then. I’m not sure what it supports now. It was similar to dcraw and it took about the same two seconds to process.

I wonder how the cameras generate their JPEGs? My new camera can take ten frames a second. These frames are made from the raw pixel data. My guess is that the JPEG processing takes place later with a parallel process. I’ve never noticed any delay to the monitor which needs similar processing to create the image shown.

I have been thinking about this because I am writing a photo website to handle my raw workflow.

I was planning to upload my DNGs and generate JPEGs from them as needed on the server running on Linux. There doesn’t seem to be any way to do this without rolling my own code and reverse engineering what Adobe has done. This is more than I want to do.

Google+ and Google Picasa say they support DNG. I not sure whether they do more than dcraw on the processing side.

Flickr doesn’t support DNG. Interesting ideas thread:

flickrideas

You can make edits in Adobe Lightroom and see them in Adobe Photoshop. It gives the illusion that there exists a well known way to process the raw data. Can you make edits in Lightroom and see them in Aperture? Adobe can share the processing software between Adobe apps but do they share this code with other companies?

I found interesting information on Apple Aperture.

https://thephotosexpert.com/forum/raw-vs-dng/8766#.VIJHAAAgLA

From Walter Rowe
http://www.walterrowe.com/

Hi Joseph,

Let me see if I can be more clear.

Yes, Aperture can import a raw file in DNG format. And it knows the camera make and model and provides unique raw file processing based on that information. Aperture also consumes all of the meta data embedded in the DNG file like IPTC, keywords, contact info, ratings, labels, etc. All is well and good on the import side. The export side is where Aperture is incomplete regarding DNG.

Aperture can export a DNG master (eg. “Export master..”), provided it was imported as a DNG. It appears to only spit out the original DNG file you imported and any meta data you have written back to the master DNG file inside Aperture.

There seem to be two things missing from Aperture’s “Export master..” process for DNG files. The first is including an embedded, fully-rendered preview. This is an sRGB JPG that is fully baked with all your Aperture adjustments. Second is all of the adjustment settings themselves. I think Apple could add both of these easily enough if they chose.

The Adobe products include these pieces of data. I can “edit” a DNG in Lightroom, save everything back to the DNG file, open it in Photoshop, and see all of my adjustments from Lightroom. Likewise, I can make adjustments to a DNG file in Adobe Camera Raw, save them, and see these adjustments in Lightroom. And both products will embed the updated, baked preview inside the DNG file.

The embedded preview can be consumed by image management tools like MediaOnePro (formerly iView Media Pro). This frees image mgmt tools from needing to know how to interpret and render raw sensor data from different camera makers, and lets these tools include color-accurate thumbnails and previews in the image management database.

The DNG file format is nice. It retains all of the manufacturer’s original raw file data, can include the original raw file itself, can incorporate a “baked” preview with all your adjustments, can include all of the raw adjustments, and can include all of your meta data. It is a nicely packaged file format with everything you need for long term image management.

It would be nice to see Apple fully support all the features of the DNG file format in the “Export master..” process.

More information about DNG can be found on Adobe’s DNG page.

Does that help?
Walter

From a practical point of view I can get JPEGS by creating them using Adobe apps and uploading them to my website. But this adds manual steps I would rather not have.

What I am beginning to think that the best solution for my website is to change the raw preview setting so the full resolution JPEG is embedded in my DNG files.

When you edit a raw file the preview needs to be updated to match. This is the case with Adobe applications.

I need to figure out an easy way to update all my DNG files to have full resolution previews. Once that is done all my website needs to do is extract the embedded previews.

Words for Rain

They say that Eskimos have a 100 words for snow. As a native of Seattle, I wonder how many words for rain I know. This is the list I came up with.

Rain
Showers
Mist
Drizzle
Downpour
Cloudburst
Sleet
Torrential downpour
Monsoon
Pouring
Cats and dogs
Pineapple Express
Freezing rain
Sprinkle
Liquid sunshine
Deluge
Drencher
Flood
Flurry
Precipitation
Raindrops
Rainfall
Rainstorm
Sheets
Wet weather
Squall

Git Grep and Open File

Git grep is a search command built around the Git ls-files command.

git ls-files | grep -v -f ignore.txt | xargs grep -n

The command line takes all the files checked into version control, strips out the unwanted files, then searches the remaining for a text string or regular expression.

The result is a list of matching lines with their filename and line numbers.

I wrote an Emacs command called slf-git-grep to run it. You select some text then press a key (ctrl+space) to find the selected text in all your source files. It opens a buffer for the results and the matching text is colored red.

This allows you to navigate your source with just a few keystrokes.

If there is no selection, it prompts you for the search string (or regular expression).

It works with multiple repositories on the same machine. Each one has its own results buffer that’s appended to.

The Git root folder is found mostly automatically by searching relative to the current file. If it cannot find it that way, it uses the Git root found last. If it still isn’t found, you are prompt for it.

You may have source checked in that you don’t want to look at. For example third party libs. You can exclude these files using ignore.txt.
The grep command looks for the ignore.txt file in the Git root directory and if not found it looks in your home folder.

The results buffer shows the relative filename and line number. This allows you to go to the found line with one keystoke (ctrl+o). That’s done using another command I wrote called slf-open-file-under-cursor.

slf-open-file-under-cursor is an emacs command which opens the file under the cursor and goes to the specified line number.

The command parses the current line to extract the filename and line number. Then it opens the file and goes to the line.

The command supports common file/line formats generated by grep, a couple stack traces and logging.

Filename is either a full path or a filename relative to the current directory.

Download from github at:

https://github.com/flenniken

Send Email using Python and 1and1

The code you use to send email depends on whether the code is running on 1and1.com or on an external machine.

If you are on an external machine like your desktop, you use an SMTP script to send through your 1and1 email account. The following script called sendmail.py sends an email with python.

#!/usr/bin/python
import smtplib
from email.mime.text import MIMEText
me = "steve@example.com"
you = "sam@comcast.net"
password = "mypassword"
msg = MIMEText("This is a test message body.")
msg['Subject'] = 'Test message'
msg['From'] = me
msg['To'] = you
session = smtplib.SMTP("smtp.1and1.com", 587)
session.login(me, password)
session.sendmail(me, you, msg.as_string())
session.quit()

Run it like this: python sendmail.py

On 1and1 you need a mailbox type email. These types of emails have a password that you use in the script. It is easy to create a mailbox, if you don’t have one, by by logging into 1and1 and using their web interface.

If you copy this script to your web space on 1and1 and run it there, it will fail. 1and1 does not allow SMTP within 1and1. To send an email when running on 1and1 use this script:

#!/usr/bin/python
import os
mail_to = "sam@comcast.net"
mail_from = "steve@example.com"
subject = "test message"
header = """From: {0}
To: {1}
Subject: {2}
""".format(mail_from, mail_to, subject)
msg = header + "a test from me"
sendmail = os.popen("/usr/lib/sendmail -t", "w")
sendmail.write(msg)
sendmail.close()

Just open sendmail and write to it!

Firefox backspace

On Ubuntu Firefox the backspace key it does nothing by default.  You can change a setting to make it go to the previous page. Filefox has a bunch of settings you can access by typing about:config in the address bar. By changing browser.backspace_action from 2 to 0, it does the trick.

When you type about:config in the address bar you get an interesting warning dialog.

After you promise to be careful, type browser.backspace_action into the search box. Then change the 2 to a 0.

XCAPE

I’ve configured my keyboard to generate a special key code when you press the Caps Lock key by itself. I use this as a mode switch in Emacs which I have configured somewhat like VIM.

For VIM users the escape key is used to switch between typing and command mode and escape is pressed all the time.

The problem is that it is a long reach to hit the escape key. The Caps Lock key is in a prime location and who needs the Caps Lock key anyway? (You can move it if you want.)

On Ubuntu I use the program xcape to accomplish this remapping. I will show you how to download, install, configure and test xcape.

Note:
On the Mac you can use KeyRemap4MacBook and PCKeyboardHack to do the same thing.

Here is the link to xcape: https://github.com/alols/xcape#readme

First install the necessary dependencies. It requires git, gcc, make libx11-dev and libxtst-dev libraries. Open a terminal and type:

sudo apt-get install git gcc make libx11-dev libxtst-dev

Next make a folder for xcape in your home folder:

cd
mkdir xcape
cd xcape

Download the source code for xcape:

git clone https://github.com/alols/xcape.git .

Build the application. This makes a executable called xcape.

make

ls
LICENSE Makefile README.md xcape xcape.c

To see the usage statement:

./xcape -h
./xcape: invalid option — ‘h’
Usage: ./xcape [-d] [-t timeout_ms] [-e <mapping>]
Runs as a daemon unless -d flag is set

I am using the Ubuntu system settings to turn the Caps Lock into a control key and xcape to generate Alt+R when it (now a control key) is pressed by itself.

Use the system settings to make Caps Lock a control key:

  • Open system settings Keyboard Layout
  • Click Options
  • Click “Ctrl key position”
  • Check “Caps Lock as Ctrl”
  • Close the dialogs

If you are a VIM user and you want Caps Lock by itself to generate an escape, use the defaults and run xcape like:

./xcape

I want the left control to generate Alt_L+R when pressed on its own, so I run xcape like:

./xcape -e ‘Control_L=Alt_L|R’

You can remap multiple keys at once using the semicolon to separate commands:

xcape -e ‘Shift_L=Escape;Control_L=Control_L|O’

The list of key names is found in the header file X11/keysymdef.h (remove the XK_ prefix). Here is a link to the file:

http://cgit.freedesktop.org/xorg/proto/x11proto/plain/keysymdef.h

Run xev to test the keycodes generated when you type a key.

xev

For my case the system generates a Control_L press and release then the Alt_L+R press and release combination.

Note:
If you change the keyboard system settings, rerun xcape or it will continue to run with the old settings.

Note:
Make sure you only run one xcape process. To see the xcape process or processes:

ps aux | grep xcape
steve 13342 0.0 0.0 11984 688 ? Ssl 09:25 0:00 ./xcape
steve 13467 0.0 0.0 11984 684 ? Ssl 09:44 0:00 ./xcape
steve 13564 0.0 0.0 11984 688 ? Ssl 10:01 0:00 ./xcape -e Control_L Alt_L R

Quit the extra running xcape processes above:

kill 13342
kill 13467

To make xcape run once when you login, add the xcape to your .bashrc file,

emacs ~/.bashrc

add this line to the bottom of the file:

# Generate a Alt+R when the control key is pressed and released by itself.
# Run xcape once.
if [ -z $XCAPE ] ; then
export XCAPE=1
~/xcape/xcape -e ‘Control_L=Alt_L|R’
fi

Number of Open Files Ubuntu

On Linux you are limited to the number of files you can have open at one time. This note tells how to determine the number of open files on Ubuntu and how to increase the limit.

To determine how many open files a process has, figure out the process id then run the lsof command.

To figure out the process id use the ps command, for example:
ps aux

Since there is so much output, it is often useful to grep for the process you are interested in, for example to find the tomcat process:

ps aux | grep tomcat
steve 21405 0.0 3.3 428148 69060 ? Sl 10:07 0:00 /usr/lib/jvm/java-6-sun/bin/java …

Here the process id is 21405.

Once you have the process id, you can see the open files using the lsof command. For example:

lsof -p 21405
”COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 21405 steve cwd DIR 8,1 4096 407944 /home/steve/apache”-tomcat-6.0.26/bin
java 21405 steve rtd DIR 8,1 4096 2 /
java 21405 steve txt REG 8,1 47308 400238 /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/java”

To determine the number of open files, count the number of lines output by the lsof command, for example:
lsof -p 21405 | wc -l
119

An alternate way to see the open files.
ll /proc/21405/fd

To see the list sorted by filename where the names are in numerical sequence:
ll /proc/21405/fd | sort -g +7 -7

To sort by the Name column:
lsof -p 24226 | sort +8 -8 >~/tmp/open.txt

The FD column is the File Descriptor column. It is either the number of the file or one of the following:

  • cwd current working directory
  • Lnn library references (AIX)
  • err FD information error (see NAME column)
  • jld jail directory (FreeBSD)
  • ltx shared library text (code and data)
  • Mxx hex memory-mapped type number xx
  • m86 DOS Merge mapped file
  • mem memory-mapped file
  • mmap memory-mapped device
  • pd parent directory
  • rtd root directory
  • tr kernel trace file (OpenBSD)
  • txt program text (code and data)
  • v86 VP/ix mapped file

 

Determine the Limit

 

Each user has a limit for the number of open files. This limit applies to each process run by the user. For example say the limit is 1024 and the user has three processes running, each process can open 1024 files for a total of 3072.

To determine the soft limit:
ulimit -Sn
1024

To determine the hard limit:
ulimit -Hn
2048

ulimit -n shows you the soft limit. The soft limit is the limit applied for opening files. The hard limit is the limit you can increase the soft limit to.

 

Increase the Limit

 

To increase the limit to 1080 use the following command:

ulimit -Sn 1080

You can change the hard limit too, ulimit -Hn 2040. ulimit -n 2040 changes both the soft and hard limits to the same value. Once you change the hard limit, you cannot increase it above the value you just set without rebooting.

If you try to set the soft limit above the hard limit you get the following message:
ulimit -Sn 3000
bash: ulimit: open files: cannot modify limit: Invalid argument

Note: Once you reboot the limit is reset.

You cannot determine the limit of the root user using ulimit. For example:

sudo ulimit -n
sudo: ulimit: command not found

To make the limits bigger and to make the change permanent, edit your configuration file and reboot. On Ubuntu you edit the following file:

sudo nano /etc/security/limits.conf

Add lines like these:

steve soft nofile 4000
steve hard nofile 5000

You can use * in the limit.conf file instead of a user name to specify all users, however this does not apply to the root!

  • soft nofile 20000
  • hard nofile 30000

The limit.conf file is applied during the boot process. If you start a process during the boot process before the limits are applied, you will get the default 1024 value. You can record the starting limit in a file right before starting your process, then check the value to make sure it’s the expected value.

ulimit -n >mylimit.txt

You cannot start a process late enough in the boot process! For example: “sudo update-rc.d tomcat defaults 99 01” is at the end and it is still too late.

The work around is to force the limit to be set before starting the process. Put “ulimit -n 4000” before starting your process, then the limit.conf file is processed here.

 

Testing the Limit

 

I wrote a program called openmany that I use to test the open file limit. It creates a bunch of files in a folder then opens them.

java -jar openmany

Usage: openmany [-c] number
c Continue to run holding on to the open files.
number The number of files to open.

java -jar openmany 100
Creating 100 files in folder openmany.
Opening the files.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

To remove the directory of files created by the program:

rm -r openmany/

When setting the limit at 60,000, the system ran out of memory at about 30,000. So the effective limit is dependent on the java memory size allocated to the program.

Here is an example of trying to open more than the limit:
ulimit -n
1024
java -jar openmany.jar 1050

1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 Exception in thread “main” java.io. FileNotFoundException: openmany/1019.txt (Too many open files)

at java.io. FileInputStream.open (Native Method)
at java.io. FileInputStream.<init> (FileInputStream.java:106)
at java.io. FileReader.<init> (FileReader.java:55)
at openmany.Main.main (Main.java:49)

You can run the program as root and test its limits too:

sudo java -jar openmany.jar 1050

 

Set System Wide Limits

 

There is another file limit in the system, the total number of files that can be opened by all processes.

To see the file max value:
sysctl -a | grep fs.file-max
fs.file-max = 170469

Since it is so big there is no reason to change it.

 

References

 

Documentation for lsof:

http://man-wiki.net / index.php / 8:lsof%23SYNOPSIS

Changing the limit:

http://stackoverflow.com / questions / 34588 / how-do-i- change-the- number-of-open- files- limit- in-linux

http://www.cyberciti.biz / faq / linux-increase- the-maximum- number-of- open- files/

I love you too!

Wow, I’m getting lots of blog comments! I didn’t realize how much I touched you with my posts. You really like me!

“I have simply found your site and revel in each article. We admire your own expertise.”

I’m so happy. You “revel in each article” and admire my expertise!

“Appreciate it for the superb writeup. Anyhow, exactly how could we talk?”

You want to connect up too?

“We recently came all through your post and therefore are currently examining together. We need to communicate my admiration of the crafting ability as well as capability to help to make target audience study through the use of the starting to the finish. I would like to research more recent articles and to talk about my personal suggestions with you.”

You want to personally tell me how much you admire my writing skill? Oh, and you have some suggestions to make my articles better? Tell me more.

“Its like you read my mind! You appear to understand a lot about this, like you authored the e book in it or something. I believe that you can perform with some pics to drive the message home a little bit, but other than that, this is great blog. an awesome read. I will certainly be back..”

So you are thinking about Java JNI and Emacs too! Yes I know, a few pictures would make the posts more interesting. I’m glad you will be visiting my site often.

“Awesome web site you have right here however i had been interested in should you understood of any discussion boards that cover the same topics talked about in this article? Id really like to be a part of network where I’m able to get responses using their company experienced people who reveal the same curiosity. If you have any kind of recommendations, make sure you let me know. Cheers!”

You love me! I’m sorry I cannot recommend any company men with the same curiosity about Java programming. But if I do find them I’m sure to let you know.

“grey Ugg Boots For Men You actually make it seem so easy with your presentation but I find this topic to be really something that I think I would never understand. It seems too complicated and very broad for me. I am looking forward for your next post, I will try to get the hang of it! Kenly Uggs on sale”

I know it complicated determining the best direction to take. Both Java JNI and native applications have their benefits. I’m happy you’re hanging in there. Oh, you’re interested in boots?

“The post is absolutely fantastic! Lots of great information and inspiration, both of which we all need! Also like to admire the time and effort you put into your blog and detailed information you offer! I will bookmark your site!”

So you really like the post about my Apple ID problem. It is fantastic! You’re bookmarking my site, the ultimate compliment!

“i discovered your website on reddit as well as thought i’d can be found in and have a appear. fascinating concept you have nevertheless i will possess a number of other folks y you’ve got an interest. they can assist or maybe might not however its well worth a chance.”

I wonder who all my fans are and how can I find out more about them? Oh, I see they provided an email and a website link. I wonder whether I can find out more by following the links…

“I really wish I hadn’t seen this as I really want one now!”

dance wear, urban wear, http://www.partywears…
Luckner @ aol.com

What do you want?  Lets see … party wear …

“My brother recommended I would possibly like this web site.
He used to be entirely right. This post truly made my day. You can not consider just how a lot time I had spent for this information! Thanks!”

http://ereaderplanet.co.za/ kindle-south-africa/ kindle-ebook…
NelomsScianna335 @ aol.com

I’m excited to hear that you are talking over my posts with your brother! I’m thrilled that my post about WordPress on 1and1 made your day. Wow, South Africa fan. It really is the World Wide Web!

“Great issues altogether, you just received a new reader. What could you suggest about your put up that you simply made a few days in the past? Any positive?”

MccarronLongnecker45 @ gnumail.com
http://ereaderplanet.co.za/ kindle-south-africa/ the-kindle-reader-and…

Keep the comments coming about my put ups.  I love you too!

Load Emacs Documentation on your Kindle

There is a lot of free content you can load onto your Kindle. It just a matter of converting to the right format for your Kindle. It is a great device for reading manuals. I’ve loaded my Kindle with all the Python documentation as well as the emacs documentation. Here I show how to load the emacs manual. You can use the same methods for others.

These steps can be done on the Windows and Macintosh platforms as well as Ubuntu. Here are the steps in detail for Ubuntu.

Configuration:

  • Kindle 2, Model D00511
  • Ubunutu 10.04 LTS

Download Emacs Single Page Document

Emacs provides a single page HTML file of all the documentation. Download this page to your machine.

cd ~/tmp

wget http://www.gnu.org/ software/ emacs/ manual/ html_mono/ emacs.html

You can see the file in the tmp folder.

ll -h
-rw-r–r– 1 steve steve 3.3M 2011-09-25 11:55 emacs.html

Note: Python documentation comes in a zip file of many HTML files. From all these files you can create an ebook for your Kindle. Just point the converter at the content.html file.

Install calibre

You use calibre to convert the html file to a book (mobi file). If you don’t have calibre, you can install it with:

sudo apt-get install calibre

Here is the current version:

ebook-convert –version
ebook-convert (calibre 0.8.20)
Created by: Kovid Goyal <kovid@kovidgoyal.net>

Note: There are many options you can specify when converting from html to mobi. The options are listed here:
http://manual.calibre-ebook.com/cli/ebook-convert-3.html#html-input-to-mobi-output

Convert HTML to Mobi

If you use the defaults for conversion, the index at the beginning of the book is one big table that runs for many pages. You cannot select any of the links so it is pretty useless. To fix this, specify linearize-tables option and the table will be converted to regular text.

Convert the emacs.html file into a mobi file and turn the tables into regular text. It takes a few minutes to convert:

ebook-convert emacs.html emacs.mobi –linearize-tables –title “The Emacs Editor” –authors gnu.org –comments “The Emacs Editor Kindle book generated from gnu.org single page HTML file.”
 

You can see the file created:

ll -h
total 4.8M
-rw-r–r– 1 steve steve 3.3M 2011-09-25 11:55 emacs.html
-rw-r–r– 1 steve steve 1.5M 2011-09-25 13:20 emacs.mobi

Copy to Kindle

Copy the book to your Kindle. Plug it into the USB port then:

cp emacs.mobi /media/Kindle/documents/
eject Kindle

The content page generated by calibre is at the end of the document. The book is called “The Emacs Editor”

Now you can store the manual under your pillow.