Category Archives: Linux

Git Grep and Open File

Posted on by

Git grep is a search command built around the Git ls-files command.

git ls-files | grep -v -f ignore.txt | xargs grep -n

The command line takes all the files checked into version control, strips out the unwanted files, then searches the remaining for a text string or regular expression.

The result is a list of matching lines with their filename and line numbers.

I wrote an Emacs command called slf-git-grep to run it. You select some text then press a key (ctrl+space) to find the selected text in all your source files. It opens a buffer for the results and the matching text is colored red.

This allows you to navigate your source with just a few keystrokes.

If there is no selection, it prompts you for the search string (or regular expression).

It works with multiple repositories on the same machine. Each one has its own results buffer that’s appended to.

The Git root folder is found mostly automatically by searching relative to the current file. If it cannot find it that way, it uses the Git root found last. If it still isn’t found, you are prompt for it.

You may have source checked in that you don’t want to look at. For example third party libs. You can exclude these files using ignore.txt.
The grep command looks for the ignore.txt file in the Git root directory and if not found it looks in your home folder.

The results buffer shows the relative filename and line number. This allows you to go to the found line with one keystoke (ctrl+o). That’s done using another command I wrote called slf-open-file-under-cursor.

slf-open-file-under-cursor is an emacs command which opens the file under the cursor and goes to the specified line number.

The command parses the current line to extract the filename and line number. Then it opens the file and goes to the line.

The command supports common file/line formats generated by grep, a couple stack traces and logging.

Filename is either a full path or a filename relative to the current directory.

Download from github at:

https://github.com/flenniken

XCAPE

Posted on by

I’ve configured my keyboard to generate a special key code when you press the Caps Lock key by itself. I use this as a mode switch in Emacs which I have configured somewhat like VIM.

For VIM users the escape key is used to switch between typing and command mode and escape is pressed all the time.

The problem is that it is a long reach to hit the escape key. The Caps Lock key is in a prime location and who needs the Caps Lock key anyway? (You can move it if you want.)

On Ubuntu I use the program xcape to accomplish this remapping. I will show you how to download, install, configure and test xcape.

Note:
On the Mac you can use KeyRemap4MacBook and PCKeyboardHack to do the same thing.

Here is the link to xcape: https://github.com/alols/xcape#readme

First install the necessary dependencies. It requires git, gcc, make libx11-dev and libxtst-dev libraries. Open a terminal and type:

sudo apt-get install git gcc make libx11-dev libxtst-dev

Next make a folder for xcape in your home folder:

cd
mkdir xcape
cd xcape

Download the source code for xcape:

git clone https://github.com/alols/xcape.git .

Build the application. This makes a executable called xcape.

make

ls
LICENSE Makefile README.md xcape xcape.c

To see the usage statement:

./xcape -h
./xcape: invalid option — ‘h’
Usage: ./xcape [-d] [-t timeout_ms] [-e <mapping>]
Runs as a daemon unless -d flag is set

I am using the Ubuntu system settings to turn the Caps Lock into a control key and xcape to generate Alt+R when it (now a control key) is pressed by itself.

Use the system settings to make Caps Lock a control key:

  • Open system settings Keyboard Layout
  • Click Options
  • Click “Ctrl key position”
  • Check “Caps Lock as Ctrl”
  • Close the dialogs

If you are a VIM user and you want Caps Lock by itself to generate an escape, use the defaults and run xcape like:

./xcape

I want the left control to generate Alt_L+R when pressed on its own, so I run xcape like:

./xcape -e ‘Control_L=Alt_L|R’

You can remap multiple keys at once using the semicolon to separate commands:

xcape -e ‘Shift_L=Escape;Control_L=Control_L|O’

The list of key names is found in the header file X11/keysymdef.h (remove the XK_ prefix). Here is a link to the file:

http://cgit.freedesktop.org/xorg/proto/x11proto/plain/keysymdef.h

Run xev to test the keycodes generated when you type a key.

xev

For my case the system generates a Control_L press and release then the Alt_L+R press and release combination.

Note:
If you change the keyboard system settings, rerun xcape or it will continue to run with the old settings.

Note:
Make sure you only run one xcape process. To see the xcape process or processes:

ps aux | grep xcape
steve 13342 0.0 0.0 11984 688 ? Ssl 09:25 0:00 ./xcape
steve 13467 0.0 0.0 11984 684 ? Ssl 09:44 0:00 ./xcape
steve 13564 0.0 0.0 11984 688 ? Ssl 10:01 0:00 ./xcape -e Control_L Alt_L R

Quit the extra running xcape processes above:

kill 13342
kill 13467

To make xcape run once when you login, add the xcape to your .bashrc file,

emacs ~/.bashrc

add this line to the bottom of the file:

# Generate a Alt+R when the control key is pressed and released by itself.
# Run xcape once.
if [ -z $XCAPE ] ; then
export XCAPE=1
~/xcape/xcape -e ‘Control_L=Alt_L|R’
fi

Number of Open Files Ubuntu

Posted on by

On Linux you are limited to the number of files you can have open at one time. This note tells how to determine the number of open files on Ubuntu and how to increase the limit.

To determine how many open files a process has, figure out the process id then run the lsof command.

To figure out the process id use the ps command, for example:
ps aux

Since there is so much output, it is often useful to grep for the process you are interested in, for example to find the tomcat process:

ps aux | grep tomcat
steve 21405 0.0 3.3 428148 69060 ? Sl 10:07 0:00 /usr/lib/jvm/java-6-sun/bin/java …

Here the process id is 21405.

Once you have the process id, you can see the open files using the lsof command. For example:

lsof -p 21405
”COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 21405 steve cwd DIR 8,1 4096 407944 /home/steve/apache”-tomcat-6.0.26/bin”
java 21405 steve rtd DIR 8,1 4096 2 /
java 21405 steve txt REG 8,1 47308 400238 /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/java”
…

To determine the number of open files, count the number of lines output by the lsof command, for example:
lsof -p 21405 | wc -l
119

An alternate way to see the open files.
ll /proc/21405/fd

To see the list sorted by filename where the names are in numerical sequence:
ll /proc/21405/fd | sort -g +7 -7

To sort by the Name column:
lsof -p 24226 | sort +8 -8 >~/tmp/open.txt

The FD column is the File Descriptor column. It is either the number of the file or one of the following:

  • cwd current working directory
  • Lnn library references (AIX)
  • err FD information error (see NAME column)
  • jld jail directory (FreeBSD)
  • ltx shared library text (code and data)
  • Mxx hex memory-mapped type number xx
  • m86 DOS Merge mapped file
  • mem memory-mapped file
  • mmap memory-mapped device
  • pd parent directory
  • rtd root directory
  • tr kernel trace file (OpenBSD)
  • txt program text (code and data)
  • v86 VP/ix mapped file

 

Determine the Limit

 

Each user has a limit for the number of open files. This limit applies to each process run by the user. For example say the limit is 1024 and the user has three processes running, each process can open 1024 files for a total of 3072.

To determine the soft limit:
ulimit -Sn
1024

To determine the hard limit:
ulimit -Hn
2048

ulimit -n shows you the soft limit. The soft limit is the limit applied for opening files. The hard limit is the limit you can increase the soft limit to.

 

Increase the Limit

 

To increase the limit to 1080 use the following command:

ulimit -Sn 1080

You can change the hard limit too, ulimit -Hn 2040. ulimit -n 2040 changes both the soft and hard limits to the same value. Once you change the hard limit, you cannot increase it above the value you just set without rebooting.

If you try to set the soft limit above the hard limit you get the following message:
ulimit -Sn 3000
bash: ulimit: open files: cannot modify limit: Invalid argument

Note: Once you reboot the limit is reset.

You cannot determine the limit of the root user using ulimit. For example:

sudo ulimit -n
sudo: ulimit: command not found

To make the limits bigger and to make the change permanent, edit your configuration file and reboot. On Ubuntu you edit the following file:

sudo nano /etc/security/limits.conf

Add lines like these:

steve soft nofile 4000
steve hard nofile 5000

You can use * in the limit.conf file instead of a user name to specify all users, however this does not apply to the root!

  • soft nofile 20000
  • hard nofile 30000

The limit.conf file is applied during the boot process. If you start a process during the boot process before the limits are applied, you will get the default 1024 value. You can record the starting limit in a file right before starting your process, then check the value to make sure it’s the expected value.

ulimit -n >mylimit.txt

You cannot start a process late enough in the boot process! For example: “sudo update-rc.d tomcat defaults 99 01” is at the end and it is still too late.

The work around is to force the limit to be set before starting the process. Put “ulimit -n 4000” before starting your process, then the limit.conf file is processed here.

 

Testing the Limit

 

I wrote a program called openmany that I use to test the open file limit. It creates a bunch of files in a folder then opens them.

java -jar openmany

Usage: openmany [-c] number
c Continue to run holding on to the open files.
number The number of files to open.

java -jar openmany 100
Creating 100 files in folder openmany.
Opening the files.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

To remove the directory of files created by the program:

rm -r openmany/

When setting the limit at 60,000, the system ran out of memory at about 30,000. So the effective limit is dependent on the java memory size allocated to the program.

Here is an example of trying to open more than the limit:
ulimit -n
1024
java -jar openmany.jar 1050
…
1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 Exception in thread “main” java.io. FileNotFoundException: openmany/1019.txt (Too many open files)

at java.io. FileInputStream.open (Native Method)
at java.io. FileInputStream.<init> (FileInputStream.java:106)
at java.io. FileReader.<init> (FileReader.java:55)
at openmany.Main.main (Main.java:49)

You can run the program as root and test its limits too:

sudo java -jar openmany.jar 1050

 

Set System Wide Limits

 

There is another file limit in the system, the total number of files that can be opened by all processes.

To see the file max value:
sysctl -a | grep fs.file-max
fs.file-max = 170469

Since it is so big there is no reason to change it.

 

References

 

Documentation for lsof:

http://man-wiki.net / index.php / 8:lsof%23SYNOPSIS

Changing the limit:

http://stackoverflow.com / questions / 34588 / how-do-i- change-the- number-of-open- files- limit- in-linux

http://www.cyberciti.biz / faq / linux-increase- the-maximum- number-of- open- files/