RegEx Help

This ML based regex generator is quite handy! 

https://www.autoregex.xyz/home

QuickTip: Split Single Column into Multiple Columns

Consider a single column file which begins with

$ cat file.txt
1
yes
single
125K
no
2
no
married
100K
no

...

Suppose you want to split it into 5 columns so that it looks like

1 yes single 125K no

2 no married 100K no

...

You can use either,

$ xargs -n5 < file.txt

or if you want some control over the delimiter

$ paste - - - - - -d, < file.txt

1,yes,single,125K,no

2,no,married,100K,no

QuickTip: Reducing PDF Size using GhostScript

The command is:
gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/screen -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf

Source.

nohup and disown

GNU Screen is good for "detaching" jobs associated with a particular terminal, so that the job carries on even when, for example, the terminal is killed, or you log out from a server.

"nohup" and "disown" are other useful Linux commands with a far shallower learning curve.

Case A: You know before-hand that you want to run a background job without interruption

nohup foo&

Case B: You submitted several background jobs, but now you want the jobs to persist even after you kill the terminal

foo1&
foo2&
disown

Or if you want to disown only a particular job, use the "jobs" command to find out job-ids and,

disown %1 # disowns only foo1

Here is a nice thread on StackExchange on the difference between nohup and disown.

Background and Foreground Jobs in Linux

Linux/Unix lets you control how you interface with jobs quite conveniently. Here is a cheat-sheet I keep for my own use:

Send Foreground to Background

• Start foreground job in terminal
• Press Ctrl+Z to suspend job
• Type "bg" to send it to background
• Check background jobs with jobs, top, or ps commands

Example:

$ sleep 1000    # job running in foreground
^Z
[1]+  Stopped                 sleep 1000

$ bg
[1]+ sleep 1000 &

$ jobs
[1]+  Running                 sleep 1000 &

Send Background to Foreground

• Suppose you have multiple background jobs running 
• Use jobs command to list them
• "fg" brings last background job into foreground
• fg %1 brings job #1 listed in output of the jobs command

Example:

$ sleep 200&   # submit job#1
[1] 9074

$ sleep 100&  # submit job #2
[2] 9075

$ fg       # bring job #2 to foreground
sleep 100

^Z       # suspend it 
[2]+  Stopped                 sleep 100

$ bg   # put it back into background
[2]+ sleep 100 &

$ jobs   # check if both jobs running
[1]-  Running                 sleep 200 &
[2]+  Running                 sleep 100 &

$ fg %1  # bring job #1 to foreground
sleep 200

^C  # kill it using Ctrl+C

$ jobs  # check to see if job#2 is still running
[2]+  Running                 sleep 100 &

Setting up SpellCheck in TeXmaker

As the official documentation suggests, you go to "Configure Texmaker" -> "Editor" -> "Spelling dictionary" and set up the location.

The default location on my Linux distribution was: /usr/share/myspell/en_GB.dic

As the wikipedia entry says:

MySpell was the former spell checker included with OOo Writer of the free OpenOffice.org office suite.

The current spell checker of choice for OpenOffice is Hunspell. We can configure TeXmaker to use this dictionary by pointing to the location: /usr/share/hunspell/en_US.dic

Note, you can try the command locate hunspell to figure out where the "dic" file rests on your installation.

Splitting a large text file by number of lines and tags

Say you have a big file (text, picture, movie, etc) and you want to split it into many small parts. You may want to do this to email it to someone in more manageable chunks, or perhaps analyze it using a program that cannot handle all of the data at once.

The Linux command split lets you chop your file into chunks of specified size. To break a big file called "BigFile.mpg" into multiple smaller chunks "chunkaa", "chunkab" etc. you say somthing like.

split -b 10M BigFile.mpg chunk

Consider a simpler case, where the big file is a text file. For concreteness assume that BigFile.txt looks like:

# t = 0
particle1-coordinates
particle2-coordinates
...
particleN-coordinates

# t = 1
particle1-coordinates
particle2-coordinates
...
particleN-coordinates
...
# t = tfinal
particle1-coordinates
particle2-coordinates
...
particleN-coordinates

You may generate one of these, if you are running a particle-based simulation like MD, and printing out the coordinates of N particles in your systems periodically. For concreteness say N = 1000, and tfinal = 500.

If this file were too big, and you wanted to split it up into multiple files (one file for each time snapshot) then you could still use the split command as follows

split -l 1002 BigFile.txt chunks

The 1002 includes the two additional lines: the time stamp and the blank line after the time snapshot.

You can also use awk instead, and use the fact that the "#" tag demarcates records

awk 'BEGIN {c=0} /#/{next; c++} {print > c ".dat"}' BigFile.txt

would do something very similar. It would match the "#" tag and create files 0.dat etc. containing the different time-stamps. The advantage of this method is that you have more flexibility in naming your chopped pieces, and you don't have to know the value of "N" before-hand.

Finally, say you wanted to create chopped pieces in a different way. Instead of chopping up timestamps, you wanted to store the trajectories of individual particles in separate files. So while the methods above would have created 500 files with 1000 (+2) lines each, you now want to create 1000 files with 500 lines. One of the easiest ways is to use sed.

sed -n 1~10p  prints every tenth line starting with the "1"st line. You can use this to write a simple shell script.

npart=1000;
ndiff=$((npart + 2))
n=1;
while [ $n -le $npart ]
do
  nstart=$((n+1))
  sed -n $nstart ~ $ndiff'p' rcm > $n.dat
  n=$((n + 1))
done

Note the single quotes around "p" in the line containing the sed command.

Linux: Forcing cp to overwrite

As a precaution, I have the following three lines in my .bashrc file.

# SAFETY ALIASES
       
   alias rm="rm -i"
   alias mv="mv -i"
   alias cp="cp -i"

When I try to move or copy something onto a file that already exists it gives me a warning prompt. So far, so good.

Sometimes, I intentionally want to overwrite a bunch of files. With mv, I just say something to the effect of

mv -f dir1/*.dat .

to move all the *.dat files from dir1 into the current working directory. Unfortunately cp -f dir1/*.dat . does not work. A trick is to use the command "yes".

So yes | cp -f dir1/*.dat . seems to fix the problem.

Tokenize bash variables

Here is a useful set of bash features that I had to use today.

From the source linked above:

Given, foo=/tmp/my.dir/filename.tar.gz

We can use bash expressions to tokenize or extract different portions of the variable.

path = ${foo%/*} (/tmp/my.dir)

file = ${foo##*/} (filename.tar.gz)

base = ${file%%.*} (filename)

ext = ${file#*.} (tar.gz)

This gives us four combinations for trimming patterns off the beginning or end of a string:

${variable%pattern}: Trim the shortest match from the end

${variable##pattern}: Trim the longest match from the beginning

${variable%%pattern}: Trim the longest match from the end

${variable#pattern}: Trim the shortest match from the beginning

Detaching from terminals using GNU screen

GNU screen is a Linux utility, that I did not know much about it, until a couple of years ago. It is a very handy program, and I'll start with my most frequent use of the utility.

If you use Linux, you like terminals. Let's say you start a program like Firefox or emacs from the terminal, by issuing a command like

$ emacs &

The program runs in the background and let you do other work in the terminal. However, when you exit the terminal, by either typing exit or closing the terminal window, your program (emacs in this case) is also killed.

GNU screen allows you to detach the program from the particular terminal from which it was invoked, thereby letting the program persist beyond the life-time of the terminal.

It is very simple to use. Fire your program from a terminal, in background mode.

$ emacs &

Then invoke screen (preinstalled on pretty much all Unix like systems)

$ screen

You may see that the title of your terminal window has changed somewhat, but that is not important.

You now type Ctrl-A d. You should see something like [detached] echoed on your screen.

Now go ahead, and kill your terminal.

Your emacs window persists!

Magic!

There are several other good uses for screen. You could fire programs like Thunderbird, Firefox, emacs etc. from a terminal and leave them running for months. If you've logged into a remote server, screen can function as a multiplexer (multiple tabs).

There are a number of places to learn more about screen. Here are some of them.

Install LAMMPS with FFTW on your Desktop

Earlier I wrote about how to install LAMMPS and AtomEye on a Desktop without FFTW.

The following document now shows how to download, compile and build the freely available fftw library with LAMMPS to consider electrostatic effects.

Building LAMMPS with FFTW

Linux/Unix is Sleazy!

Rediscovered this on the internet!

$ unzip

$ strip

$ touch

$ finger

$ mount

$ fsck

$ more

$ yes

$ unmount

$ sleep

Hilarious!

Combining data from independent simulation runs using a bash script

Today I came across a problem that I have solved several times before. From my simulations, I generate a bunch of files called stat1, stat2, ... statN, which contain the following data:

$cat stat1
567.20 0.88
45.29 3.08
296.58 21.50
0.33 0.14

The first column are some properties in a particular simulation run, and second column is the standard error. The "N" different "stat" files are N independent simulation runs. When I finally report, I like to report the average properties and associated standard errors. The following shell script DataAgg.sh creates a new file TotalProp which contains exactly that.

$cat TotalProp
567.49 0.24
43.57 0.45
289.91 1.61
0.67 0.10
The shell script is here:

$cat DataAgg.sh

i=0
for s in stat*
do

let i=i+1

if [ $i == 1 ]; then
    awk '{print $1}' $s > TmpProp
    awk '{print $2*$2}' $s > TmpErr2Prop
else
    awk '{print $1}' $s > tmp
    paste tmp TmpProp > more
    awk '{print $1+$2}' more > TmpProp

    awk '{print $2}' $s > tmp
    paste tmp TmpErr2Prop > more
    awk '{print $1+$2}' more > TmpErr2Prop
fi
done

awk '{print $1/n}' n=$i TmpProp > more; mv more TmpProp
awk '{print sqrt($1)/n}' n=$i TmpErr2Prop > more; mv more TmpErr2Prop
paste TmpProp TmpErr2Prop > more
awk '{printf("%6.2f\t%6.2f\n",$1, $2)}' more > TotalProp

rm -f TmpProp
rm -f TmpErr2Prop
rm -f more
rm -f tmp

Note I don't need to know how many "stat"s there are, and how many rows each of the "stat"s has. The only precondition is that I know what the common prefix ("stat") of my datafiles is, and that those files contain only the two numerical columns mentioned above.

Copying entire directory

In Unix/Linux, it is easy to copy an entire directory with the standard program "cp"

cp -r dir1 dir2

creates a copy of directory with all it directories.

Computing and Me

My first brush with a computer was in the summer of 1986, when my dad bought me a ZX Spectrum with 48Kb RAM (wikipedia). The first six months or so, I spent endless afternoons being wowed by games such as Dynamite Dan and Commando, until I realized that more could be done.

The computer had a primitive BASIC language compiler, and slowly at first, and later with great enthusiasm (like the one that characterizes kids who've just learnt to bicycle), I began writing programs, and that was when I essentially fell in love with the machine. My grades at school suffered initially, but if I can make any claims to being somewhat logical, then that slice of history ('86-'92) had a lot to do with it. It was a great time.

Later at IIT Bombay (95-99) I was introduced to this thing called Unix (which atleast then was the dominant if not only OS on campus), and later during my senior undergrad thesis to Linux. I remember the fun and suffering as the gurus (more "keedas", really) unleashed "write"s to remote machines with ominous messages, popping up "xeyes" on unsuspecting classmates surfing the web for naughty stuff on lynx :). It was my introduction to the networked environment, and I loved it.

During graduate school at Michigan, we were forced in large part to use Apples, which were the only computers with departmental support. Although things did get better after OS X, it wasn't until in 2004, that I switched back to Linux, pretty much full time. Now about 5 years thence, all my machines run some kind of Linux, including the Ubuntu laptop on which I am doing my writing right now.

I am immensely impressed with what I can do, both personally and professionally using extremely high quality and free software (programs on my University webpage). I routinely use Gnuplot, Octave, OpenOffice, GIMP, Maxima (occasionally), awk, JabRef, LaTeX , which run gracefully under Linux. Other things that I like are the (i) ability to automate routine tasks with shell scripts (ii) the ability to schedule jobs and control their priority from anywhere, and (iii) multiple Desktops, which I cannot live without anymore.

There is much more I have to say about this topic, but I'll come back to it later.