Hints and tips on bash

These are some helpful hints on various topics.

apache bash cmd database java javascript linux mod_rewrite mssql mysql oracle os perl php regex rsync ssh unix web_design windows

 

Search and replace in bash scripting

To get the value of variable X with all occurances of 'yy' replaced by 'zz' use the following expression:

${X//yy/zz}

For example:

ORIG="foo to the bar"
REPLACED=${ORIG//bar/baz}
echo $REPLACED

The result is:
foo to the baz

Argument list too long, or how to use xargs

Suppose you want to remove all jpeg files from several directories. You use
rm -f `find . -name *.jpg -type f`
and get "Argument list too long". What you have to do is to use xargs to automatically split the long list to several shorter and pass them to rm:
find . -name *.jpg -type f | xargs rm -f
This way several rm commands are executed, but your files are finally gone.

Creating files with fixed length and random content

Handy command to create file with fixed length and random content is  dd. Some examples:

File test1.bin, 10k size, containing only zero bytes: 
 dd bs=1024 count=10 if=/dev/zero of=test1.bin

File test2.bin, 100k size and random content:
 dd bs=1024 count=100 if=/dev/random of=test2.bin

Above example is rather slow, especially for large files, so here's another way to generate pseudo-random file:
 dd bs=1024 count=100 if=/dev/hda of=test2.bin skip=1000
This will copy 100k from the hard drive 1М after the beginning.

Getting the last word of bash string

To get the last word of a bash string use awk '{print $NF}':

echo "one two three" | awk '{print $NF}'

returns

three

One-liner for transfering SSH public key to remote host

cat ~/.ssh/id_rsa.pub | ssh hostname "cat >> ~/.ssh/authorized_keys"

Rsync over ssh with non-default port

If you use rsync over ssh to a server running on port different from 22, for example 555, use the following format:

rsync -a -e "ssh -p 555" rsyncuser@remoteserver:/data/to/sync /archive/

More on rsync

Prefix each output line with timestamp

Imagine the case: you were writing a program or a script that outputs some information in the console. Just when everything is almost complete you realize that you actually need timestamp on each message output. For example you need to know how long it takes to backup each of your users' home dirs. The solution is to go through all output commands and wrap them in a function that appends timestamps. Believe it or not, there's a lazier way to do this - it's a simple bash one-liner:
 
> ./your_program | while read line; do stamp=`date`; echo "$stamp $line"; done
 
You can try it with
 
> du -sc /home | while read line; do stamp=`date`; echo "$stamp $line"; done