not-so-quick guide to bash shell


1. for beginners 2. getting into the details 3. bash scripts

1. for beginners

On Macs, launch Bash with Terminal, located in the Applications/Utilities folder.

Some easy one-liners to begin with

Use Ctrl-C to quit current command

bash$ echo Hello world!
Hello world!
bash$ look zoon #print words starting with 'zoon'
zoon
zoonal
zoonerythrin
zoonic
 ...
bash$ tr a-z A-Z #turns input to upper case
hello
HELLO
bash$ rev #writes input backwards
hola
aloh
bash$ echo a* #print names of all files in current directory starting with 'a'

To make a quick text file:

bash$ cat > quick.txt
this is 
my quick file 
#press ^D (Ctrl-D) to end text entry
bash$ cat quick.txt
this is
my quick file

Basic shell navigation, commands, features

pwd = print working directory (your current location)
cd directory = move into that directory
ls = list contents of current directory
echo = print something onscreen- a string, variable, filename etc etc
cat myfile = display contents of a text file
open . = open current directory in Finder
Ctrl-A = go to start of line
Ctrl-E = go to end of line
Completion: Press [TAB] while typing a command/filename to auto-complete it; it beeps if ambiguous – press [TAB] again and the alternatives are shown.
Drag n drop: For commands using/requiring files/folders with their full path, drag and drop the file/folder onto the command line. e.g. type "cd " then drag a folder from Finder to make that the current directory.
Permissions: If you aren’t permitted to execute a command, put sudo (super-user do) at the start of the command, which will ask for the admin password before continuing.
Inspector: [CMD]-I = launch the Terminal Inspector to change settings for shell fonts, colours, cursor, etc.

history = display shell history – all commands you’ve run
and = scroll through previously used commands
Ctrl-R = search history. Ctrl-R, type one or more letters, then Ctrl-R again repeatedly to step back through your history.
history n = display last n commands

Help!

Type help, or to see all available commands, hold ESC until prompted, then answer y.
man command = display manual pages on a command.
Multi-page results: Navigate the manual pages, the output of less etc a page at a time with : SPACE = see next page, b = back one page, q = quit back to terminal prompt, /string = search – highlights occurrences of string. Cycle forwards through the highlighted strings with n, or backwards with N.
info bash = many pages of info about bash. [SPACE] = next page, B = back, U = Up a level, N = Next, P = Prev. Move cursor to any section title and press ENTER to go to that section.
man bash = exhaustive bash documentation
man man = display more info on the man command
Help: help command. Some commands also have short help messages, use command -h
help * = help on all builtin commands – very useful to read!
help = nice 1-page summary of builtin commands + options
apropos blang or man -k blang = search manual pages for blang.
type blang = searches the environment for matching commands & displays them. use option -a to display all matches, not just the first.
locate blang = find a file or command

Controlling input/output

Pipes: A concept central to Unix. Input can be taken from, and output to, files or other commands, with pipelines |, which make the output of one command the input of the next. E.g. cat * | wc -l counts the total number of lines in all files in current directory – this takes the output of cat and feeds it to wc. The idea is to use a range of simple tools together, rather than try to do everything with one program.
Less: Use | less at the end of a command line to display results one page at a time. Or less myfile to display a text file. q=quit, SPACE=next page, ENTER=next line, b=prev page, /=search mode: Type /string to find string in the less output, and n to cycle forward, N to cycle back, through occurrences of string.

Quotes: These are very important in bash, and often used to protect strings from the various expansions the shell performs on every input line.
'single quotes' = treat all contents literally, i.e. ignore the special character meanings in bash.
"double quotes" = treat everything literally except $ (variable/arithmetic substitution) and command substitution.

bash$ a=5;echo "$a" '$a'
5 $a

`backticks` = command substitution – evaluate contents as a shell command. NB Backticks are deprecated; use $() instead always.
$'' = Ansi-C quoting. Like single quotes, but evaluates backslash-escaped characters, octal, hex, and Unicode.

Escaping: use backslash \ before a special character to use it literally, e.g. before a space in a file/folder name.

Input/output redirection:
command < infile = command reads input from a file, not from standard (keyboard) input
command > outfile = write output to a file
command >> outfile = append output to a file

File/directory commands

cd = return to home directory
cd – = return to previous directory
cd d = enter subdirectory d
cd .. = go up to parent directory
ls -1 ~/Documents = list files in the /Documents directory, one per line
ls D* = list files in subdirectories beginning with D.
ls | wc -l = displays the number of files in current directory
ls *20.jpg = list all files with filenames ending in ..20.jpg
ls /Applications = lists contents of Applications folder
ls ~/Documents | wc -l = displays num of items in Documents folder
ls -d .* = show all hidden files in current directory.
Other ls options: -a = include hidden files, -h = human-friendly: shows file sizes in MB, GB etc, -l = show more info, -S = sort by file size, -R = include subdirectories

df -hl = displays space used/available on hard disk and all available drives
open filename = open a file in its default application
file myfile.sxj = displays info on what type of file it is

find / -type f -name myfile -print = find the file myfile
find / -type d -print = display all directory names
find / -type f -name \*.txt -print = display all filenames ending in .txt
find / -size +20000000c = find files bigger than 20MB
++ -type f = files, d = directories, l = symlinks
++ -name filename
++ -name “*.txt” = All .txt files. Glob characters like “*” must be quoted so the shell doesn’t expand them before find gets them.
++-mtime -7 = All files modified less than 7 days ago.

mkdir – create a new folder/directory
cp – copy files/folders
mv – move/rename files/folders
rm – remove (delete) files/folders. NB Be careful with this!

Shell pattern matching (Globbing)

Shell pattern matching occurs with: filename globbing (pathname expansion), [[ = and != ]], case statements, help, string parameter operators (e.g. ${var#word}, etc. Regular expressions are only used within bash by [[ string =~ regex ]].
ALWAYS use quotes around arguments to simple commands that could be interpreted as globs.

* = Match any string of zero or more characters.
? = Match any single character.
[abc…] = Match any one of the enclosed characters; a hyphen can specify a range (e.g., a-z, A-Z, 0–9).
[!abc…] = Match any character not enclosed as above.
~ = Home directory of the current user.
~name Home directory of user name.
~+ = Current working directory ($PWD).
~- = Previous working directory ($OLDPWD).

echo ? = print list of all one-letter files in current directory
echo ????? = all files whose name has 5 letters
ls * = list contents of directories (and subdirectories)
echo * = print names of files and directories

Image commands

sips -Z 1280 *.* = resize all image files in the current directory so the width/height (whichever is larger) isn’t greater than 1280 pixels.
sips -s format png input.jpg ––out output.png = converts a jpeg image to png. (-s = set a property value.)
Other sips options:
++-s formatOptions low/high/best/percent = set image quality.
++-z height width = resize to height x width pixels (doesn’t preserve aspect ratio!)
++-r angle = rotate clockwise by angle degrees.
++-f horizontal/vertical = flip
++––resampleWidth width = resizes so width is width pixels.
++––resampleHeight height = resizes so height is height pixels.
sips -s formatOptions high ––resampleHeight 670 *.* = resizes every image in the current directory to have height 670 pixels (preserving aspect ratio) with high quality.
for a in *.jpg; do sips -s format png $a ––out ${a%.*}.png ; done = convert every jpeg image in the current directory to png.

Internet commands

ifconfig = display network information

curl -O http://www.somewebsite.com/files/file.mp4 = download a file to current directory
curl -L -o myfile.dmg 'http://www.somewebsite.com/files/getdmg?id=24' = download file to current directory and rename. -L allows curl to follow any redirects. Putting the URL in single quotes allows non-alphanumeric characters to be written as is.

host URL/IPaddress = look up IP address/host name e.g. host blang.com
Use host -a blang.com to display more information.
whois blang.com = displays info about who owns the domain name blang.com
wget – download multiple web pages/files
wget -r -A .pdf http://url-to-webpage-with-pdfs/ = download all pdfs from a website.

Other commands

date

bash$ date "+%a %-d %b %Y"
Wed 1 Mar 2017
bash$ date +%-d/%-m/%y     # day, month, year
1/3/17

rs – reshape a data array

jot – print sequential or random data

bash$ jot -r 25 | rs 5 #print 5x5 array of random integers between 0-100
50  75  95  99  97
5   49  55  86  46
79  86  79  2   14
45  42  75  98  19
12  32  33  78  93
bash$ jot -s ''  -c 26 z a
zyxwvutsrqponmlkjihgfedcba

csplit – split a file into multiple files, based on regexes, number of lines etc.

2. getting into the details

More file/directory commands

stat /path/filename = displays info about the file
strings filename = displays text strings from a binary file
xxd filename = hex dump of a file
stat -x
myfile = display important information about a file
du -h = size of current directory/files, in human-readable units. (disk usage)
find * -size +5000000c = show files larger than 5,000,000 bytes in the current directory & subdirectories (in newer bash versions, -size +5M works)
ls -l | awk '$5/1000000>5' = show info for files larger than 5MB
od -c – display files/stream with backslash-style escape characters.

bash$ echo "Hi there! How's things?" | od -c
0000000    H   i       t   h   e   r   e   !       H   o   w   '   s    
0000020    t   h   i   n   g   s   ?  \n                                
0000030

More shell navigation commands

Double (left) mouse click = select word
Triple (left) mouse click = select line

[TAB][TAB] = list all available commands
string[TAB][TAB] = list all available commands starting with string
/[TAB][TAB] = list subdirectories
$[TAB][TAB] = list system variables
Ctrl-rr = use previous history search string
Ctrl-k = delete from cursor to end of line
Ctrl-u = delete from cursor to start of line
Ctrl-w = delete word (from cursor to the previous space)
Ctrl-– = undo last editing command
Ctrl-l = clear screen, put current line at top of screen
Ctrl-t = transpose (swap) 2 characters
Ctrl-y = paste deleted text at cursor (yank)
Ctrl-] = reads next keypress and moves cursor forwards to that character
Ctrl-xx = toggle between the start of line and current cursor position
Ctrl-v – literal next. Used for entering special characters into arguments
e.g. to change a space to a TAB character with sed: (for the TAB, press Ctrl-v then TAB)

bash$ echo t u v w x | sed 's/ /        /g'
t       u       v       w       x
META commands: These may work instead with the ALT/META key on your system. Otherwise Esc = METAfy the next key pressed.
Esc f = move forward 1 word
Esc b = move back 1 word
Esc t = transpose (swap) 2 words
Esc d = delete from cursor to end of word
Esc BACKSPACE = delete from cursor to start of word (Esc DEL on some systems)
Esc y = yank-pop (paste a previously deleted item with each press)
Esc Ctrl-] = reads next keypress and moves cursor backwards to that character
Esc Ctrl-e = expand the line as the shell does: alias, history, word expansions etc
Esc . = paste last word of previous command
Esc u = make every character upper case, from the cursor to the end of the current word.
Esc l = make every character lower case, from the cursor to the end of the current word.
Esc c = Capitalize the character under the cursor and move to the end of the word.
Esc ? = List all possible completions (e.g. on empty line, will display all commands)
Esc * = Insert all possible completions
Esc { = Put all filename completions, comma-separated, between { }.

There are many more, and they’re customizable. See GNU bash manual
See a list of your current shell key bindings with:

bind -p | grep -Ev 'not bound|self-|do-' | sed 's/"//g' | less

More shell history commands

!! = Repeat previous command (can be used as a part of a line or script)
!-2 = 2nd last command
!-2$ = last argument from 2nd last command
!n = Repeat command number n in the history
!$ = the last parameter from the previous command
!* = all parameters from the previous command
!blang = run the most recent command that started with blang
!blang:p = print out the command that !blang would run and add it to the command history
!$ = last word of the prev command
!$:p = print out that word
!string = Execute the last command that begins with string
^str – removes ‘str‘ from the previous command and executes
^str1^ str2 = substitutes str1 for str2 in the previous command and executes:

bash$ ecco hello
-bash: ecco: command not found
bash$ ^cc^ch
echo hello
hello

Operations on text/text files

echo -e = enable backslash-escaped characters like \n and \t. e.g.:

echo -e Hi "\t" There "\n\n\n"
cat filename = displays the file
cat file1 file2 file3 > fileall = adds (concatenates) the files together and saves as fileall.

diff -y file1 file2 = displays all differences between 2 similar files, -y means ‘format in 2 columns’. Also can be used to display the differences between 2 similar directories.
vimdiff file1 file2 [file3] = colourful visual display of the differences between 2 or 3 text files.

wc -l filename = displays num of lines in the file
wc "My Stuff" = count lines, words, bytes. (Quotes needed if filename/directory has spaces.)
head myfile = display first 10 lines of myfile
head * | less = display first 10 lines of all files in current directory, a page at a time
tail myfile = display last 10 lines of myfile
tail +3 myfile = displays myfile, starting with the 3rd line.
grep (Global Regular Expression Print) finds a particular pattern of characters in files, with many options for different searches. See regex/grep/sed page.
grep -ic but article.txt = displays the number of lines containing ‘but’ in article.txt.
++-i = ignore (upper/lower) case
++-c = display only the number of lines, not the lines themselves.
++-n = show line number where match was found.
++-o = show only matching part. This can be used to find the number of matches, e.g.:

bash$ echo '12ag 25r4 qw 6z' | grep -o [0-9] | wc -l
6

++-R = recursive search, through all sub-directories of current folder.
++-e = search multiple patterns i.e. grep -e pattern1 -e pattern2

bash$ grep ^[^aeiou]*a[^aeiou]*e[^aeiou]*i[^aeiou]*o[^aeiou]*u[^aeiou]*$ /usr/share/dict/words |
> tr "\n" " "     # Find words containing all the vowels in order
abstemious abstemiously abstentious acheilous acheirous acleistous affectious annelidous arsenious arterious bacterious caesious facetious facetiously fracedinous majestious

This regex (regular expression) means: Match a word beginning with 0 or more non-vowels, followed by an ‘a’, then 0 or more non-vowels, then an ‘e’, then etc. That also could be written:

grep $(printf "%s" {^,a,e,i,o,u}"[^aeiou]*")$ /usr/share/dict/words |
> tr "\n" " " #or:
c=[^aeiou];grep ^$c*a$c*e$c*i$c*o$c*u$c*$  /usr/share/dict/words | tr "\n" " "
sed – stream editor. See my regex/grep/sed page.

sed 's/this//g' #delete all occurrences of 'this'
sed 's/this/that/g' < myfile.txt #replaces all occurrences of 'this' with 'that'
textutil -cat rtf -output combined.rtf *.doc = combine all Word documents in the current directory into a single rich-text document called combined.rtf
textutil -convert html foo.rtf = converts foo.rtf into foo.html.
textutil -convert rtf -font Times -fontsize 10 foo.txt = converts foo.txt into foo.rtf, using Times 10 for the font.
textutil -cat html -title "Several Files" -output index.html *.rtf = loads all RTF files in the current directory, concatenates their contents, and writes the result out as index.html with the HTML title set to “Several Files”.
textutil -convert html myfile.rtf; open myfile.html = convert specified rich text file to HTML, save as myfile.html and display page in default browser
(format can be one of: txt, html, rtf, rtfd, doc, wordml, or webarchive)
cut – cuts columns of text from files. Specify the character or field positions. -d = Specify the field delimiter character. -f = which fields to select. Reads from a file or read from std input with e.g.
cut -d ' ' -f 2 myfile = print the 2nd word from each line of myfile.

paste – pastes columns of text together, side-by-side

xargs – to format input to be used as command arguments.
Can also be used to format text, like this:

bash$ echo "1 2 3 4 5 6
> 7 8 9 10
> 11 12" > example.txt
bash$ cat example.txt | xargs
1 2 3 4 5 6 7 8 9 10 11 12
bash$ cat example.txt | xargs -n 3
1 2 3
4 5 6
7 8 9
10 11 12
bash$ echo "splitXsplitXsplitXsplit" | xargs -d X 
split split split split
bash$ echo "splitXsplitXsplitXsplit" | xargs -d X -n 2 
split split 
split split 
bash$ cat example.txt | xargs -n 1 -I {} echo -n "{}, "
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 

The arbitrary character(s) after -I (here “{}”) indicate to xargs the string to be used for substitution.

tr – translates a set of characters into another
tr 'a-z' 'A-Z' < myfile = changes all letters to capitals
tr ' ' '\n' < myfile.txt = turns text into a single column of words.

bash$ tr ' ' '\t' < example.txt       #Convert spaces to tabs
1        2        3        4        5       6
7        8        9        10
11        12

tr -s char = squeeze characters (delete repetitions)

bash$ echo 'Lots       of    spaces' | tr -s ' '
Lots of spaces

tr -d = delete characters from a set

bash$ cat example.txt | tr -d 1
 2 3 4 5 6
7 8 9 0
 2

tr -c set = complement character set (everything except the characters included)
tr -dc set = Delete characters not in a set

bash$ echo hello 1 char 2 next 4 | tr -dc '0-9 \n' 
 1  2  4
bash$ echo 1456243526545342 | tr 123456 EADGBE
 EGBEAGDBAEBGBDGA
bash$ echo 'This is ROT-13 encrypted.' | tr 'A-Za-z' 'N-ZA-Mn-za-m'
Guvf vf EBG-13 rapelcgrq.

tr -dc [:print:] < myfile1 > myfile2 = remove non-printable characters

sort lines
sort -u myfile = display the file with lines sorted in alphabetical order, removing duplicate lines
sort myfile | uniq -c = sorts the file into alphabetical order, counts (and display number of) duplicate lines, and removes them.
++-n = numeric order
++-r = reverse order
++-M = by month – Jan > Feb etc
++-f = fold lower into upper case (i.e. sorting ignores case)

uniq – report or omit repeated lines
++-c = precede each line with N, the number of times it was repeated
++-d = don’t output lines that aren’t repeated
++-u = don’t output repeated lines
++-f N = ignore the first N fields
++-s N = ignore the first N characters
awk – stream editor expanded into a programming language. See my AWK page.)

awk '{print $2, $4}' myfile # print the 2nd and 4th word from each line.
awk 'length < 60' myfile # print all lines shorter than 60 characters.
awk 'NR>=3{print} NR==7{exit}' myfile.txt # print from the 3rd to the 7th line.

Note on quoting

bash$ echo 'Don\'t quote me' # This doesn't work.

Bruce Barnett says “I bet many of you programmers are confused by this. All of us are very familiar with strings in programming languages like C. This is why we get confused. [In bash] the quotes turn substitution on and off. They are not used to indicate the starting and ending of a string.”

bash$ echo 'Don'\''t do that' # This works.

e.g. shell script printcol:

#!/bin/bash
# printcol
# e.g. printcol 3
awk '{print $'$1'}' 

Notice how the 1st command-line argument $1 isn’t single-quoted, but is expanded by the shell, to become in the awk program $n, meaning in awkese the nth word/field on the input line.

More on redirecting input/output

<< label = Here-document. Accepts multi-line input from a file or keyboard, which starts and ends with an arbitrary string label.
● Use brackets to redirect the output from a group of commands. This actually runs the commands in a subshell. e.g. (pwd; ls; cd ../; pwd; ls) > outputfile
< <( ) = process substitution e.g. for reading lines from output of commands
e.g. To make the shell scroll down, with command line at top:

(while read line;do tput home;tput ri;echo "$line";done;tput home;tput ri)< <(ls -l)

echo “$(<myfile)” works like cat myfile.

Other commands

defaults write com.apple.finder AppleShowAllFiles TRUE ; killall Finder = show hidden files in Finder
defaults write com.apple.finder AppleShowAllFiles FALSE ; killall Finder = don’t show hidden files in Finder (the default)
export PS1="\w" = changes the prompt to the full path of the current directory.
++\d = current date, \t = current time, \h = host name, \# = command number, \u = user name, \W = current (working) directory
last -l $USER = displays how long you’ve been logged in
cal = displays calendar for the current month
cal 1967 = displays calendar for 1967.
echo $HOME = display the value of HOME

alias ll='ls -l' = make a shorthand alias ll, to save typing. This defines a new command ll that runs ls -l
unalias
x = removes alias x
alias = list all aliases

Doing maths:

bc – basic calculator – actually a whole calculator language, with syntax similar to C.
++scale=num = set the number of decimal places in answers. quit = exit.

bash$ var1=$(echo "scale=4; 3.44 / 5" | bc);echo The answer is $var1
.6880

Let is the old way, (( )) the new.

let Y=(X+2)*10 X=Y+1 # no spaces allowed within 'words'
Y=$(( ( X +2 ) * 10, X = Y+1 )) # any spacing is allowed
bash$ g=5
bash$ ((g+=2));echo $g
7
bash$ ((g++));echo $g
8

If the value of the expression is non-zero, the return status is 0; otherwise the return status is 1:

bash$ ((1));echo $?
0
bash$ ((0));echo $?
1

Or use declare:

bash$ declare -i n #declare the variable n as an integer type
bash$ n=6/4; echo $n
1

Two ways of doing floating point maths easily on the bash command line
Put these functions in your .bash_profile:

calc() { awk "BEGIN{print $*}"; } # awk calc
pc() #python calc
{ python -c "print($*)"; }
#then:
bash$ calc 2.4^3.1
15.0888
bash$ pc 2.4**3.1
15.0888051157

Dozens more ways here.

Variables

set = display all variables (system or user-defined) and their values

Ways to assign a value to a variable:
1. explicit definition: var=value – no whitespace allowed!
2. read from keyboard: read var. e.g.

bash$ echo -n "Enter name: " ; read name ; echo Name is $name
Enter name: Snurgel
Name is Snurgel

3. read from a file:

bash$ read message < quick.txt;echo $message
this is

To read and display the entire file:

bash$ while read message; do echo $message;done < quick.txt
this is
my quick file

4. command substitution – run the command and use the output as the value to be assigned.

bash$ d=$(date) #old style: d=`date`

5. read from a stream – process substitution

bash$ read etad < <(date | rev) ; echo $etad
7102 TSE 33:52:61 5  beF nuS

6. here string – to be able to return variables from subshells

bash$ read a b c <<< $(echo "d e f" | awk '{ print $3, $2, $1 }')
bash$ echo $a $b $c
f e d

ls -l | while read a b c d; do echo owner is $c; done

typeset = declare local variables in a function
typeset -i x = declare local integer variable x
Setting variables with typeset makes them much faster. [test this]
declare -p = prints all parameters, with appropriate set instructions
declare -l mystring = string automatically converted to lower case
declare -u mystring = string automatically converted to upper case
declare -r myvar = creates read-only variable
declare -a myarray = indexed array (indexed by integers)
declare -A myarray = associative array (indexed by strings)

Environment variables

$* = all the command line parameters as a single text value
$@ = all the command line parameters as separate text values
$# = the number of command line parameters
$? = the exit status of the most recently used foreground process
$- = the current command line option flags
$$ = The process ID of the current shell
$! = the process ID of the most recently executed background process
$0 = the name of the command from the command line
$_ = the absolute pathname of the shell

bash$ set one two three four # set some fake positional parameters
bash$ echo ${!#} # get the last ('nth', not 'previous') argument
four

(# stores the number of arguments, so !# will reference the last argument)

for v in $(compgen -v);do echo $v=${!v};done # Print environment variables

String variable operators

bash$ string=abcdefghijklmnopqrstuvwxyz 
bash$ echo ${#string} = print the length of the variable/string
26
bash$ echo ${string:4} = substring from character 4 to end of string
efghijklmnopqrstuvwxyz 
bash$ echo ${string:4:8} = substring from char 4 with length 8 
efghijkl 
bash$ echo ${string:(-1)} = substring from last character to the end 
z 
bash$ echo ${string:(-2):2} = substring from 2nd last char, length 2. 
yz 
bash$ var="This is a line of text" 
bash$ echo ${var/line/REPLACED}    #Replacing some text in a variable
This is a REPLACED of text

${name#pattern} = remove (shortest) front-anchored pattern

bash$ f=/home/my/path/thing.txt
bash$ echo ${f#*.}   # get extension - remove pattern from beginning of string
txt

${name##pattern} = remove (longest) front-anchored pattern

bash$ n=${f##*/};echo "$n"   # Get filename
thing.txt

${name%pattern} = remove (shortest) rear-anchored pattern – starting at the end of the string, remove the shortest string containing the pattern

bash$ echo ${n%.*}   #remove filename extension
thing
bash$ echo ${f%/*}     #Get directory path
/home/my/path/
for f in *.foo; do mv $f ${f%foo}bar; done # Rename *.foo to *.bar

${name%%pattern} = remove (longest) rear-anchored pattern
${name/pattern/string} = replace the first occurrence of pattern with string
${name//pattern/string} = replace all occurrences

● These string operations using the ${name … } notation can also be applied to all string elements in an array, with the ${name[@]} or ${name[*]} notation.
Indirection
${!param} = the referenced parameter is not param itself, but the parameter whose name is stored as the value of param, e.g.

bash$ read -rep 'Which variable to inspect? ' v; \
> printf 'The value of "%s" is: "%s"\n' "$v" "${!v}" 

Arrays

bash$ a[6]=iurhi  # set an element
bash$ a[blah]=4  # associative arrays are allowed, i.e. the index can be a string
bash$ a=(z 1 2 3 haha hi "&^%")   # set an array. NB the first element is a[0].
bash$ echo ${a[@]}   # expands to all array elements.
z 1 2 3 haha hi &^%
bash$ echo ${a[6]} 
&^%
bash$ echo ${#a[@]}     # number of elements
7
bash$ echo ${a[@]:2:3}   # slicing - 3 elements starting at a[2]
2 3 haha
bash$ echo ${#a[4]}   # gives the length of a[4], i.e."haha"
4
bash$ echo "${a[@]%a}"   # substring removal - remove trailing "a"
z 1 2 3 hah hi &^%
bash$ echo "${a[@]#h}"   # remove leading "h"
z 1 2 3 aha i &^%
bash$ echo "${a[@]/h/j}"   # replace first "h" in each word with "j"
z 1 2 3 jaha ji &^%
bash$ echo "${a[@]//h/j}"   # replace all occurrences
z 1 2 3 jaja ji &^%

myfiles=(*.txt) = Create an array myfiles that contains the names of all .txt files in the current directory.
echo ${myfiles[@]} = print the members of that array
for f in ${myfiles[@]};do echo $f;done = print the members of that array, 1 per line
read -ra myarray = Chop a line into fields and store the fields in myarray. $IFS=delimiter

bash$ c=(0 1 1 1 0 1 1 1) #set array
bash$ q=([1]=2 [4]=7 [5]=8) # another way
bash$ echo ${#q[@]}
3
bash$ echo ${q[@]}
2 7 8
bash$ echo ${!q[@]} # ${!name[@]} expands to the array indices
1 4 5
bash$ files=(*) # make array of names of files in current dir
bash$ c=(${b[@]}) #copy one array into another

myarray+=(newmember) = add to array
array=( “${array[@]// /_}” ) = Recreate array with spaces in elements as underscores
array=( “${array[@]:2:3}” ) = Recreate array only with elements from index 2 to 4

Brace expansion

bash$ echo test.{txt,jpg,png}
test.txt test.jpg test.png

mv info{,.old} = mv info info.old
for i in {1..$n} = WRONG – the Bash parser performs brace expansion before any other expansions or substitutions. Use a C-style for loop.

bash$ echo {1..20..2} #now has inc, since bash 4.0 (2009)
1 3 5 7 9 11 13 15 17 19
bash$ echo {009..012} # also leading zeros
009 010 011 012

● Single quotes inhibit all expansions—the characters enclosed by the quotes pass through the expansions unscathed.
● Double quotes permit some expansions and inhibit others. Word expansion and command, arithmetic, and process substitution take place—the double quotes only affect how the result is handled—but brace and tilde expansion do not.

Compound statements/flow control

Test/If-then statements

Use [[ to compare strings, files, (( for integers. (The only benefit of [ is compatibility with some other and ancient shells.)

Integer/integer variable tests:
if [[ n1 test n2 ]]; then do_if_true; [else do_if_false;] fi
e.g. if [[ $v -eq 5 ]]; then echo Yup; else echo Nope; fi
NB The spaces inside the square brackets are necessary.
-eq =++-ge \ges++-gt >++-le \les++-lt <++-ne \neq
String tests: =++ !=++ <++ >
[[ -n str1 ]] – checks if str1 has length > 0
[[ -z str1 ]] – checks if str1 has length 0.
NB The > and < symbols must be escaped (e.g. \>), or the shell uses them as redirection symbols, with the string values as filenames. Also, what is considered greater or less than is different from the order used by sort.
File tests:
[[ -e file ]] checks if file exists. Check if the file: -d = is a directory, -f = is a file, -r = is readable by you, -s = is not empty, -w = is writable by you, -x = is executable by you.
file1 -nt/-ot file2 – checks if file1 is newer/older than file2
[[ condition1 ]] && [[ condition2 ]] = AND
[[ condition1 ]] || [[ condition2 ]] = OR

For testing commands use if; for numerical tests use (( )).
Numeric: val++/–– = post-increment/-decrement, ++/––val = pre-increment/-decrement, ! = logical NOT, ~ = bitwise NOT, ** = exponentiation, <</>> = bitwise shift, & = bitwise AND, | = bitwise OR

if
if test-commands; then
++do_stuff;
[elif other-test-commands; then
++do_other_stuff;]
[else do_yet_other_stuff;]
fi

The form is : if list; then list; [ elif list; then list; ] [ else list; ] fi
Unlike the usual programming if-then, these are lists of commands, and the list in if list then takes the return value of the last command in list, e.g.

bash$ if ((x = 4 , x += 5 , x -= 3)); echo $x, 5;((x > 5 ))
> then echo Greater
> else echo Not greater
> fi
6, 5
Greater
for .. in
for var in list
do
++commands
done


for w in $(<myfile.txt) = loops over words
for f in *.txt = loops over all .txt files in current dir

bash$ for letter in a b c; do echo -n ..$letter; done
..a..b..c

If ‘in list’ is not present, the commands are executed once for each positional parameter, as if ‘in “$@”’ had been specified.

bash$ sum() { g=0;for f;do ((g=g+f));done;echo $g; }
bash$ sum 1 2 3
6

Using a command to generate a list:

for f in $(find . -name *.txt) # loops over all .txt files in dir and subdirs
for (( ; ; )) (C-style for loop)
for ((i=0;i<10;i++))
do
++echo -n $i", "
done

bash$ for ((i=0;i<10;i++)); do echo -n $i", "; done
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 

More complex for loop:

for (( i=0, j=0 ; i+j < 10 ; i++, j++ )) 
do 
    echo $((i*j)) 
done 

Floating point loop:

for fp in $(seq 1.0 .01 1.1) 
do 
    echo $fp 
done 

Or:

seq 1.0 .01 1.1 | \
while read fp 
do
    echo $fp 
done 
while
while test-commands; do stuff; done

Do stuff as long as test-commands has a 0 exit status.

until
until test-commands; do stuff; done

Do stuff as long as test-commands has a non-0 exit status.

case

You can put more than one pattern in a case using |.

case myvar in
pattern1 | pattern2) commands1;;
pattern3) commands2;;
*) default commands;;
esac


The patterns are file glob type patterns, not regexes. e.g. * always matches.

Aliases & functions

● Functions, unlike aliases, can take arguments.
● Putting alias definitions in single quotes means variables won’t be evaluated until the command is run. Inside double quotes, they’re evaluated immediately, and never change after that – which is probably not what you want.
type -a blang = print info on an alias/function/builtin
builtin blang = ignore shell functions and aliases to run the actual built-in command
command blang = ignore shell functions and aliases to run the actual external command
\blang = prevent alias expansion
declare -f = display all function definitions
declare -F = display all functions, name only
declare or set = display all shell parameters, aliases and functions

Sample functions

rep () # repeat - looper from Brian Fox
# e.g. rep 10 echo hello
{ 
    local count="$1" i;
    shift;
    for i in $(seq 1 "$count");
    do
        eval "$@";
    done
}

Int to char (via hex)

bash$ chr() { echo -ne "\x$(printf "%x" $1)"; }
bash$ chr 65
A

flocate – A useful function that locates files by filename. (locate is fast but often returns hundreds of entries with the search string in the path. Flocate searches only on the filename.)

flocate() { locate "*/$1"; }

words – prints 1 argument on each line – note that { } isn’t needed.

bash$ words()
> for word
> do
> echo "$word"
> done
bash$ words of love
of
love
bash$ declare -f words
words () 
{ 
    for word in "$@";
    do
        echo "$word";
    done
}

hr () # print a horizontal line the width of the window
{ 
    T=''
    read r r < <(stty size)
    while ((r-->0))
    do
        T=$T-
    done
    echo $T
}

Functions can have redirection on their first line, which then always applies, e.g. myfunc() { } < tmp

Processes & job control

ps -A = display all running processes
ps -axcro command,pcpu | head = shows the 10 top CPU-consuming processes
++-c = give just short names of processes
++-v = more columns of information

kill 123 = kill process number 123 (stop it instantly)
killall Blang = kill the application Blang
top = display running processes
bg = put job in background
jobs = list background jobs
fg = end background job in foreground
use nohup before a command to keep running it after the shell is closed: nohup command &

Exit Codes & Control Operators

Every command returns an exit code on completion, 0 if successful, some other number 1-255 if not, depending on the command.

Control operators:
command1 && command2 = if command1 returns 0 then run command2
command1 || command2 = if command1 doesn’t return 0 then run command2

Displaying text in colour

I have this in my bash profile, so I can always use these commands:

alias red='tput setaf 1'
alias green='tput setaf 2'
alias yellow='tput setaf 3'
alias blue='tput setaf 4'
alias magenta='tput setaf 5'
alias cyan='tput setaf 6'
alias white='tput setaf 7'
alias bred='tput setab 1'
alias bgreen='tput setab 2'
alias byellow='tput setab 3'
alias bblue='tput setab 4'
alias bmagenta='tput setab 5'
alias bcyan='tput setab 6'
alias bwhite='tput setab 7'
alias normal='tput sgr0' # back to normal text

Put this in scripts to return altered colours/fonts to normal in the event of a Ctrl-C quit:

trap control_c SIGINT
control_c() {
    tput sgr0
    exit 3 #for c=3...
}

Timing commands

time for ((i=0;i<5000;i++));do result=$(($i%2)); done = get times for 5000 loops, on my machine:
expression++++++++++++time
result=$(($i%2))++++++0.249s
result=`expr "$i%2"`+24.389s
let result=$i%2+++++++0.216s
result=$((i%2))+++++++0.141s
result=$[i%2]++++++++0.156s

Bash scripts

Use: #!/usr/bin/env bash as the shebang in all bash scripts. This finds the bash in the $PATH.
(#!/bin/sh or #!/bin/bash have been more common. #!/bin/sh/ is fine if bash isn’t needed, if bash features aren’t used.)
bash filename = load a shell script/program from a text file in the current directory, and run it in a new subshell.
Run a user-written script like a shell command (just by typing its name) by adding its directory to the command search path, and giving it execute permission with chmod +x scriptname. OR add to the .bash_profile file as a function – e.g. myfunc() { blahbla; blah; }
.bash_profile is read when Bash is invoked as a login shell.
.bashrc is executed every time a new shell starts.
aliases and functions should be in .bashrc, otherwise they’re not available to subshells. But DONT put things that will grow in .bashrc, e.g. PATH=$PATH:blah/blah/blah would be expanded with each new shell.
Running/executing vs sourcing a script
source script.sh or . script doesn’t start a new process, so variables can be shared e.g.

bash$ echo x=22 > setx.sh
bash$ chmod +x setx.sh
bash$ setx.sh
bash$ echo $x

bash$ source ./setx.sh # same as . ./setx.sh
bash$ echo $x
22

set = list all shell variables
env = list all exported variables, available to subshells – not many!
export -f myfunc = exports a function so subshells can see it
exit = exits current shell
Debugging
set -v = switch on verbose mode++++set -x = switch on echo mode
set +v = switch off++++++++++++++++set +x = switch off
Make xtrace (-x) more useful with source file + line number

export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'

Hmm this seems to give weird line numbers sometimes…
Dealing with errors, interrupts, exiting
Use || for error messages:
command || echo “command didnt work!”
trap -l = list trappable signals
To do stuff in a script after a Ctrl-c, before quitting:

trap 'echo INT signal received;exit' INT

${:?} – use for parameter errors, e.g.

DIR=${1:?"Error. Must give a directory."}
SRC=${2:?"Error. Must give a source file."}
GV=${3:?"Error. Must give a giblet vendor."}

Some sample scripts

Download something, if interrupted keep resuming until download is completed
I added this to my .bash_profile:

dl () # usage: dl myfilename "fileURL"
{ 
    until curl -kLC - -o "$1" "$2"; do
        :;
    done
}

Print the number of files with each extension, sorted

for i in *.*;do echo ${i#*.};done | awk '{a[$0]+=1} END{print NR" files :";
for (j in a) print a[j]"\t"j}' | sort -n -r

or this, which also gets the files with no extension:

ls -1 | awk '{split($0,e,".");a[e[2]]+=1} END{print NR" files :";
for (j in a) print a[j]"\t"j}' | sort -n -r


Strip html from a file
sed 's|<[^>]*>||g' myfile = remove the pattern “<, then 1 or more occurrences of any character but >, then >” anywhere that pattern’s found. Can’t just use <*> because it will always just find the last > in the line, not the next.


Reverse the numbering on sequentially numbered files

num_of_files=1034;for ((f=$num_of_files;f>-1;f--));do
cp /my/path/0$f.png /my/path2/0$(($num_of_files-f)).png; done

This copies images numbered 00.png – 01034.png so that /my/path/00.png becomes /my/path2/01034.png, 01 becomes 01033 etc etc.


Replace text in all files in a directory

find . -name *.cpp -print0 |  xargs -I{} -0 sed -i 's/Copyright/Copyleft/g' {}

Convert programs/scripts to HTML-safe and QuickLatex-safe for pasting online
It takes the contents of the clipboard and transforms < into &lt;, > into &gt; and $ into \$.
pbpaste | sed -e 's/</\&lt;/g' -e 's/>/\&gt;/g' -e 's/\$/\\$/g' | pbcopy

(This script doesn’t work applied to itself. I was going to paste here the HTML needed to produce that line on the page, but the HTML needed to display that properly would be much longer again… etc. Very like something out of Hofstadter’s Gödel, Escher, Bach, which features such things as a record that can’t be played by a certain record player, in illustration of Gödel’s ideas.)


Scrolling ad banner

a='SHEEP SUITS 50% OFF!! BAAGAIN!! ';while 
    printf '\r%s' "${a:i}${a::i++}";do 
    ((i>${#a})) && i=0;sleep 0.1;done


Find out the frequency of words in a file

#!/bin/bash 
# word_freq.sh 
if [ $# -ne 1 ]; 
then 
    echo "Usage: $0 filename"; #see how the pros do it
    exit -1 
fi 
filename=$1 
egrep -o "\b[[:alpha:]]+\b" "$filename" | awk '{ count[$0]++ } 
END{ printf("%-14s%s\n","Word","Count") ; 
for(ind in count) 
    {  printf("%-14s%d\n",ind,count[ind]);  } 
}'

then call the script perhaps with: bash word_freq.sh myfile.txt | sort -n +1 | less


Function to convert downloaded youtube mp4 video to audio only:

vtoa () { ffmpeg -i "$1" -vn -acodec copy "${1%.*}".aac; }

I like having scripts operate silently on the contents of the clipboard.
Convert SPACEs in clipboard to underscores, to make filenames more unix-friendly:

uscore() { pbpaste | tr " " _ | pbcopy; }

Convert newlines to spaces – useful to turn a copied paragraph of text with unwanted newlines into one long line:

oneline() { pbpaste | tr '\n' ' ' | pbcopy; } # OR :
oneline() { pbpaste | awk 1 ORS=' ' | pbcopy; }

Thue-Morse function

bash$ xnotx() { echo -n "$* ";for a in $*;do echo -n $((1-a))" ";done;echo; }
bash$ tm() { s=1; for ((c=1;c<=$1;c++)); do
> s=$(xnotx $s); done; echo $s | xargs -n 64 | tr -d ' ' | tr 1 ' '; }
bash$ tm 10
 00 0  00  0 00 0  0 00  00 0  00  0 00  00 0  0 00 0  00  0 00 
0  0 00  00 0  0 00 0  00  0 00  00 0  00  0 00 0  0 00  00 0  0
0  0 00  00 0  0 00 0  00  0 00  00 0  00  0 00 0  0 00  00 0  0
 00 0  00  0 00 0  0 00  00 0  00  0 00  00 0  0 00 0  00  0 00 
0  0 00  00 0  0 00 0  00  0 00  00 0  00  0 00 0  0 00  00 0  0
 00 0  00  0 00 0  0 00  00 0  00  0 00  00 0  0 00 0  00  0 00 
 00 0  00  0 00 0  0 00  00 0  00  0 00  00 0  0 00 0  00  0 00 
0  0 00  00 0  0 00 0  00  0 00  00 0  00  0 00 0  0 00  00 0  0
0  0 00  00 0  0 00 0  00  0 00  00 0  00  0 00 0  0 00  00 0  0
 00 0  00  0 00 0  0 00  00 0  00  0 00  00 0  0 00 0  00  0 00 
 00 0  00  0 00 0  0 00  00 0  00  0 00  00 0  0 00 0  00  0 00 
0  0 00  00 0  0 00 0  00  0 00  00 0  00  0 00 0  0 00  00 0  0
 00 0  00  0 00 0  0 00  00 0  00  0 00  00 0  0 00 0  00  0 00 
0  0 00  00 0  0 00 0  00  0 00  00 0  00  0 00 0  0 00  00 0  0
0  0 00  00 0  0 00 0  00  0 00  00 0  00  0 00 0  0 00  00 0  0
 00 0  00  0 00 0  0 00  00 0  00  0 00  00 0  0 00 0  00  0 00 

Customizing your bash prompt

Example 1: Put in .bash_profile :

export PS1='\![$?]\T \W$ '

i.e. history number – return status – time – current directory
Example 2: I put in .bash_profile :

PROMPT_COMMAND=prompt
prompt () 
{ 
    s=$?;
    PS1='$(bold;yellow;hr;reset)';
    if ((SHLVL>1)); then
        prstring='';
        for ((i=1; i<SHLVL; i++))
        do
            prstring+="+";
        done;
        PS1+="$(bold;cyan;echo -n $prstring;reset)";
    fi;
    if [[ s -eq '0' ]]; then
        PS1+="\!$(green;echo -n [0];reset)";
    else
        PS1+="\!$(bold;red;echo -n [$s];reset)";
    fi;
    PS1+="\T \W$ "
}

Which looks like:

i.e. yellow line (much easier to find when scrolling back up to prev command) – shell level – history num – red/green return status – time – current directory.
Prompt string format codes

\a = The ASCII bell character (007)
\A = The current time in 24-hour HH:MM format
\d = The date in “Weekday Month Day” format
\D {format} = The format is passed to strftime(3) and the result is inserted into the prompt string; an empty format results in a locale-specific time representation; the braces are required
\e = The ASCII escape character (033)
\H = The hostname
\h = The hostname up to the first “.
\j = The number of jobs currently managed by the shell
\l = The basename of the shell’s terminal device name
\n = A carriage return and line feed
\r = A carriage return
\s = The name of the shell
\T = The current time in 12-hour HH:MM:SS format
\t = The current time in HH:MM:SS format
\@ = The current time in 12-hour a.m./p.m. format
\u = The username of the current user
\v = The version of bash (e.g. 2.00)
\V = The release of bash; the version and patchlevel (e.g. 3.00.0)
\w = The current working directory
\W = The basename of the current working directory
\# = The command number of the current command
\! = The history number of the current command
\$ = If the effective UID is 0, print a #, otherwise print a $
\nnn = Character code in octal
\\ = Print a backslash.
\[ = Begin a sequence of nonprinting characters, such as terminal control sequences
\] = End a sequence of nonprinting characters

Miscellaneous

stty size = returns height & width of current shell window

script myfile = captures everything from the shell window to the file.
++-a = append (put at end of existing file)
++Ctrl-D to stop recording.

Try disabling Unicode support; bash may run 2x faster: export LC_ALL=C

See also

Books

Daniel Barnett – Macintosh Terminal Pocket Guide
Albing, Vossen, Newham – Bash Cookbook
Arnold Robbins – bash Pocket Reference
Steve Parker – Shell Scripting: Expert Recipes for Linux, Bash, and More
Richard Blum & Christine Bresnahan – Linux Command Line and Shell Scripting BIBLE
Hal Pomeranz’s Unix Command-line Kung Fu – only 33pp but well worth reading, packed full of useful tips often not mentioned elsewhere.

WWW

The Grymoire – home for UNIX wizards by Bruce Barnett
Mendel Cooper – Advanced Bash-Scripting Guide: An in-depth exploration of the gentle art of shell scripting
(An amazing work. I doubt there’s anything better or more comprehensive. Update: Except these next 3 sites:)
The Bash Hackers Wiki
Greg’s Wiki BashGuide
bash reference manual online at gnu.org

osxdaily Terminal tips Add colour to the terminal
Nate Landau: shell scripting utilities my bash profile shell script template


unix-tar


pipes-1964
Doug McIlroy thinks up pipes in 1964.


Output of tree, which I adapted from a program on Mendel Cooper’s site (I think). This version shows files, not just directories, and is in colour:

#!/bin/bash
# tree.sh
NUM_OF_COLOURS=7
trap control_c SIGINT
control_c() {
    tput sgr0
    echo
    exit 0
}
search () {
for dir in *; do
    if [[ -d "$dir" ]] ; then # ==> If it is a directory (-d)...
        declare -i zz=0 # ==> keeps track of directory level.
        tput setab 0 #black background
        while ((zz!=$1)); do     # Keep track of inner nested loop.
            tput setaf $((zz++ % $NUM_OF_COLOURS +1))
            echo -n "| "        # ==> Display vertical connector symbol,
        done
        if [[ -L "$dir" ]] ; then # ==> If directory is a symbolic link...
            g=$(ls -l "$dir")
            tput setaf $((zz % $NUM_OF_COLOURS+1))
            echo -n "L__> $dir "
            tput setab 2
            tput setaf 0 #green
            echo -n "${g#*$dir} " #just show the link at end of ls -l line.
            tput setab 0
            echo
        else #so it's a normal directory
            tput setaf $((zz % $NUM_OF_COLOURS+1))
            echo -n "+==="
            tput setaf 0
            tput setab $((zz % $NUM_OF_COLOURS+1))  
            echo -n " $dir "
            tput setab 0
            echo
            ((numdirs+=1)) #==> Increment directory count.
            if cd "$dir" ; then         # ==> If can move to subdirectory...
                search $(( $1 + 1))      # ==> Function calls itself.
                cd .. # return up one level
            fi
        fi
#delete these lines before the FI to show just directories.
    else
        if [[ -f $dir ]] ; then # continue; fi
            for ((j=0;j<$1;j++)); do
                tput setaf $((j % $NUM_OF_COLOURS +1))
                echo -n "| "
            done
            tput setaf $(($1 % $NUM_OF_COLOURS +1))
            echo -n 'L__ '
            tput sgr0 #set normal - all text printed in bold except fnames
            tput setaf $(($1 % $NUM_OF_COLOURS +1))
            echo "$dir" 
            tput bold #set back to bold  #i.e. delete up to HERE.
        fi
    fi
done
}

if [[ $# != 0 ]] ; then
    cd $1   # Move to indicated directory.
  #else   # stay in current directory
fi
tput bold
echo 'Coloured Tree v1.'
echo 'Ctrl-S = pause. SPACE = continue. Ctrl-C = quit'
echo "Initial directory = $PWD"
numdirs=0
search 0
tput sgr0 #=back to normal, i think
echo "Total directories = $numdirs"

exit 0

One thought on “not-so-quick guide to bash shell

Leave a Reply

Your email address will not be published. Required fields are marked *