shine-notes

Saturday, August 5, 2023

Bash notes expanded

BASH help

The Linux Command Line: https://wiki.lib.sun.ac.za/images/c/ca/TLCL-13.07.pdf

Template
————————
#!/bin/bash

# Replace with description and/or purpose of this shell script

GLOBAL_VAR1="one"
GLOBAL_VAR2="two"

function function_one() {
    local LOCAL_VAR1="one"
    # Replace with function code
    return 0
}

# Main body of the shell script starts here

# Exit with an explicit status

exit 0

————————
Regular Expression

Special characters —>       . * []. ^.   $   {}. \. +. ?   |. ()
^ starting
$ ending
[[:alpha:]]
[[:digit:]]
() grouping —> Sat(urday)?
Tests

[ ] numeric comparison -eq , -ge, -lt , …
[ ] string comparision - = , != , < , > -n (greater than length zero), -z (length zero)
() -> sub shell
(( )) -> is (( expression )). operations—> val++ val— ! ~ ** << >> & | && ||
[[ ]] -> is [[ advanced string operations ]]

Files
-d - file is directory
-e - file exists
-f - file exists and its a file
-s - file exists and not empty
-O - file exists and owned by me
File -nt file - newer than -ot older than

[ condition 1 ] && [ condition 2 ]
[ condition 1 ] || [ condition 2 ]

Reading from a file
cat file | while read line
do
     echo $line
done

Structured commands

If test $DEBUG
then
<<>>>
else
   <<<>>>
fi

Exit Status
echo $?

Trapping signal

trap commands signal
trap “echo shine” SIGINT
trap -- SIGINT

sed

sed ’s/test/trial/‘ data.txt

Pattern matching and delete
sed ‘/shine/d’

awk

awk ‘{print “Hello World”}’
awk -F: ‘{print $1}’ file.txt
{$4=“shine”; print $0}

awk ‘BEGIN {print “Start”} {print $0} END {print “End”}’ data3.txt

Marker

shjabbar@3c063015fbe4 Downloads % cat << EOF
heredoc> This is Shine
heredoc> How are you ?
heredoc> EOF
This is Shine
How are you ?
shjabbar@3c063015fbe4 Downloads %

Array

https://opensource.com/article/18/5/you-dont-know-bash-intro-bash-arrays

Definition:
bash-3.2$ mytest=(zero one two three four)
bash-3.2$ echo $mytest
zero

Print n-th variable:
bash-3.2$ echo ${mytest[2]}
two

Print whole array variable:

bash-3.2$ echo ${mytest[*]}
zero one two three four
bash-3.2$
bash-3.2$
bash-3.2$ mytest[2]=seven
bash-3.2$ echo ${mytest[2]}
seven
bash-3.2$
bash-3.2$ unset mytest[2]
bash-3.2$
bash-3.2$ echo ${mytest[2]}

bash-3.2$ echo ${mytest[*]}
zero one three four
bash-3.2$
bash-3.2$ unset mytest
bash-3.2$ echo ${mytest[*]}

bash-3.2$

Subshell
Using a list of commands in background

(sleep 2; echo "Woke up, Now going to sleep.."; sleep 2)&

Sort

Sort k-th element numerically

sort -t ':' -k 3 -n /etc/passwd

Grep
Get count of match
grep -c t /etc/passwd

Grep multiple fields
grep -e root -e _timed /etc/passwd

Cat

With line number
cat -n schema
shjabbar@3c063015fbe4 Downloads % cat -n schema | head
     1   {
     2        "id": "http://json-schema.org/draft-04/schema#",

Read
read -p "Enter the extension that needs to be backed up " EXTENSION

Date
88e9fe8514cd:shell shjabbar$ date +%F
2021-09-07

88e9fe8514cd:shell shjabbar$ date +%Y
2021
88e9fe8514cd:shell shjabbar$ date +%y
21
88e9fe8514cd:shell shjabbar$

Seq

88e9fe8514cd:shell shjabbar$ seq -w 02 2 18
02
04
06
08

Declare

$ declare -i x=10
$ echo $x
10
$ x=ok
$ echo $x
0
$ x=15
$ echo $x
15

Case

case "$1" in
    start|START)
        echo "Command is start"
        ;;
    stop|STOP)
        echo "Command is stop"
        ;;
    *)
        echo "Usage: $0 start|stop"
        ;;
esac

Shift

88e9fe8514cd:shell shjabbar$ cat shift.sh
#!/bin/bash

# total number of command-line arguments
echo "Total arguments passed are: $#"

# $* is used to show the command line arguments
echo "The arguments are: $*"

echo "The First Argument is: $1"
shift 2

echo "The First Argument After Shift 2 is: $1"
shift

echo "The First Argument After Shift is: $1"
88e9fe8514cd:shell shjabbar$ ./shift.sh shine mohamed jabbar puthiyaveettil house
Total arguments passed are: 5
The arguments are: shine mohamed jabbar puthiyaveettil house
The First Argument is: shine
The First Argument After Shift 2 is: jabbar
The First Argument After Shift is: puthiyaveettil
88e9fe8514cd:shell shjabbar$

Logger

(21-09-10 6:00:34) <0> [/var/log]
dev-dsk-shjabbar-2a-b6139a9a % pwd
/var/log

(21-09-10 6:00:37) <0> [/var/log]
dev-dsk-shjabbar-2a-b6139a9a % ls user.log
user.log

(21-09-10 6:00:39) <0> [/var/log]
dev-dsk-shjabbar-2a-b6139a9a % logger "Message" -s -t myscript -i
myscript[32594]: Message

(21-09-10 6:00:46) <0> [/var/log]
dev-dsk-shjabbar-2a-b6139a9a %

Debugging

#/bin/bash -x

Or

set -x
<commands>
set +x

-e option to stop at error
-v option to print shell command as it is
-x option to print shell command after expansion and substitution

DEBUG=true
If $DEBUG
then
<<>>>
Else
   <<<>>>
Fi

PS4=‘+ ${BASH_SOURCE}:${LINENO}:$FUNCNAME[0]}()

BASH tricks:

Repeat command (previous command)
!!
!a
!^
!$
!!:3
^old_string^new_string
^old_string^new_string:&
^remove_string

Current command
!#
!#:1

To record a session
script
exit

Clear history
history -c

Strip out blank or comments:
grep -E -v "^#|^$" changename.sh

Remote VIM (pretty cool)
vim scp://dev-dsk-shjabbar-2a-b6139a9a.us-west-2.amazon.com//home/shjabbar/get_snitch_graph

Print out as a table / column:
column -t
88e9fe8514cd:shell shjabbar$ mount | column -t
/dev/disk1s5 on         /                     (apfs,                                local,    read-only,    journaled)
devfs         on         /dev                  (devfs,                               local,    nobrowse)

Less with color:
grep gp_host_names get_snitch_graph
| less -R

Preserve color when piping to grep
—color=never
ls -l --color=always | grep get_ --color=never

Append text to file using sudo

echo $ENVIRONMENT | sudo tee -a /etc/motd

Display block of strings
awk '/Shadow/,/edit/' exercise4.sh

Delete block of strings

sed '/Shadow/,/edit/d' exercise4.sh

Remove a character or a set

cat exercise4.sh | tr -d [aeiou]

uniq -c

Get your public IP using curl

curl ifconfig.me/ip
204.246.162.45

Open network ports:
lsof -Pni
dev-dsk-shjabbar-2a-b6139a9a % sudo lsof -Pni
COMMAND     PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
unbound    2759    named    3u IPv4    1622      0t0 UDP 127.0.0.1:53

Send mail
mail shjabbar@amazon.com
echo "Isn't this great" | mail -s "Test mail 2" -a get_snitch_graph shjabbar@amazon.com

dev-dsk-shjabbar-2a-b6139a9a % mail shjabbar@amazon.com
Subject: Test mail
Isn't this great
EOT

(21-09-19 5:44:37) <0> [~]
dev-dsk-shjabbar-2a-b6139a9a %

Create an SSH tunnel:

ssh -N -L local-port-opened-in-local-host:end-host:end-port remote-host-where-the-connection-will-be-initiated

ssh -N -L 8000:localhost:80 linuxserver

Simplifying multi-hop SSH

In .ssh/config
# Configuration for CPT
host *.cpt*.amazon.com *.cpt*.amazon.com. *.cpt? *.cpt??
          ProxyCommand ssh bastion-cpt.amazon.com nc %h %p

Download webpage:
wget
curl -o file.html www.google.com

Read:
read VAR
read -n 1 VAR
read -p “Enter Value: “ VAR

Sum a column:

awk '{sum += $3} END {print sum}'

2021-09-21-05:10:29 big-mac-prod-tools-cmh-cmh52-50001.cmh50 cmh52:prod > df 2> /dev/null | awk '{sum += $3} END {print sum}'
118810236
2021-09-21-05:10:58 big-mac-prod-tools-cmh-cmh52-50001.cmh50 cmh52:prod >

Automatic Yes:
yes | command

Pkill:
pkill -9 httpd

Sort process with memory usage:
ps aux | sort -nk 4 | tail -5 | cut -c1-140

Make a quick backup
cp cscope.files{,.bak}

Sequence:
echo 10.0.0.{0..7}
10.0.0.0 10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4 10.0.0.5 10.0.0.6 10.0.0.7

Truncate a file:
> file
cat /dev/null > file

Create multiple directory:
mkdir {1..4}

Delete empty directory
find . -type d -empty -delete

List of all files with matching string
grep -rl bash *

Recursive ls:

ls -lR
find . -type f -ls

Replace string recursively with backup
find . -type f -iname "*.sh" -exec sed -i.bak 's/echo/ECHO/g' {} \;

Print Nth line:
awk 'NR==4' exercise4.sh
cat exercise4.sh | awk 'NR==4'

Convert text to image
sudo yum install ImageMagick
echo shine | convert label:@- hine.png

FOR loop

for VARIABLE in 1 2 3 4 5 .. N
do
   command1
   command2
   commandN
done

for (( a = 1; a < 10; a++ ))
for (( a = 1, b = 10; a < 10; a++, b— ))

break
continue

Directory Name and base name
(21-10-07 21:04:28) <0> [~]
dev-dsk-shjabbar-2a-b6139a9a % basename bin/cscope.sh
cscope.sh

(21-10-07 21:04:41) <0> [~]
dev-dsk-shjabbar-2a-b6139a9a % man basename

(21-10-07 21:06:08) <0> [~]
dev-dsk-shjabbar-2a-b6139a9a % dirname bin/cscope.sh
bin

(21-10-07 21:06:16) <0> [~]
dev-dsk-shjabbar-2a-b6139a9a %

Function Parameters
shjabbar@3c063015fbe4 shell % cat rest_arg.sh
#!/usr/bin/env bash

all_args=$@
first_arg=$1
second_args=$2
rest_args=("${all_args[@]:2}")
shift
all_args_after_shift=$@

echo "All arguments - \$@ - $all_args"
echo "All arguments after shift - \$@ - $all_args_after_shift"
echo "Rest of the arguments - (\"\${all_args[@]:2}\") - ${rest_args[@]}"

shjabbar@3c063015fbe4 shell % ./rest_arg.sh 1 2 3 4 5 6 6
All arguments - $@ - 1 2 3 4 5 6 6
All arguments after shift - $@ - 2 3 4 5 6 6
Rest of the arguments - ("${all_args[@]:2}") - 2 3 4 5 6 6
shjabbar@3c063015fbe4 shell %

Checks

-n non-zero length
-z zero length
-f file exists

Not new line:
echo -n Dg Dg

command << marker
Dish’s
marker

Shopt

shopt -s nullglob

Wednesday, September 22, 2021

Bash notes

Sunday, December 22, 2019

Madina Arabic Reader Book 1 by Dr. V. Abdur Rahim

Chapter 1:

Chapter 2:

Chapter 3:

Chapter 4:

Chapter 5:

Chapter 6:

Chapter 7:

Chapter 8:

Chapter 9:

Chapter 10:

Wednesday, December 18, 2019

The Linux Command Line by William Shotts

Linux commands/shell
Prompt
date
cal
df
free
exit
pwd
cd
ls
file
less
Various system directories
cp
mv
mkdir
rm
ln
ln -s
type
which
help
--help
man
apropos
whatis
info
alias
unalias
cat
sort
uniq
grep
wc
head
tail
tee
echo
echo $((2 + 2))
echo $(($((5**2)) * 3))
echo $(((5**2) * 3))
echo Five divided by two equals $((5/2))
echo Front-{A,B,C}-Back
echo Number_{1..5}
echo {001..15}
88e9fe8514cd:~ shjabbar$ echo a{A{1,2},B{3,4}}b
aA1b aA2b aB3b aB4b
88e9fe8514cd:~ shjabbar$
echo $USER
printenv | less
ls -l $(which cp)
file $(ls -d /usr/bin/* | grep zip)
ls -l `which cp`
echo this is a       test
echo "$USER $((2+2)) $(cal)"
88e9fe8514cd:Downloads shjabbar$ echo $(cal)
October 2019 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
88e9fe8514cd:Downloads shjabbar$ echo "$(cal)"
    October 2019
Su Mo Tu We Th Fr Sa
       1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31

88e9fe8514cd:Downloads shjabbar$ echo '$(cal)'
$(cal)
88e9fe8514cd:Downloads shjabbar$

88e9fe8514cd:Downloads shjabbar$ echo text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER
text /Users/shjabbar/shine.txt a b foo 4 shjabbar
88e9fe8514cd:Downloads shjabbar$ echo "text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER"
text ~/*.txt {a,b} foo 4 shjabbar
88e9fe8514cd:Downloads shjabbar$ echo 'text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER'
text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER
88e9fe8514cd:Downloads shjabbar$

sleep 10; echo -e "Time's up\a"
We could also do this:
sleep 10; echo "Time's up" $'\a'

8 – Advanced Keyboard Tricks (lot of keyboard tricks)
clear
history
ctrl-l to clear
Alt-* Insert all possible completions.
!string - Repeat last history list item starting with string.
!?string - Repeat last history list item containing string.
script
ctrl-c to stop script

9 – Permissions
id – Display user identity
chmod – Change a file's mode
u+x Add execute permission for the owner.
   u-x
+x Add execute permission for the owner, group, and world. This is
Remove execute permission from the owner. equivalent to a+x.

umask – Set the default file permissions
su -
su
sudo
bash-3.2$ sudo -l
Matching Defaults entries for shjabbar on 88e9fe8514cd:
    env_reset, env_keep+=BLOCKSIZE, env_keep+="COLORFGBG COLORTERM", env_keep+=__CF_USER_TEXT_ENCODING, env_keep+="CHARSET LANG LANGUAGE LC_ALL LC_COLLATE LC_CTYPE", env_keep+="LC_MESSAGES LC_MONETARY LC_NUMERIC LC_TIME", env_keep+="LINES COLUMNS",
    env_keep+=LSCOLORS, env_keep+=SSH_AUTH_SOCK, env_keep+=TZ, env_keep+="DISPLAY XAUTHORIZATION XAUTHORITY", env_keep+="EDITOR VISUAL", env_keep+="HOME MAIL", lecture_file=/etc/sudo_lecture

User shjabbar may run the following commands on 88e9fe8514cd:
    (ALL) ALL
    (root) NOPASSWD: /usr/local/bin/elevate-admin
bash-3.2$

chown [owner][:[group]] file...
passwd

10 - Processes

ps
ps x
ps aux
top
jobs
fg
bg
ctrl-z
kill
kill -1
killall xlogo
pstree
vmstat
xload
tload

Part 2 Configuration and the Environment

11 - The Environment

printenv
printenv USER
set | less
The set command, when used without options or arguments, will display both the shell and environment variables, as well as any defined shell functions.
Unlike printenv, its output is courteously sorted in alphabetical order.

/etc/profile: A global configuration script that applies to all users.
~/.bash_profile: A user's personal startup file. This can be used to extend or override settings in the global configuration script.
~/.bash_login: If ~/.bash_profile is not found, bash attempts to read this script.
~/.profile: If neither ~/.bash_profile nor ~/.bash_login is found, bash attempts to read this file. This is the default in Debian-based distributions, such as Ubuntu.

/etc/bash.bashrc: A global configuration script that applies to all users.
~/.bashrc: A user's personal startup file. It can be used to extend or override settings in the global configuration script.

if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

The export command tells the shell to make the contents of PATH available to child processes of this shell.

13 - Customizing the Prompt

[me@linuxbox ~]$ echo $PS1
[\u@\h \W]\$

[shjabbar@88e9fe8514cd.ant.amazon.com ~] bash-3.2$ echo $PS1
(\d \t)\n[\033[1;41m\u@\H\[\033[0m\] \w] \s-\v$
(Mon Nov 18 23:09:08)
[shjabbar@88e9fe8514cd.ant.amazon.com ~] bash-3.2$

14- Package management

Debian Style (.deb)   Debian, Ubuntu, Linux Mint, Raspbian
Red Hat Style (.rpm) Fedora, CentOS, Red Hat Enterprise Linux, OpenSUSE

Distributions                               Low-Level Tools   High-Level Tools
=============                               ===============   ================
Debian style                                dpkg              apt, apt-get, aptitude
Fedora, Red Hat Enterprise Linux, CentOS    rpm               yum, dnf

apt-get install package_name
yum install package_name

dpkg -i package_file
rpm -i package_file

apt-get remove package_name
yum erase package_name

15 - Storage Media

cat /etc/fstab
mount
tail -f /var/log/messages
sudo fdisk /dev/sdb

[me@linuxbox ~]$ sudo mkfs -t ext4 /dev/sdb1
mke2fs 2.23.2 (12-Jul-2011)
Filesystem label=

[me@linuxbox ~]$ sudo fsck /dev/sdb1
fsck 1.40.8 (13-Mar-2016)

dd if=input_file of=output_file [bs=block_size [count=blocks]]
dd if=/dev/sdb of=/dev/sdc
dd if=/dev/sdb of=flash_drive.img

cdrom commands

16 - Networking

ping
traceroute
ip a
netstat -ie
netstat -r
ssh

17 - Searching for files

locate
updatedb
find ~
find ~ -type d
find ~ -type f -name "*.JPG" -size +1M

find ~ ( -type f -not -perms 0600 ) -or ( -type d -not -perms 0700 ) -delete
find ~ ( -type f -not -perms 0600 ) -or ( -type d -not -perms 0700 ) -exec ls '{}' ';'

[me@linuxbox ~]$ mkdir -p playground/dir-{001..100}
[me@linuxbox ~]$ touch playground/dir-{001..100}/file-{A..Z}

18 - Archiving and Backup

gzip -tv foo.txt.gz
foo.txt.gz: OK

ls -l /etc | gzip > foo.txt.gz
gunzip -c foo.txt | less
zcat foo.txt.gz | less
zless
bzip2
bunzip2
bzcat
bunzip2 foo.txt.bz2

tar cf playground.tar playground

c Create an archive from a list of files and/or directories.
x Extract an archive.
r Append specified pathnames to the end of an archive
t List the contents of an archive.

tar xf archive.tar pathname
find playground -name 'file-A' | tar cf - --files- from=- | gzip > playground.tgz
find playground -name 'file-A' | tar czf playground.tgz -T -
find playground -name 'file-A' | tar cjf playground.tbz -T -
ssh remote-sys 'tar cf - Documents' | tar xf -
zip and unzip are pretty good and contain useful examples. However, the main use of these programs is for exchanging files with Windows systems

rsync options source destination
rsync -av playground foo
sudo rsync -av --delete /etc /home /usr/local /media/BigDisk/backup
sudo rsync -av --delete --rsh=ssh /etc /home /usr/local remote-sys:/backup
rsync -av –delete rsync://archive.linux.duke.edu/ fedora/linux/development/rawhide/Everything/x86_64/os/ fedora-devel

19 - Regular Expressions

grep
-i --ignore-case
-v        --invert-match
-c --count
-l        --files-with-matches
-L        --files-without-match
-n        --line-number
-h        --no-filename
grep -h '.zip' dirlist*.txt

[me@linuxbox ~]$ grep -h '^zip$' dirlist*.txt
zip

[me@linuxbox ~]$ grep -i '^..j.r$' /usr/share/dict/words
Major
major

[me@linuxbox ~]$ grep -h '[bg]zip' dirlist*.txt
bzip2
bzip2recover
gzip

[me@linuxbox ~]$ grep -h '[^bg]zip' dirlist*.txt
bunzip2
gunzip
funzip

[me@linuxbox ~]$ grep -h '^[A-Z]' dirlist*.txt
MAKEDEV
ControlPanel
GET

POSIX meta characters           - ^ $ . [ ] *
Extended Regular Expressions    ( ) { } ? + |

[me@linuxbox ~]$ echo "AAA" | grep -E 'AAA|BBB'
AAA

#This expression will match the filenames in our lists that start with either bz, gz, or zip.
grep -Eh '^(bz|gz|zip)' dirlist*.txt

? - Match an Element Zero or One Time
* - Match an Element Zero or More Times
+ - Match an Element One or More Times
{ } - Match an Element a Specific Number of Times

[me@linuxbox ~]$ echo "(555) 123-4567" | grep -E '^$?[0-9]{3}$? [0- 9]{3}-[0-9]{4}$'
(555) 123-4567

find . -regex '.*[^-_./0-9a-zA-Z].*'

vim search:    /([0-9]\{3\}) [0-9]\{3\}-[0-9]\{4\}

20 - Text Processing

cat -A foo.txt
^IThe quick brown fox jumped over the lazy dog. $
[me@linuxbox ~]$

[me@linuxbox ~]$ cat -ns foo.txt
1 The quick brown fox
2
3 jumped over the lazy dog.
[me@linuxbox ~]$

[me@linuxbox ~]$ sort > foo.txt
c
b
a
[me@linuxbox ~]$ cat foo.txt
a
b
c

sort file1.txt file2.txt file3.txt > final_sorted_list.txt
-b --ignore-leading-blanks
-f --ignore-case
-n --numeric-sort
-r --reverse
-k --key=field1[,field2]
-m --merge
-o --output=file
-t --field-separator=char

[me@linuxbox ~]$ du -s /usr/share/* | sort -nr | head
509940 /usr/share/locale-langpack
242660 /usr/share/doc
197560 /usr/share/fonts

[me@linuxbox ~]$ ls -l /usr/bin | sort -nrk 5 | head
-rwxr-xr-x 1 root   root 8234216 2016-04-07 17:42 inkscape
-rwxr-xr-x 1 root   root 8222692 2016-04-07 17:42 inkview
-rwxr-xr-x 1 root   root 3746508 2016-03-07 23:45 gimp-2.4

[me@linuxbox ~]$ sort --key=1,1 --key=2n distros.txt
Fedora 5 03/20/2006
Fedora 6 10/24/2006
Fedora 7 05/31/2007
Fedora 8 11/08/2007
Fedora 9 05/13/2008

[me@linuxbox ~]$ sort -k 3.7nbr -k 3.1nbr -k 3.4nbr distros.txt
Fedora 10     11/25/2008
Ubuntu 8.10   10/30/2008
SUSE 11.0     06/19/2008

[me@linuxbox ~]$ sort -t ':' -k 7 /etc/passwd | head
me:x:1001:1001:Myself,,,:/home/me:/bin/bash
root:x:0:0:root:/root:/bin/bash
dhcp:x:101:102::/nonexistent:/bin/false

sort foo.txt | uniq
sort foo.txt | uniq -c

cut -f 3 distros.txt | cut -c 7-10
cut -d ':' -f 1 /etc/passwd | head

[me@linuxbox ~]$ paste distros-dates.txt distros-versions.txt
11/25/2008 Fedora 10
10/30/2008 Ubuntu 8.10
06/19/2008 SUSE 11.0
05/13/2008 Fedora 9

[me@linuxbox ~]$ join distros-key-names.txt distros-key-vernums.txt | head
11/25/2008 Fedora 10
10/30/2008 Ubuntu 8.10
06/19/2008 SUSE 11.0

comm
diff
patch

[me@linuxbox ~]$ diff -Naur file1.txt file2.txt > patchfile.txt
[me@linuxbox ~]$ patch < patchfile.txt
patching file file1.txt

echo "lowercase letters" | tr a-z A-Z

tr -d '\r' < dos_file > unix_file

[me@linuxbox ~]$ echo "aaabbbccc" | tr -s ab
abccc

[me@linuxbox ~]$ echo "front" | sed 's/front/back/'
back

#line number based
[me@linuxbox ~]$ echo "front" | sed '2s/front/back/'
front

sed 's/$[0-9]\{2\}$\/$[0-9]\{2\}$\/\([0-9]\{4\}\ )$/\3-\1-\2/' distros.txt
We now have three subexpressions ()
sed -f distros.sed distros.txt

[me@linuxbox ~]$ sed -i 's/lazy/laxy/; s/jumped/jimped/' foo.txt
The sed option -i tells sed to edit the file “in-place,”

aspell check foo.txt
aspell -H check foo.txt

21 - Formatting Output

nl
[me@linuxbox ~]$ nl distros.txt | head
1 SUSE 10.2
2 Fedora 10
3 SUSE 11.0

nl markup used in sed
\:\:\: Start of logical page header
\:\:    Start of logical page body
\:      Start of logical page footer

[me@linuxbox ~]$ echo "The quick brown fox jumped over the lazy dog." | fold -w 12 -s
The quick
brown fox
jumped over
the lazy
dog.

[me@linuxbox ~]$ fmt -cw 50 fmt-info.txt
`fmt' reads from the specified FILE arguments
(or standard input if none are given), and writes
to standard output.

[me@linuxbox ~]$ printf "I formatted the string: %s\n" foo
I formatted the string: foo

[me@linuxbox ~]$ printf "I formatted '%s' as a string.\n" foo
I formatted 'foo' as a string.

groff

zcat /usr/share/man/man1/ls.1.gz | groff -mandoc -T ascii | head
zcat /usr/share/man/man1/ls.1.gz | groff -mandoc > ~/Desktop/ls.ps

22 – Printing

skipping

23 – Compiling Programs

skipping

24 – Writing Your First Script

#!/bin/bash
# This is our first script.
echo 'Hello World!'

25 – Starting a Project

mv "$filename" "${filename}1"

#-r (read- only) option
declare -r TITLE="Page Title"

variable=value

CURRENT_TIME="$(date +"%x %r %Z")"
TIMESTAMP="Generated $CURRENT_TIME, by $USER"

Here Documents

command << token
text
token

cat << _EOF_
<html>
        <head>
                <title>$TITLE</title>
        </head>
        <body>
                <h1>$TITLE</h1>
                <p>$TIMESTAMP</p>
        </body>
</html>
_EOF_

If we change the redirection operator from << to <<-, the shell will ignore leading tab

26 – Top-Down Design

function step2 {
    echo step 2
    return
}

echo step 1
step2
echo step 3

#!/bin/bash
# local-vars: script to demonstrate local variables foo=0 # global variable foo
funct_1 () {
local foo # variable foo local to funct_1
foo=1
    echo "funct_1: foo = $foo"
}

27 – Flow Control: Branching with if

x=5
if [ "$x" -eq 5 ]; then
    echo "x equals 5."
else
    echo "x does not equal 5."
fi

if [ “$x” -eq 5 ]; then echo "equals 5"; else echo "does not equal 5"; fi

(19-12-15 15:34:46) <0> [~/workbook]
dev-dsk-shjabbar-2a-70ace2a6 % echo $?
0

(19-12-15 15:34:50) <0> [~/workbook]
dev-dsk-shjabbar-2a-70ace2a6 % ls /ere
ls: cannot access /ere: No such file or directory

(19-12-15 15:34:56) <2> [~/workbook]
dev-dsk-shjabbar-2a-70ace2a6 % echo $?
2

(19-12-15 15:34:59) <0> [~/workbook]
dev-dsk-shjabbar-2a-70ace2a6 %

[me@linuxbox ~]$ true
[me@linuxbox ~]$ echo $?
0
[me@linuxbox ~]$ false
[me@linuxbox ~]$ echo $?
1

[me@linuxbox ~]$ if true; then echo "It's true."; fi
It's true.
[me@linuxbox ~]$ if false; then echo "It's true."; fi
[me@linuxbox ~]$

dev-dsk-shjabbar-2a-70ace2a6 % if true;false;true;then echo "It's true."; fi
It's true.

The test and [ commands support a wide range of useful expressions and tests.

test_file () {
    # test-file: Evaluate the status of a file
    FILE=~/.bashrc
    if [ -e "$FILE" ]; then
        if [ -f "$FILE" ]; then
        echo "$FILE is a regular file."
fi
if [ -d "$FILE" ]; then
       echo "$FILE is a directory."
fi
if [ -r "$FILE" ]; then
       echo "$FILE is readable."
fi
if [ -w "$FILE" ]; then
       echo "$FILE is writable."
fi
if [ -x "$FILE" ]; then
       echo "$FILE is executable/searchable."
fi
else
echo "$FILE does not exist"
return 1
fi
}

#!/bin/bash
# test-string: evaluate the value of a string
ANSWER=maybe
if [ -z "$ANSWER" ]; then
    echo "There is no answer." >&2
    exit 1
fi
if [ "$ANSWER" = "yes" ]; then
    echo "The answer is YES."
elif [ "$ANSWER" = "no" ]; then
    echo "The answer is NO."
elif [ "$ANSWER" = "maybe" ]; then
    echo "The answer is MAYBE."
else
    echo "The answer is UNKNOWN."
fi

#!/bin/bash
# test-integer: evaluate the value of an integer.
INT=-5
if [ -z "$INT" ]; then
    echo "INT is empty." >&2
    exit 1
if [ "$INT" -eq 0 ]; then
    echo "INT is zero."
else
    if [ "$INT" -lt 0 ]; then
        echo "INT is negative."
    else
        echo "INT is positive."
    fi
    if [ $((INT % 2)) -eq 0 ]; then
        echo "INT is even."
    else
        echo "INT is odd."
fi
fi

[[ ]]
#!/bin/bash
# test-integer2: evaluate the value of an integer.
INT=-5
if [[ "$INT" =~ ^-?[0-9]+$ ]]; then
    if [ "$INT" -eq 0 ]; then
        echo "INT is zero."
    fi
fi

The ~ is actually part of the operator =~ which performs a regular expression match of the string to its left to the extended regular expression on its right.

[[ "string" =~ pattern ]]

[[ ]]
[me@linuxbox ~]$ FILE=foo.bar
[me@linuxbox ~]$ if [[ $FILE == foo.* ]]; then > echo "$FILE matches pattern 'foo.*'"
> fi
foo.bar matches pattern 'foo.*'

(( ))
# test-integer2a: evaluate the value of an integer.
INT=-5
if [[ "$INT" =~ ^-?[0-9]+$ ]]; then
if ((INT == 0)); then
        echo "INT is zero."
    else
if ((INT < 0)); then
echo "INT is negative."
        else
             echo "INT is positive."
fi
if (( ((INT % 2)) == 0)); then
             echo "INT is even."
        else
             echo "INT is odd."
        fi
fi else
    echo "INT is not an integer." >&2
exit 1 fi

Operation   test    [[ ]] and (( ))
=========   ====    ===============
AND         -a      &&
OR          -o      ||
NOT         !       !

AND and negation

[[ ]]
AND and negation
if [[ "$INT" -ge "$MIN_VAL" && "$INT" -le "$MAX_VAL" ]]; then
        echo "$INT is within $MIN_VAL to $MAX_VAL."
    else
        echo "$INT is out of range."
    fi

    if [[ ! ("$INT" -ge "$MIN_VAL" && "$INT" -le "$MAX_VAL") ]]; then
            echo "$INT is outside $MIN_VAL to $MAX_VAL."
        else
            echo "$INT is in range."
        fi

OR

[ ]
if [ "$INT" -ge "$MIN_VAL" -a "$INT" -le "$MAX_VAL" ]; then echo "$INT is within $MIN_VAL to $MAX_VAL."
else
     echo "$INT is out of range."
fi

if [ ! $ "$INT" -ge "$MIN_VAL" -a "$INT" -le "$MAX_VAL" $ ];
then
        echo "$INT is outside $MIN_VAL to $MAX_VAL."
    else
        echo "$INT is in range."
fi

command1 && command2

command1 || command2

[me@linuxbox ~]$ [[ -d temp ]] || mkdir temp
[ -d temp ] || exit 1

28 – Reading Keyboard Input

echo -n "Please enter an integer -> "
read int

#!/bin/bash
# read-multiple: read multiple values from keyboard
echo -n "Enter one or more values > "
read var1 var2 var3 var4 var5
echo "var1 = '$var1'"
echo "var2 = '$var2'"
echo "var3 = '$var3'"
echo "var4 = '$var4'"
echo "var5 = '$var5'"

read -e -p "What is your user name? " -i $USER

echo "You answered: '$REPLY'"

if read -t 10 -sp "Enter secret passphrase > " secret_pass; then

      echo -e "\nSecret passphrase = '$secret_pass'"

else

      echo -e "\nInput timed out" >&2

      exit 1

fi

IFS=":" read user pw uid gid name home shell <<< "1:2:3"
dev-dsk-shjabbar-2a-70ace2a6 % print $user $pw $uid
1 2 3

[[ -z "$REPLY" ]] && invalid_input
(( "$(echo "$REPLY" | wc -w)" > 1 )) && invalid_input

Validating Input
Menus

29 – Flow Control: Looping with while / until

while commands; do commands; done

#!/bin/bash
# while-count: display a series of numbers
count=1
while [[ "$count" -le 5 ]]; do
    echo "$count"
    count=$((count + 1))
done
echo "Finished."

#!/bin/bash
# until-count: display a series of numbers
count=1
until [[ "$count" -gt 5 ]]; do echo "$count"
    count=$((count + 1))
done
echo "Finished."

dev-dsk-shjabbar-2a-70ace2a6 % while ls; do
while> sleep 3
while> done
bash-howto if.sh read.sh shell_function.sh while
bash-howto if.sh read.sh shell_function.sh while
bash-howto if.sh read.sh shell_function.sh while
bash-howto if.sh read.sh shell_function.sh while
bash-howto if.sh read.sh shell_function.sh while
^C%
(19-12-16 6:32:25) <130> [~/workbook]
dev-dsk-shjabbar-2a-70ace2a6 %

#!/bin/bash
# while-read: read lines from a file
while read distro version release; do
    printf "Distro: %s\tVersion: %s\tReleased: %s\n" \
        "$distro" \
        "$version" \
        "$release"
done < distros.txt

#!/bin/bash
# while-read2: read lines from a file
sort -k 1,1 -k 2n distros.txt | while read distro version release; do
printf "Distro: %s\tVersion: %s\tReleased: %s\n" \
      "$distro" \
done

30 – Troubleshooting

#!/bin/bash -x
# trouble: script to demonstrate common errors
number=1
if [ $number = 1 ]; then
    echo "Number is equal to 1."
else
    echo "Number is not equal to 1."
fi

[me@linuxbox ~]$ trouble
+ number=1
+ '[' 1 = 1 ']'
+ echo 'Number is equal to 1.'
Number is equal to 1.

#!/bin/bash
# trouble: script to demonstrate common errors
number=1
set -x # Turn on tracing
if [ $number = 1 ]; then
    echo "Number is equal to 1."
else
    echo "Number is not equal to 1."
fi
set +x # Turn off tracing

31 – Flow Control: Branching with case

case word in
[pattern [| pattern]...) commands ;;]...
esac

case "$REPLY" in
q|Q) echo "Program terminated."
exit
;;
a|A) echo "Hostname: $HOSTNAME"
uptime
;; b|B) df -h
;;
c|C) if [[ "$(id -u)" -eq 0 ]]; then
             echo "Home Space Utilization (All Users)"
             du -sh /home/*
        else
             echo "Home Space Utilization ($USER)"
             du -sh "$HOME"
        fi
;;
*) echo "Invalid entry" >&2
exit 1
;; esac

case "$REPLY" in
0) echo "Program terminated."
    exit
    ;;
esac

32 – Positional Parameters

[me@linuxbox ~]$ posit-param a b c d
$0 = /home/me/bin/posit-param
$1 = a
$2 = b
$3 = c
$4 = d
$5 =
$6 =
$7 =
$8 =
$9 =

Note: You can actually access more than nine parameters using parameter expan- sion.
To specify a number greater than nine, surround the number in braces as in ${10}, ${55}, ${211}, and so on.

Number of arguments: $#

[me@linuxbox ~]$ posit-param a b c d
Number of arguments: 4
$0 = /home/me/bin/posit-param
$1 = a
$2 = b
$3 = c
$4 = d
$5 =
$6 =
$7 =
$8 =
$9 =

#!/bin/bash
# posit-param2: script to display all arguments
count=1
while [[ $# -gt 0 ]]; do
    echo "Argument $count = $1"
    count=$((count + 1))
    shift
done

[me@linuxbox ~]$ posit-param2 a b c d
Argument 1 = a
Argument 2 = b
Argument 3 = c
Argument 4 = d

PROGNAME="$(basename "$0")"

echo "$FUNCNAME: usage: $FUNCNAME file" >&2

file_info () {
    # file_info: function to display file information
    if [[ -e "$1" ]]; then
        echo -e "\nFile Type:"
        file "$1"
        echo -e "\nFile Status:"
        stat "$1"
    else
        echo "$FUNCNAME: usage: $FUNCNAME file" >&2
        return 1
fi }

$* Expands into the list of positional parameters, starting with 1.
   When surrounded by double quotes, it expands into a double-quoted string
   containing all of the positional parameters, each separated by the first
   character of the IFS shell variable (by default a space character).
$@ Expands into the list of positional parameters, starting with 1.
   When surrounded by double quotes, it expands each positional
   parameter into a separate word as if it was surrounded by double quotes.

Always use "$@"

#!/bin/bash
# posit-params3: script to demonstrate $* and $@
print_params () {
    echo "\$1 = $1"
    echo "\$2 = $2"
    echo "\$3 = $3"
    echo "\$4 = $4"
}
pass_params () {
    echo -e "\n" '$* :'; print_params $*
    echo -e "\n" '"$*" :'; print_params "$*"
    echo -e "\n" '$@ :';   print_params $@
    echo -e "\n" '"$@" :'; print_params "$@"
}
pass_params "word" "words with spaces"

[me@linuxbox ~]$ posit-param3
$* :
$1 = word
$2 = words
$3 = with
$4 = spaces
"$*" :
$1 = word words with spaces
$2 =
$3 =
$4 =
$@ :
$1 = word
$2 = words
$3 = with
$4 = spaces
"$@" :
$1 = word
$2 = words with spaces
$3 =
$4 =

usage () {
    echo "$PROGNAME: usage: $PROGNAME [-f file | -i]"
    return
}

# process command line options
interactive=
filename=
while [[ -n "$1" ]]; do
    case "$1" in
      -f | --file) shift
                    filename="$1"
                    ;;
      -i | --interactive) interactive=1
                    ;;
      -h | --help) usage
                    exit
                    ;;
        *)          usage >&2
                    exit 1
                    ;;
    esac
done

33 – Flow Control: Looping with for

for variable [in words]; do
commands
done

for i in {A..D}; do echo $i; done

for i in distros*.txt; do echo "$i"; done

for (( expression1; expression2; expression3 )); do
commands
done

#!/bin/bash
# simple_counter: demo of C style for command
for (( i=0; i<5; i=i+1 )); do
    echo $i
done

34 – Strings and Numbers

[me@linuxbox ~]$ echo "${a}_file"

${11}

${parameter:-word}

[me@linuxbox ~]$ foo=
[me@linuxbox ~]$ echo ${foo:-"substitute value if unset"}
substitute value if unset
[me@linuxbox ~]$ echo $foo

[me@linuxbox ~]$ foo=bar
[me@linuxbox ~]$ echo ${foo:-"substitute value if unset"}
bar
[me@linuxbox ~]$ echo $foo

${parameter:=word}

[me@linuxbox ~]$ foo=
[me@linuxbox ~]$ echo ${foo:="default value if unset"}
default value if unset
[me@linuxbox ~]$ echo $foo
default value if unset
[me@linuxbox ~]$ foo=bar
[me@linuxbox ~]$ echo ${foo:="default value if unset"}
bar
[me@linuxbox ~]$ echo $foo
bar

${parameter:?word}

[me@linuxbox ~]$ foo=
[me@linuxbox ~]$ echo ${foo:?"parameter is empty"}
bash: foo: parameter is empty
[me@linuxbox ~]$ echo $?
1
[me@linuxbox ~]$ foo=bar
[me@linuxbox ~]$ echo ${foo:?"parameter is empty"}
bar
[me@linuxbox ~]$ echo $?
0

${parameter:+word}
If parameter is unset or empty, the expansion results in nothing. If parameter is not empty, the value of word is substituted for parameter; however, the value of parameter is not changed.

[me@linuxbox ~]$ foo=
[me@linuxbox ~]$ echo ${foo:+"substitute value if set"}

[me@linuxbox ~]$ foo=bar
[me@linuxbox ~]$ echo ${foo:+"substitute value if set"}
substitute value if set

[me@linuxbox ~]$ foo="This string is long."
[me@linuxbox ~]$ echo "'$foo' is ${#foo} characters long."
'This string is long.' is 20 characters long.

[me@linuxbox ~]$ foo="This string is long."
[me@linuxbox ~]$ echo ${foo:5}
string is long.
[me@linuxbox ~]$ echo ${foo:5:6}
string

[me@linuxbox ~]$ foo="This string is long."
[me@linuxbox ~]$ echo ${foo: -5}
long.
[me@linuxbox ~]$ echo ${foo: -5:2}
lo

${parameter#pattern}
${parameter##pattern}

[me@linuxbox ~]$ foo=file.txt.zip
[me@linuxbox ~]$ echo ${foo#*.}
txt.zip
[me@linuxbox ~]$ echo ${foo##*.}
zip

${parameter%pattern}
${parameter%%pattern}

[me@linuxbox ~]$ foo=file.txt.zip
[me@linuxbox ~]$ echo ${foo%.*}
file.txt
[me@linuxbox ~]$ echo ${foo%%.*}
file

[me@linuxbox ~]$ foo=file.txt.zip
[me@linuxbox ~]$ echo ${foo%.*}
file.txt
[me@linuxbox ~]$ echo ${foo%%.*}
file

${parameter/pattern/string}
${parameter//pattern/string}
${parameter/#pattern/string}
${parameter/%pattern/string}

[me@linuxbox ~]$ foo=JPG.JPG
[me@linuxbox ~]$ echo ${foo/JPG/jpg}
jpg.JPG
[me@linuxbox ~]$ echo ${foo//JPG/jpg}
jpg.jpg
[me@linuxbox ~]$ echo ${foo/#JPG/jpg}
jpg.JPG
[me@linuxbox ~]$ echo ${foo/%JPG/jpg}
JPG.jpg

#!/bin/bash
# ul-declare: demonstrate case conversion via declare
declare -u upper
declare -l lower
if [[ $1 ]]; then
        upper="$1"
        lower="$1"
        echo "$upper"
        echo "$lower"
fi

#!/bin/bash
# ul-param: demonstrate case conversion via parameter expansion
if [[ "$1" ]]; then
        echo "${1,,}"
        echo "${1,}"
        echo "${1^^}"
        echo "${1^}"
fi

[me@linuxbox ~]$ ul-param aBc
abc
aBc
ABC
ABc

All c logical and arithmetic operation in (( --- ))

bc

35 – Arrays

declare -a a

Using the -a option, this example of declare creates the array a.

[me@linuxbox ~]$ days=(Sun Mon Tue Wed Thu Fri Sat)

[me@linuxbox ~]$ days=([0]=Sun [1]=Mon [2]=Tue [3]=Wed [4]=Thu [5]=Fri [6]=Sat)

printf "%02d\t%d\t%02d\t%d\n" \
        "$i" \
        "${hours[i]}" \
        "$j" \
"${hours[j]}"

[me@linuxbox ~]$ animals=("a dog" "a cat" "a fish")
[me@linuxbox ~]$ for i in ${animals[*]}; do echo $i; done
a
dog
a
cat
a
fish
[me@linuxbox ~]$ for i in ${animals[@]}; do echo $i; done
a
dog
a
cat
a
fish
[me@linuxbox ~]$ for i in "${animals[*]}"; do echo $i; done
a dog a cat a fish
[me@linuxbox ~]$ for i in "${animals[@]}"; do echo $i; done
a dog
a cat
a fish

[me@linuxbox ~]$ a[100]=foo
[me@linuxbox ~]$ echo ${#a[@]} # number of array elements
1
[me@linuxbox ~]$ echo ${#a[100]} # length of element 100
3

${!array[*]}
${!array[@]}

[me@linuxbox ~]$ foo=([2]=a [4]=b [6]=c)
[me@linuxbox ~]$ for i in "${foo[@]}"; do echo $i; done
a
b
c
[me@linuxbox ~]$ for i in "${!foo[@]}"; do echo $i; done
2
4
6

+=

[me@linuxbox ~]$ foo=(a b c)
[me@linuxbox ~]$ echo ${foo[@]}
a b c
[me@linuxbox ~]$ foo+=(d e f)
[me@linuxbox ~]$ echo ${foo[@]}
a b c d e f

unset foo

unset 'foo[2]'

declare -A colors
colors["red"]="#ff0000"
colors["green"]="#00ff00"
colors["blue"]="#0000ff"

echo ${colors["blue"]}

36 – Exotica

{ ls -l; echo "Listing of foo.txt"; cat foo.txt; } | lpr

Process Substitution

<(list)
>(list)
read < <(echo "foo")
echo $REPLY

[me@linuxbox ~]$ echo <(echo "foo")
/dev/fd/63

#!/bin/bash
# trap-demo: simple signal handling demo
trap "echo 'I am ignoring you.'" SIGINT SIGTERM
for i in {1..5}; do
    echo "Iteration $i of 5"
    sleep 5
done

[me@linuxbox ~]$ trap-demo Iteration 1 of 5 Iteration 2 of 5
^CI am ignoring you. Iteration 3 of 5
^CI am ignoring you.
Iteration 4 of 5
Iteration 5 of 5

tempfile=$(mktemp /tmp/foobar.$$.XXXXXXXXXX)

88e9fe8514cd:~ shjabbar$ tempfile=$(mktemp /tmp/foobar.$$.XXXXXXXXXX)
88e9fe8514cd:~ shjabbar$ echo $tempfile
/tmp/foobar.28696.XnAEcahBws
88e9fe8514cd:~ shjabbar$

Asynchronous Execution

$! shell parameter, which will always contain the process ID of the last job put into the background.

Named Pipes

[me@linuxbox ~]$ mkfifo pipe1
[me@linuxbox ~]$ ls -l pipe1
prw-r--r-- 1 me me 0 2009-07-17 06:41 pipe1

Terminal 1:
[me@linuxbox ~]$ ls -l > pipe1

Terminal 2:
[me@linuxbox ~]$ cat < pipe1

Saturday, March 10, 2018

History of Western Philosophy, Russel: Ch-26 Cynics and Sceptics

10/11/2016
- Four schools of philosophy founded about time of Alexander
- Cynics, Sceptics, Stoics and Epicurean
- Antisthenes, disciple of Socrates - Cynic
-> Believed in return to nature
- Diogenes, disciple of Antisthenes - Cynic
-> decided to live like a dog, therefore called 'cynic' which mean 'canine'
- lived like Indian Fakir
- Scepticism, a doctrine of school by Pyrrho
- doubt everything
- In later years the academy adopted this view
- Sceptics returned back in modern age with rise of science.

History of Western Philosophy, Russel: Ch-25 Hellenistic World

10/9/2016
- Greek antiquity divided into three
1. free city state - freedom and disorder
2. Alexander and Philip - subjection and disorder
3. Roman empire - subjection and order
- Hellenistic age - second empire
- best work in science and math
- Alexander considered himself as God in Asia
- After Alexander's death his empire divided into three - under 3 generals
- European - Antigonus
- African - Ptolemy
- Asia - Seleucus
- Specialization characterized this age.

History of Western Philosophy, Russel: Ch-24 Early Greek Mathematics and Astronomy

10/3/2016
- Made great advances in mathematics and astronomy
- Pretty much everyone contributed
- geometry
- till Archimedes
- Helio centric system
- until roman conquered Greece and killed Archimedes