Linux CLInotes

D:\KINGSTON\csn08701 CS(QA)\Week
3\linux3CLInotes.docx 2
© Jim Jackson 3-Nov-19
Operating Systems
GNU/Linux & UNIX
Command Line Interface.
This is a quick reference document to aid students carrying out labs and
coursework. The document provides more detail about specific techniques
and commands than can be covered in a lecture.
Contents
grep - line based pattern matching ............................................................... 2
Regular Expression Basics ............................................................... 3
Input, Output & Error Redirection ..................................................................... 5

Pipes | .......................................................................................................... 6
Command $(ubstitution) ............................................................................... 6
Asynchronous vs Sequential Statements ( ; &) ............................................ 7
Linux Command Combinations ........................................................................ 8

sort - rearranging files or text output............................................................. 8
cut - isolating fields in text files ..................................................................... 9
uniq - remove adjacent matching lines ....................................................... 10
find - searching for objects in the filesystem ............................................... 11
man pages -command detail ...................................................................... 12
echo ........................................................................................................... 13
Linux Pathname Manipulation Commands .................................................... 13
dirname ...................................................................................................... 13
basename .................................................................................................. 13
readlink -f ................................................................................................... 13
diff - show differences between files........................................................... 14
cmp - check if two files are the same ......................................................... 14
md5sum - calculate a file's hash ................................................................ 15
1
Contents
CLI Commands
Complex actions can be created by combining command utilities together.
There are several important Linux commands which allow the user to create
composite actions that are more sophisticated or specific than can be
achieved with a single command on its own.
grep - line based pattern matching
grep is a very useful utility for filtering the content of a file or the output
produced by another command on a line by line basis. By default, grep
displays all lines which contain a specified pattern or string.
If is very useful if applied to large text files where the user is only interested in
a small number of lines, the irrelevant lines can be filtered out.
grep "string" filename
Example:
# grep fred userslist.txt
Lists only the lines from the file userslist.txt that contain the string ”fred”.
If the search string contains spaces it must be in double quotes:
# grep "Critical Error" install_log.txt
To omit lines containing a string and pass all other lines through, use the -v
option
# grep -v "Permission denied" filename
To specify that the target string must be at the start of a line it is prefixed by a
caret "^".
# grep "^The" mydocument.txt
The above would only show lines which begin “The….”

Note that the "^" character has a special meaning (the start of a line). The "$"
character can be used similarly to represent the end of a line.
# grep "Bye$" mydocument.txt
2
Contents
The special meaning of the "^" and "$" characters using in the target search
string are examples of regular expressions. Regular expression syntax can
be very sophisticated, however users can create their own effective search
expressions with a basic regular expression subset.
Note that, by default grep only supports basic regular expressions to use
the extended regular expression features the the ”-E” option must be
specified or alternatively use Extended Grep (egrep).
Regular Expressions
. (dot) matches any single character
[abc] matches any single character from the set "a ,b or c"
[^abc] matches any single character except "a ,b or c"
[A-Z] matches any single uppercase alphabetic character
[0-9] matches any single numeric digit
Mon|Tue matches the strings “Mon” OR “Tue” (Extended)
^ matches the start point of a line (when used inside [ ] negates)
$ matches the end point of a line
() Used to group (Extended)

matches the literal character "n" where "n" would normally
have special regex meaning.
\n Note the backslash is used as the escape character's special
effect.
Backslash escape examples:

\. Matches a literal "."
\$ Matches a literal "$"
\) Matches a closing bracket character
Example
(EH|FK)[0-9][0-9] Matches “EH” or “FK” followed by any two digits.
EH10 and FK12 would match but EH2 would not.
3
Contents
Expression Quantifiers
Often it is more convenient to match multiples.
In this case we follow the expression with a "*", "+", "?" or {n}
* matches any number (including none) of the previous item
Examples.
.* Matches any number of any character (i.e. everything)
[0-9]* Matches any string of digits (e.g. phone numbers)
+ matches one or more of the previous character (Extended)

? matches none or one of the previous character (Extended)
Matches exactly n of the previous item. (Extended)
{n}
Example
[A-Z]{3} Matches any uppercase string of 3 letters.
Combined Example
(EH|FK)[0-9][0-9] Matches “EH” or “FK” followed by one or more

digits. EH2 and FK12 would match but so would
EH7262542.
UNIX style regular expressions are used in many places in computing

wherever pattern matching or string substitution is required.
There is an excellent on-line regular expression testing tool here:-

http://regexpal.com
4
Contents
Input, Output & Error Redirection

Normally, if the user does not specify otherwise, a command takes its input
from the console (keyboard) and delivers its output to the console (screen).
Try running the “cat” command without any parameters, just “cat” on its own.
You will find that whatever you type gets repeated on the screen. The cat
command is reading its input from the keyboard and outputting it to the
screen, you can terminate this by pressing <Ctrl><D>.
To redirect screen output to a file use:-
# command > filename sends command output to “filename”

OR
# command >> filename appends output to “filename”
Example
# ls ~ >> homelist.txt
To take input from a file rather than the keyboard:-
# command < filename take input from “filename”
Error messages produced by commands ( permission denied, directory

missing etc.) can also be redirected.
Standard Error Redirection
# command 2> filename sends command error output to “filename”
# command 2>> filename appends error output to “filename”
Both standard error can also be combined with standard output so that the
whole can be redirected as one.
# command > comb.txt 2>&1
Here, error(2) output is merged in standard(1) output and the result is

redirected into the file comb.txt.
5
Contents
Pipes |
Another powerful technique is to take the output from one command or set of
commands, and use that as input to another. This is referred to as a “pipe”.
grep is useful as a command to "pipe" output through.
# command1 | command2 The output from command1 is used as input

to command2
# ls public_html | grep html
“ls” will list all the files in the directory public_html, grep will then filter this list
so that only lines containing the string “html” will pass through and be
displayed.
Command $(ubstitution)
The result of one command can also be embedded inside another.
For example, the pwd command returns the present working directory; if you
would like to search for any line mentioning this directory from a file then use:-
# grep `pwd` bigdirlist.log

(note the quote characters are backquotes not apostrophes)
Here pwd is run first and the result embedded in the command, so it becomes
grep /home/demo bigdirlist.log

before the grep operation is executed.
An alternative syntax uses $( ) to indicate when a command is to be

embedded.
Example:
# grep $(pwd) bigdirlist.log
6
Contents
Asynchronous vs Sequential
Statements ( ; &)
It is possible to enter multiple commands on a command line by separating
them with semi-colons.
This is useful if the user expects that one of the commands might take a long
while to complete.
Example:
# bigpayrolljob ; archivesystem ; getupdates
Each command will not start until the previous one has completed; This is a
“sequential” list.
Alternatively we could run commands simultaneously so that they are not

dependent on each other (asynchronous). To do that we follow the command
with an ampersand.
# bigpayrolljob & archivesystem & getupdates &

[1] 10355
[2] 10356
[3] 10357
#
Note: the number in square brackets is the "job" number and the number
following it the process ID (PID)
If “&” is put after a command it runs “in the background”, which means that the
command prompt comes back right way and another command can be
entered while the first is still running.
So the previous commands could just as easily be run as:-
# bigpayrolljob &
[1] 10355
# archivesystem &
[2] 10356
# getupdates &
[3] 10357
#
7
Contents
Linux Command Combinations

The following commands process data in different ways and are particularly
useful when used in conjunction with other commands and | pipes.
As is the case with most Unix/Linux commands there are a huge range of
options which can be used with each, in this documents only the most
immediately relevant are mentioned. For the detailed list refer to the "man"
page for the command.
All these commands can operate on files or accept input from a "pipe".
For example the following two commands are both valid and equivalent.
# cat biglist.txt | sort
# sort biglist.txt
sort - rearranging files or text

output
By default, sort will rearrange any data input or file it receives and output it in
alphabetical order line by line.
Example of the “ls” command output “piped” the output through sort.
# ls bigdirectory | sort
the output will be in alphabetical order
Sort can also reverse the sort order with “-r”
# ls bigdirectory | sort -r
If you would prefer a numerical sort so that 9 comes before 10 for example
use “-n”
# ls bigdirectory | sort -n
sort also does not have to index on the first character on each line, it can treat
the file as separate columns and sort the output on a particular column (or
"key") much like a spreadsheet application does.
8
Contents
By default sort expects columns to separated by any whitespace (i.e. tabs or
spaces).
Example: Sorting using the third column in the text file (-k opton).
# ls bigdirectory | sort -k3

Note that the left-most column (key) is 1 not 0.
If the columns are not delimited by spaces, then the column delimiting
character can be set using the "-t" option.
# ls mydatatable.txt | sort -k2 -t:
In the above example, sort assumes the file's data columns are separated by
colons ":" and sorts the output according to the text in column 2.
cut - isolating fields in text files

Whereas grep filters line-by-line cut isolates parts of a file vertically by
choosing specific fields (columns) from text files.
cut can select a particular character position from each line using the "-c"
option. For example, here only the 8th character on each line will be output.
The first character on a line is 1.
# cut -c8 mydatatable.txt
Alternatively a range of character positions can be used. Here the 8th to 15th
characters will be output.
# cut -c8-15 mydatatable.txt
Like sort, cut is most useful when isolating delimited columns (fields) from a
tabular data file. This is achieved using the "-f" option, however unlike sort the
default column delimiter for cut is the <tab> character, so if the source data's
columns are not separated by tabs then the delimiter must be specified using
the -d option.
For example if mylogfile.txt contains:-
1:Mon:10:12:start
2:Mon:11:08:run
3:Tues:07:21:end
Then the command:-
# cat mylogfile.txt | cut -f3 -d:
Would produce the result:-

10
11
07
9
Contents
As the command takes the third field from each line (-f3), as delimited with
colons (-d:).
Note that cut is also useful for processing single line output.
For example, in the following the filetype extension is extracted from a long
file pathname by using a dot as the field delimiter. This assumes a dot would
never appear in any other position.
# echo "/root/mydocs/mylogfile.mp3" | cut -f2 -d.
uniq - remove adjacent matching

lines
Often when viewing error logs or server access data the same message is
repeated multiple times. This is tedious to view as we would normally only be
interested in unique messages not repeated lines.
uniq solves this as any output passed to it will have consecutive identical lines
removed.
Example:
If we have a file called ipv4.txt containing the following.
192.168.1.5
192.168.1.5
192.168.1.2
192.168.1.3
192.168.1.3
192.168.1.4
Then uniq would produce the following output.
# uniq ipv4.txt
192.168.1.5
192.168.1.2
192.168.1.3
192.168.1.4
uniq only removes consecutive identical lines from output, in order to remove
all duplicate lines the output would first need to be put into order using the sort
command.
Example
# sort ipv4.txt | uniq
10
Contents
find - searching for objects in the
filesystem
The “find” utility is a very powerful command which allows any file or directory
to be found depending on any of its attributes. To use find you need to specify
where in the filesystem to start looking from.
Examples
Look anywhere within the present working directory (.) for a file or directory
named foobar.txt
# find . -name foobar.txt
Search the whole of your own public_html directory for “html” files, note that
the “*” wildcard must be escaped with a “\” in this instance.
# find ~/public_html -name “*.html”
Find any files in your home directory which were modified less than 7 days
ago.
# find ~ -type f -mtime -7
Find any files that are bigger than 100000 characters (bytes) which were last
modified more than 31 days ago.
# find / -type f -mtime +31 -size +100000c
find can search based of practically any attribute, some of the most common
are shown below.
OPTION DESCRIPTION
-atime days Time in days since the file was last accessed.
-ctime days Days since the file was changed.
(i.e. edit to the content, name or permissions)
-mtime days Number of days since the file's data was last
modified.
-newer file File was modified more recently than file.
-perm 755 File's permissions are exactly 755
-size size File uses size space, in 512-byte blocks.
Append size with `b' for bytes or `k' for kilobytes.
-type type File is type type, where type can be `d' for
directory or `f' for file
-user fred File is owned by user.fred
11
Contents
find can also execute a command on all files that it has found using the
“-exec” option.
Example, find any big old files and remove them, say files over 500Kbytes
which have not been modified within the past year.
# find / -size +500k -mtime +365 -exec rm {} \;
For every file found the command rm filename ; will be executed and the
files will be removed (deleted).
Note that the “{}” shows where the found filename will be inserted.
The exec command must be terminated with a semi-colon, which in this case
has to be escaped with a back slash.
Another Example
Search for any HTML files which have permissions rwx --- --- (700) and
change them to 744.
# find ~/pubic_html -name "*.html" -type f -perm 700

-exec chmod 744 {} \;
find is the useful for making any bulk changes to the contents of the file
system
man pages -command detail

The default method for accessing help information is by using “man pages”
# man command
Example
# man find
# man grep
This brings up comprehensive documentation on the command specified.

This is useful to remind yourself about the options available with a particular
command but it’s not much use if you don’t know which command to use in
the first place.
To save space Puppy Linux does not include man pages internally, instead
the man command invokes the web browser which accesses on-line man
pages.
With many commands you can also use the ”--help” switch get a brief
overview:- Example: # find --help
12
Contents
echo
“echo” merely echoes strings or values to the standard output (screen).

It is very useful for script messages, debugging or writing text to a file.
# echo Hello there

Hello there
# echo "this is the start of a run" >> mystuff.log

# date >> mystuff.log
Linux Pathname Manipulation

Commands
dirname
Parses a pathname by returning only the directory part.
# dirname /root/mydir/subdir/foobar
/root/mydir/subdir/
basename
This command parses a pathname by removing the directory part returning
only the filename.
# basename /root/mydir/subdir/foobar.txt
foobar.txt
readlink -f
This will provide an absolute pathname given a relative one.
# readlink -f ~
/root
# readlink -f ../spot/myfile.txt
/root/spot/myfile.txt
13
Contents
Linux File Comparison Commands
diff - show differences between files

This command reads two text files line by line and shows any differences
between the two files.
It is very handy for spotting changes that have been made to configuration
files.
# diff httpd.conf httpd.conf.orig
Example result:-
< Timeout 120
---
> Timeout 60
117,118c119,120
< StartServers 2
< MaxClients 150
---
> StartServers 4
> MaxClients 300
186a189
If you just need to confirm that two text files are the same, without seeing any
differences listed, use the -q switch.
# diff -q httpd.conf httpd.conf.orig

Example result:-
Files httpd.conf and httpd.conf_orig differ
cmp - check if two files are the

same
This command compares two binary files byte by byte and reports the position
of the first difference.
# cmp track01.mp3 shangalang.mp3
Example result:-
track01.mp3 shangalang.mp3 differ: byte 25, line 1
Note that cmp has a silent option "-s" which stops is outputting to standard
output. However it will still return its "exit status" to the shell, 0(true) if the files
are the same and 1(false) if they are different.
Command "exit status" is used for decision making in scripts.
14
Contents
md5sum - calculate a file's hash
This command reads any file and computes a 128bit “message digest” based
on the file content.
This 16byte (32 hex character) digest can be considered as a unique
signature for the file.
If the file content has been altered in any way, a completely different digest
value would be produced.
# md5sum wary.7_0_1.iso
Example result:-
577b7c1c22b8a972f2eb9c8bad0ae228 wary.7_0_1.iso
MD5 digest values are very useful for verifying file integrity or identifying if two
files are the same without having to compare them byte by byte.
File repository web sites often list a file’s MD5 signature alongside its
download link so that the user can verify the file’s integrity after they have
downloaded it by running md5sum on the downloaded file.
If MD5 signatures are widely published, it can make it difficult to pass off a
hacked version of a program as the original.
Note: Although it is not easy, it is possible that a hacked version of a file could
be engineered to have the same MD5 signature as the original. For this
reason MD5 sums should only be relied apon for the detection of non-
intentional corruption. In the latter case a Secure Hash Algorithm would be
preferred, e.g. sha1sum, sha256sum or sha512sum.
15
Contents

Linux CLInotes

Uploaded by

Copyright:

Available Formats

Linux CLInotes

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Linux CLInotes

Uploaded by

Copyright:

Available Formats

D:\KINGSTON\csn08701 CS(QA)\Week

Input, Output & Error Redirection ..................................................................... 5

Linux Command Combinations ........................................................................ 8

grep - line based pattern matching

grep "string" filename

If the search string contains spaces it must be in double quotes:

# grep "Critical Error" install_log.txt

# grep -v "Permission denied" filename

The above would only show lines which begin “The….”

[^abc] matches any single character except "a ,b or c"

[A-Z] matches any single uppercase alphabetic character

[0-9] matches any single numeric digit

Mon|Tue matches the strings “Mon” OR “Tue” (Extended)

^ matches the start point of a line (when used inside [ ] negates)

$ matches the end point of a line

() Used to group (Extended)

Backslash escape examples:

* matches any number (including none) of the previous item

+ matches one or more of the previous character (Extended)

(EH|FK)[0-9][0-9] Matches “EH” or “FK” followed by one or more

UNIX style regular expressions are used in many places in computing

There is an excellent on-line regular expression testing tool here:-

Input, Output & Error Redirection

To redirect screen output to a file use:-

# command > filename sends command output to “filename”

To take input from a file rather than the keyboard:-

# command < filename take input from “filename”

Error messages produced by commands ( permission denied, directory

Standard Error Redirection

# command 2> filename sends command error output to “filename”

# command 2>> filename appends error output to “filename”

# command > comb.txt 2>&1

Here, error(2) output is merged in standard(1) output and the result is

# command1 | command2 The output from command1 is used as input

# ls public_html | grep html

# grep `pwd` bigdirlist.log

grep /home/demo bigdirlist.log

An alternative syntax uses $( ) to indicate when a command is to be

# grep $(pwd) bigdirlist.log

# bigpayrolljob ; archivesystem ; getupdates

Alternatively we could run commands simultaneously so that they are not

# bigpayrolljob & archivesystem & getupdates &

Linux Command Combinations

# cat biglist.txt | sort

sort - rearranging files or text

the output will be in alphabetical order

Sort can also reverse the sort order with “-r”

# ls bigdirectory | sort -k3

# ls mydatatable.txt | sort -k2 -t:

cut - isolating fields in text files

# cut -c8-15 mydatatable.txt

# cat mylogfile.txt | cut -f3 -d:

Would produce the result:-

# echo "/root/mydocs/mylogfile.mp3" | cut -f2 -d.

uniq - remove adjacent matching

# find ~/pubic_html -name "*.html" -type f -perm 700

man pages -command detail

This brings up comprehensive documentation on the command specified.

“echo” merely echoes strings or values to the standard output (screen).