This post is the continuation of A Simple Tutorial on Linux – Part-1

In the Part-1 we learned the following topics on Linux.

• Linux Operating System
• Linux Files & Process
• The Directory Structure
• Permissions
• Process

Keeping up the same pace, we will learn the following topics in the 2nd part of the Linux series.

• Shell Scripting
• Networking
• Files & Directories
• Chaining Unix Commands
• Pipes
• Filters
• Word Count Exercise
• Special System commands
• Environment variables

Writing first shell script

A shell script is a file containing a list of commands. Let’s create a simple command that prints two words:

1. Open a text editor to create a file myfirstscript.sh:

nano myfirstscript.sh

2. Write the following into the editor:

#!/bin/bash
name=linux
echo "hello $name world"

Note: In Unix, the extension doesn’t dictate the program to be used while executing a script. It is the first line of the script that would dictate which program to use. In the example above, the program is “/bin/bash” which is a Unix shell.

1. Press Ctrl +x to save and then “y” to exit

2. Now, by default, it would not have executable permission. You can make it executable like this:

chmod +x myfirstscript.sh

3. To run the script, use:

./myfirstscript.sh

Program returns value and arguments

Every program returns a value to the operating system. It is also referred as the exit status. In Linux, a program returns 0 if successful. Otherwise a non-zero error number.

To check the return value of the previous command you can check the value of a special variable “?”

 echo $?

Attend the quiz on echo command

Networking

Sockets and ports

There are some programs that need to keep running all the time. Such programs are called services.

You can communicate with such programs even from another computer. These programs or processes bind themselves to a port. No two programs can listen on the same port.

For example, usually, A web server listens on 80 port and an email server listens on 25 port.

You can connect and talk to any service using nc command in the following way:

 nc computer_name(or ip_address) port_number

The way a person refers to himself as “I”, a computer refers itself as localhost or IP Address 127.0.0.1.

Files & Directories

Linking

If you want to give multiple names to a single file without copying the content, you can create the link.

1. Create a file:

nano orig_text

2. Put the following text in it:

This is my valuable data

3. Save and Exit: Press CTRL+x and y

4. Check if the file has right data:

cat orig_text

5. Create Link:

ln orig_text mylink

6. Check if it is there:

ls -l mylink

Hard Links and Softlink

The link that we created previously were known as hard link. If you delete the original file, the hard link would not be impacted.

1. Delete original content created in the previous section:

rm orig_text

2. Check if it still has the content:

cat mylink

Symbolic links

A symbolic link points to a file. In case, the original file is deleted, the symbolic link would be pointing to the non-existing file.

You can create a symbolic link to a directory too.

1. Create a file:

nano orig_text1

2. Put the following text in it:

This is my not-so-valuable data

3. Save and Exit: Press Ctrl+x and y

4. Check if the file has right data:

cat orig_text1

5. Create Link:

ln -s orig_text1 myslink

6. Check if it is there:

ls -l myslink

You should see something like this

lrwxrwxrwx 1 sandeepgiri9034 sandeepgiri9034 10 Dec 14 11:45 myslink -> orig_text1

7. Check the contents of myslink using:

cat myslink

Chaining Unix Commands

Say, you want to execute a command only if another command is successful then you use &&.

And if you want a command to be executed if another command has failed, you use ||.

1. Create a file tea_ready:

touch tea_ready

2. The following command would print “Tea is ready” if the tea_ready file exists:

ls -l tea_ready && echo "Tea is ready"

3. Delete the file tea_ready:

rm tea_ready

4. Check the command from #2 again:

ls -l tea_ready && echo "Tea is ready"

5. The following will print “Tea is not ready”:

ls -l tea_ready || echo "Tea is not ready"

6. You can use brackets to avoid confusion:

(ls -l tea_ready && echo "Tea is ready") || echo "Tea is not ready"

Redirecting the output of a program

The output of a program can be saved to a file:

 myprogram > myfile

If “myfile” does not exist, it will be created. If it exists it will be overwritten.

Please follow these steps:

1. Run the following command to save the output of echo to a file:

echo "hello" > hello.out

2. Append world to it:

echo "world" >> hello.out

3. Check by typing:

cat hello.out

Pipes – Introduction

If we want to send the output of one program to another, we can use pipe. A pipe is denoted by “|”.

“echo” command prints on the standard output whatever argument is passed to it.

echo "Hi"

“wc” command prints the number of characters, words, and lines out of whatever you type on standard input. Start “wc” command, type some text and press Ctrl+d to end the input:

wc
hi
how are you
[CTRL + d]

Output-

     2       4      15

Now, if we want to count the number of words, characters in the output of any program, we can pipe the output in the following way:

 echo "Hello, World" | wc

You can also save the results using redirection, in the following way:

echo "Hello, World" | wc > wc_results

Filters

Pipes are very powerful. They can be chained to solve many problems. So, most of the commands are built in a way that they can be used with pipes as long as they read from standard input (keyboard or pipe) and write to standard output (screen or pipe )

Such programs that can be used with pipes are generally called filters. Command examples of filters are:

wc – for counting the letters, words, and lines in the input

grep – displays only the lines from the input in which keyword (which is passed as argument) is found.

sort – sorts/orders the input lines lexically (alphabetically) by default but can be changed

more – displays the input in a page-wise manner

cat – displays the content of the file passed as an argument

sed – substitute a word with another word: sed ‘s/word/another_word/’

tr – translate character ranges. For example to lowercase characters in input you can use:

tr 'A-Z' 'a-z'

uniq – Display the uniq input lines. The input lines need to be sorted. If you want to display frequency, use:

uniq -c

Word Count Exercise

Step 1:

Check the Data using cat command. Since the file is big, you can use “more” to see pagewise

 cat /cxldata/big.txt | more

Step 2:

Replace space with newline such that every line in output contains the only single word:

 cat /cxldata/big.txt | sed 's/ /\n/g' |more

For example, after replacing space with a new line in “I am ok” we should get:

I
am
ok

The “/g” is an option of sed which makes replace all occurrences of space instead of only one.

Also, note this command has three programs connected by two pipes. The output of cat is going to sed and output of sed is going to more to see the pagewise.

Step 3:

We can sort the words using sort command in the following way

cat /cxldata/big.txt | sed 's/ /\n/g' | sort|more

Note that we are using “more” command just to avoid screen-blindness (too much text scrolling).

Step 4:

We can now, count the words using uniq command 

cat /cxldata/big.txt | sed 's/ /\n/g' | sort|uniq -c|more

Please save the result of the command to a file “word_count_results” in your home directory 

cat /cxldata/big.txt | sed 's/ /\n/g' | sort|uniq -c > word_count_results

Improved Word Count Using Unix Commands

We can further improve the word frequency count by using more filters.

Improvement 1:

Translate to lower case using 

tr 'A-Z' 'a-z'

Improvement 2:

Remove non-alphanumeric characters using sed with the regular expression:

sed 's/[^0-9a-z]//g'

Improvement 3:

Replace all whitespace (multiple tabs and spaces):

 sed -E 's/[ \t]+/\n/g'

Please note that since we are using regular expressions, we need to specify “-E”

Improvement 4:

Display most frequent at the top or display the results in reverse numeric sorting:

sort -nr

Improvement 5:

If the input file is big, the sort command might use too much of memory. So, you can force sort command to use less memory say 100 MB:

 sort -S 50M

After all of these improvements, please save the results

cat /cxldata/big.txt |tr 'A-Z' 'a-z'| sed -E 's/[ \t]+/\n/g'|sed 's/[^0-9a-z]//g' | sort|uniq -c|sort -nr -S 50M > word_count_results_nice

Shell script for WordCount

A shell script is a file which contains the commands separated by a newline.

Let’s create a script to do the sorting of the data:

1. Create a file using nano text editor:

nano wordcountscript.sh

2. The first line of a script should have “#!” followed by the name of the program to execute the script with. Since we are creating a shell script, we want it to be executed using bash. So, the first line of the program should be:

#!/bin/bash

3. Add the command in the editor:

tr 'A-Z' 'a-z'| sed -E 's/[ \t]+/\n/g'|sed 's/[^0-9a-z]//g' | sort|uniq -c|sort -nr -S 50M

4. Save the file by pressing Ctrl+x and “y”

5. Now, make this file executable:

chmod +x wordcountscript.sh

6. Check if it is running:

cat /cxldata/big.txt | ./wordcountscript.sh | more

Please recall “./” means current directory and ” | more” will show the result pagewise instead of causing too much scrolling.

Also, note that whatever is the input to the script is also passed to the programs executed in the script.

Permissions of Processes – setuid

In Unix, there is a file /etc/shadow that contains (one-way) encrypted passwords for every user. The user can not see the contents of the file. This is to defend the password cracking programs.

To change the password, a user needs to use the command: passwd. This passwd command first asks you for your old password and encrypts your input and compares it against the value in the file /etc/shadow. If it matches then it updates the password file /etc/shadow with new content.

When you are not allowed to view the /etc/shadow file, how can a program (passwd) do the same when run by you?

This is where the idea of a special permission called setuid come into the picture. A program file can be given setuid permission such that program becomes the user who owns the program file instead of the user who is running it.

Usually, a program runs with the same permissions as the user who is running it. The program can read or modify only the files which user is allowed to.

A process or program is something that is running. A process is started by the user. The question is what is a process allowed to do? Can a process do something that the user is not allowed to do? If so, then the user will create programs which can do anything. Therefore, the process is allowed to do only the things an owner of the process can do. And who is the owner? The user that started the process.

Setting setuid

You can make a program setuid by giving ‘s’ instead of ‘x’ permission. If you have written a script x.sh, an example would be:

chmod +s x.sh

1. Start creating a file with the name whoownsit_username.sh in /tmp directory:

nano /tmp/whoownsit_$USER.sh

Note: If your username is sandeep1234, the filename would be whoownsit_sandeep1234.sh

2. Put the following content in the editor in the previous step:

whoami

3. Save it by pressing Ctrl+x and press “y”

4. Give it setuid permission:

chmod +sx /tmp/whoownsit_$USER.sh

5. Check if you have given correct permission by

ls -l /tmp/whoownsit_$USER.sh

It should display something like this in permissions: rwsrwsr_x

Note: Setuid doesn’t work in shell scripts. Please see
How can I get setuid shell scripts to work?

Attend the quiz on permissions

Special System commands

sudo

This command makes a program run as root (the system administrator). This command is only allowed to be used by few users. Such users are called sudoers. You can modify the sudoers using

sudo visudo

shutdown

This command makes the system shutdown. You can use the following command to shutdown the system immediately

shutdown -h now

The alternative command to shutdown immediately is: “halt”

Restart the system

To restart the system, you can use “reboot” command.

Please note that above commands (shutdown, halt, reboot) can only be run as root.

Where is my program?

To find where is your program located, you can use “which” command.

For example,

which java

would print “/usr/bin/java” which means java is a command in the directory “/usr/bin”.

To further find out, you can use:

ls -l /usr/bin/java

This would display:

lrwxrwxrwx 1 root root 22 May 18  2016 /usr/bin/java -> /etc/alternatives/java

It means that /usr/bin/java is actually a link to /etc/alternatives/java

Let us try:

ls -l /etc/alternatives/java

It should display something like:

lrwxrwxrwx 1 root root 72 May 18  2016 /etc/alternatives/java -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-0.b14.el7_2.x86_64/jre/bin/java

Further, to find out about the content of a file, you can use “file” command:

file /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-0.b14.el7_2.x86_64/jre/bin/java

This should display something like

/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-0.b14.el7_2.x86_64/jre/bin/java: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=bb856d3fd5d21b9d8c9dd63b5c781bd8f9eccb87, stripped

This means it is Linux binary.

Environment variables

Unix shell provides environment variables. To see the entire list of environment variables, use:

set

These environment variables can be used by the shell, programs or commands.

For example, the $PS1 variable is used by the shell to display the prompt. To change your prompt, you can try:

PS1='xxx>>'

The environment variable $PATH is a list of directories separated by a colon. It is used by the shell to find the file corresponding to a command.

Setting Environment variables

You can set the environment variables simply by assignment: MYVAR=VAL

The following list of commands:

MYV=myfile

ls $MYV

are equivalent to the single command:

ls myfile