The most recent version of this page is a draft.DiffThis version (2017/10/18 11:42) is a draft.
Approvals: 0/1

This is an old revision of the document!

Linux Primer

  • Article written by Balázs Lengyel (VSC Team) <html><br></html>(last update 2017-10-11 by bl).

Filesystems

Filesystem 101

  1. Everything starts at the root
    • the root is a directory
    • “/” denotes the root directory
  2. the filesystem has different kinds of objects
    1. files
    2. directories
      • containers for multiple objects
    3. links to objects, which either
      • add a second name for the same object
      • point to a position in the filesystem
  3. objects can be referenced by their path
    • absolute: /dir1/dir2/object
    • relative: dir2/object
  4. special objects in directories:
    • . — is a reference to the directory itself
    • .. — is a reference to the parent directory
  5. the system may consist of multiple filesystems
    • filesystems may be mounted at any (empty) directory

Further concepts

  • Attributes
    • Ownership
    • Access rights
    • Filesystem limits
      • Size
    • Timestamps
  • Special files
    • device
    • fifo pipe
    • socket
  • Advanced FS features
    • data integrity
    • device managment
    • subvolume support

Filesystem tree

linux_directories.jpg

<HTML> <!– ### Mountpoints

Linux presents only one tree for file access - but every filesystems has its own tree!

#### Attaching a filesystem to the current tree: {.incremental}

- takes a filesystem
- `FS` usually a device file or a network address
- takes an empty directory in the current tree
- usually `/mnt/XXX` or `/mnt`
- makes the root of the new filesystem available in place of the empty directory
- `FS` `→ /mnt/`

<div class=incremental>

This process is called mounting.

</div> –> </HTML>

Special filesystems used on VSC

NFS

  • old and reliable network filesystem
  • much slower than any local filesystem
  • simultaneous usage possible

TMPFS

  • very fast filesystem
  • uses RAM instead of other media
  • lost at shutdown

Shell

Prompt

This is how the prompt looks by default:

[myname@l3_ ~]$
  • tells you:
    • who you are
    • which computer you're on
    • which directory you're in
  • can be configured
    • variable $PS1
    • default: echo $PS1
Ways to get help when you're stuck:
  • man
    • manual for specified topic
    • find man-pages: apropos
  • whatis
  • info
  • most commands support a -h/--help flag
  • colleagues are often helpful
  • the internet is often helpful

Execution

To execute a program, we call it: 

gcc FizzBuzz.c -o FizzBuzz
./FizzBuzz
module load non-existent-module
echo $?
  • Every command that is executed will provide a return-value on exit.
    • A value of 0 means success
    • any other value means failure

History

Your shell keeps a log of all the commands you executed.
  • the history command is used to access this history
  • for fast reuse of commands try the <CTRL>-R keys or the <Up-Arrow>

Parameters

For most commands you can combine multiple single-character arguments. This doesn't change the meaning of the parameters, but is limited to single-character arguments which dont take parameters. 

# works:
COMMAND -a -b -c -d -e
COMMAND -abcde
 
# works partially:
COMMAND -j 2 -a -b
COMMAND -j 2 -ab
 
# doesn't work:
COMMAND --long-parameter --flag-parameter=<FLAGVALUE>

One thing to look out for is the order of parameters. Most of the time no specific order is required, but you should look out for things like copying the target over the source file. 

# order does matter here:
COMMAND <SRC> <DEST>

Aliases

Whenever you have a command

  1. run often and it
    • has a long list of parameters you always use
    • is dangerous
  2. which is an aggregate of many other commands
    • but you don't want to write a script

you can define an alias for it: 

alias ll='ls -alh'
alias rm ='rm -i'
alias myProject='cd $ProjectDir; testSuite; compile && testSuite; cd -'

Patterns

Patterns and expansions define multiple arguments with little overhead:
  • the most important patterns are:
    • ? — matches one character
    • * — matches any character sequence
  • the most important expansions are:
    • A{1,9}Z — expands to A1Z A9Z
    • A{1..9}Z — expands to A1Z A2Z … A9Z
ls file.???
ls *.*
echo {{A..Z},{a..z}}
echo {{A,B},{X,Y}}
echo {A..Z},{a..z}
echo {A,B}{X,Y}

Regular Expressions

Used in many programs, to define matching rules for filters or other text transfomations:
.+
\.
(A|a)p{2}le
^[^aeiouAEIOU]+$
Match any character, once or more
match a dot
apple, Apple
any line of only non-vowels
Example usage of regex:
egrep -i '(cluster|vsc)' <FILE>

For a detailed explanation see Wikipedia, or try regex101.

If you want to challenge yourself, try Regex Crossword!

Control Flow

Chaining Commands

false ;  echo "Should I be Printed?"
false && echo "Should I be Printed?"
false || echo "Should I be Printed?"
Should I be Printed?
 
Should I be Printed?

Loops

for i in *
do
  mv $i{,.bak}
done
while true
do
  echo "Annoying Hello World"
  sleep 3
done
for i in *; do mv $i{,.bak}; done
while true; do echo "Annoying Hello World"; sleep 3; done

Streams

Redirects

Write output to a file or file-descriptor
CommandRedirect Append Description
program> std.log>> std.logredirect stdout to a file
program2> err.log2>> err.logredirect stderr to a file
program2>&1 redirect stderr to stdout

Pipes

Write output into the input-stream of another process
CommandPipe Description
program| grep -i foo pipe stdout into grep
program| tee file1 file2overwrite files and stdout
program| tee -a file append to files and stdout

Environment Variables

Setting, getting and unsetting

Set

LANG=en_US.UTF-8 bash
export LANG=en_US.UTF-8

Get

env
echo ${LANG}
echo $PWD

Unset

unset LANG
env -u LANG

Use cases

Some variables that could affect you are:
$EDITOR            # the default editor for the CLI
$PAGER             # utility to read long streams
$PATH              # program paths, in priority order
if you're aiming for programming, these could be more interesting:
$LIBRARY_PATH      # libraries to link by the compiler
$LD_LIBRARY_PATH   # libraries to link at runtime
$CC                # sometimes used to set default C compiler
$CFLAGS            # default flags for compiling C

<HTML> <div class=“incremental”> <div> <hr color=“lightgrey” class=“slidy” /> </HTML>

if you have a lot of self-compiled binaries:
export PATH="./:$HOME/bin/:$PATH"

<HTML> </div> </div> </HTML>

Basic command line programs

Looking around

Looking through files is done with the ls command under Linux, which is an abreviation for list.

By default (meaning without parameters) the ls command shows the content in the current directory. It does hide elements starting with a dot though. These hidden elements can be shown by adding the -a parameter, which stands for all. If you want detailed information about the elements, add the -l parameter for long listing. You can also specify a file/directory which you want to examine, by just appending the path to the command. 

$ ls      # shows files and directories
testdir  test

$ ls -a   # includes hidden ones
.  ..  testdir  test

$ ls -l   # detailed view
drwxr-xr-x 1 myuser p12345 0 Apr 13  2017 testdir
-rw-r--r-- 1 myuser p12345 4 Apr 13 11:55 test

$ ls /tmp/
allinea-USERNAME
ssh-6E553lWZCM
ssh-C6a754pJ1d
systemd-private-a4214393983d448fbdc689791806519c-ntpd.service-LrAgBP
tmp7CJZRA
yum_save_tx.2017-04-03.12-07.VCUowf.yumtx

$ ls -alh ~
drwxr-xr-x  5 myuser mygroup   45 Jan 30  2017 .allinea
-rw-------  1 myuser mygroup  21K May  3 17:14 .bash_history
-rw-r-----  1 myuser mygroup  231 Dec  2  2016 .bashrc
drwx------  1 myuser mygroup   76 Aug 22  2017 Simulation
drwx------  2 myuser mygroup   76 Dec 12  2016 .ssh

Moving around

Moving through directories is done with the cd command, which stands for change directory.

The cd command can be called without any arguments, in which case it just switches to the home directory. Otherwise it takes a absolute (starting with a dash) or relative path as an argument and switches to that directory. The argument - (just a single dash) will cause cd to switch to the previous directory. This can be used to alternate between two directories without typing their path's every time. 

$ cd /bin   # go to an absolute directory

$ cd [~]    # go home

$ cd -      # go to previous directory

Copying & moving files around

Copying and moving files and directories is done with the cp and mv commands, which stand for copy and move respectively.

Both commands take at least two parameters, which correspond to the <SOURCE> and <DESTINATION> files or directories. For cp to work with directories as a source, it needs the -r (recursive) or -a (archive) flag.

Beware of the pitfalls! you can overwrite data and therefor lose it, without getting any confirmation prompt! If in doubt use -i (interactive) flag.
$ mv old   new     # rename old to new

$ mv old   dir/    # move old into dir

$ mv file1 file2   # overwrite file2 with file1
                   # (BEWARE)
$ cp -i input input.bak   # input to input.bak

$ cp -i input backup/     # input into backup

$ cp -a dir1/ dir2        # exact copy of dir1

Creating and deleting

directories

creating directories is done with the mkdir command, which stands for make directory.

The mkdir command takes an optional -p (parents) parameter and a path. When optional parameter is given, it will create all the ancestors of the specified directory aswell. Otherwise the command fails if the directory either exists already or its parent doesn't exist either.

deleting directories is done with the rmdir command, which stands for remove directory.

This command removes the specified directory, but only if it's already empty. it can also take an optional -p parameter, in which case it removes the specified directory and all the ancestors you included.

Finding stuff

To look at everything
  1. in your home directory
  2. and nested up to three levels deep inside it
  3. that ends in .txt
  4. or starts with log_
  5. and is an ordinary file
  6. concatenated as one stream:
find \
  ~ \
  -maxdepth 3 \
  -iname "*.txt" \
  -or -iname "log_*" \
  -type f \
  -exec cat '{}' \; | less
find \
  ~ \
  -maxdepth 3 \
  -iname "*.txt" \
  -or -iname "log_*" \
  -type f \
  -exec cat '{}' +  | less

Contents of files

viewing is done by the less command:
concatenating is done by the cat command:
$ less file.txt      # exit with 'q'

$ less -R file.txt   # keep colors

$ cat file1 file2 | less
$ cat file

$ cat -A printable

$ cat -n numbered
$ echo "VSC is great" > file
$ cat file
VSC is great
$ echo "VSC is awesome" >> file
$ cat file
VSC is great
VSC is awesome
$ cat file | grep awesome
VSC is awesome
$ grep awesome file
VSC is awesome

Space accounting

viewing used space is done by the du (disk usage) command:
viewing free space is done by the df (disk free) command:
$ du -h file1 file2   # human readable output

$ du -s dir           # summarize
$ df -h       # human readable output

$ df -t nfs   # only list filesystems of a type

Recap

$ mv space.log space.log.bak
$ df -h | grep "lv12345\|lv54321" > space.log
$ cat space.log
nfs05.ib.cluster:/e/lv12345    200G  185G   16G  93% /home/lv12345
nfs04.ib.cluster:/e/lv54321   1000G  979G   22G  98% /home/lv54321
we do this often, let's wrap it up!

Recap++

$ mv space.log space.log.bak
$ df -h | grep "lv12345\|lv54321" > space.log
$ cat space.log
nfs05.ib.cluster:/e/lv12345    200G  185G   16G  93% /home/lv12345
nfs04.ib.cluster:/e/lv54321   1000G  979G   22G  98% /home/lv54321
we do this often, let's wrap it up!
$ echo '#!/bin/bash' > spacelog.sh

$ echo 'mv space.log space.log.bak' >> spacelog.sh

$ echo 'df -h | grep "lv12345\|lv54321" > space.log' >> spacelog.sh

$ echo 'cat space.log' >> spacelog.sh
$ chmod +x spacelog.sh
$ ./spacelog.sh

Sed and awk

sed (stream editor) and awk are powerful tools when working with the command line
$ mycommand | sed "..."
$ mycommand | awk '{...}'
Using sed and awk in action
program command description
sed s/old/new/ replace old with new
sed /from/,/to/ s/old/new/ replace old with new, between from and to
awk 'print $5 $3' print columns 5 and 3 of every line

Example script: 

#!/bin/bash
 
mv space.log space.log.bak
df -h | grep "lv12345\|lv54321" > space.log
cat space.log
#!/bin/bash
 
mv space.log space.log.bak
df -h | grep "lv12345\|lv54321" > space.log
cat space.log | sed "s|/home/lv12345|ProjectA|" \
              | awk '{print $6, "free:", $4}' \
              | column -t

<HTML> <!– ### Sort and uniq

sort and uniq (unique) are used to sort and uniquify adjacent lines

–> </HTML>

Scripting

Ownership and Permissions

Just to ensure that you are able to run your scripts

chown

Change the owner of files and directories by:
chown -R user:group dirs files
# only works with root privilages

chmod

Change the mode of files and directories by:
chmod -R u=rwx,g+w,o-rwx dirs files
chmod 640                files
chmod 750                dirs
chmod 750                executables

Shebang

A little test program, which we mark as executable and hand it over to the corresponding interpreter: 

cat << EOF > test.sh
 
echo "${LANG}"
echo "${PATH}"
EOF
chmod +x test.sh
bash test.sh
Don't we have an OS, capable of executing everything it recognises as an executable?
Yes, we do!
cat << EOF > test.sh
#!/bin/bash
echo "${LANG}"
echo "${PATH}"
EOF
chmod +x test.sh
./test.sh

Functions (more like procedures)

Programming in bash would be cumbersome without functions, so here we go: 

allNumbersFromTo () {
  echo "1 2 3"
}
This isn't good, as were only getting a fixed amount of numbers. Let's try a recursive approach:
allNumbersFromTo () {
  num=$1
  max=$2
  echo "${num}"
  if [ $num -lt $max ]; then
    allNumbersFromTo "$(($num + 1))" $max
  fi
}
allNumbersFromTo () {
  min=$1
  max=$2
  for num in $(seq $min $max)
  do
    echo "${num}"
  done
}
allNumbersFromTo 1 10

Editors

General

  • Many different editors
    • Unique (dis-)advantages
    • Different look and feel
  • Editors should provide us with
    1. Simple text editing
    2. Copy and paste
    3. Search and replace
    4. Saving changes
    5. Wide availability

Two editors that satisfy our needs:

  • nano
  • vim

Common starting point

nano filename
vim filename

Nano

Nano explained

This editor is focused on being easy to use, but still providing every feature a user might need.

Interface

The interface consists of four parts, namely from top to bottom:
  • Title bar
  • Text area
  • Command line
  • Key bindings

Usage

Nothing special, key-bindings visible while editing
Feature Usage
Navigation Arrow keys
Actual editing Typing text, as usual
Cut/Paste line <CTRL>+k / <CTRL>+u
… explained in key bindings field

Short

Use this editor if you are new to the command line.

It is straight forward, but can be extended on the way.
  • Auto-indentation
  • Syntax highlighting
  • Multi-buffer

Vi(m)

Vi(m) explained

This editor is focused on productivity and efficiency, providing everything a user might need.

Interface

The simple interface consists of two parts:
  • Text area
  • Command line
Since this editor is very easy to extend, after setting up a few plugins, it will probably look quite different!

Usage

This is a multimode editor, you'll have to switch modes whenever you change what you want to do.
Feature Usage
Navigation Arrow keys
Writing change to input mode, then write as usual
Commands exit current mode, press :
… explained on next slide

Short

Use this editor if you like a challenge.

It is fast and very nice — but you'll sometimes get hurt on the way.
  • Auto-indentation, Syntax highlighting, Multi-buffer – just like nano
  • File/Project Management
  • Use a plugin manager

Vi(m) modes and keys

  • any mode:
    • back to the default mode: <ESC>
  • command mode (followed by <RETURN>):
    • save current file: w [filename]
    • quit the editor
      • after saving: q
      • without confirmation: q!
    • help: h [topic], e.g. h tutorial
    • search and replace: %s/old/new/gc
  • default mode:
    • enter input mode: i
    • enter command mode: : (colon)
    • mark
      • character-wise: v
      • line-wise: <SHIFT>-v
    • delete
      • character-wise: x
      • line-wise: dd
      • marked content: d
    • search: /abc

VI Reference

.bashrc

.bashrc

# .bashrc
 
# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
 
# User specific aliases and functions
alias sq='squeue -u $USER'
alias rm='rm -i'
 
export PATH="./:$HOME/bin:$PATH"

<HTML> <!– ## Legal {.slidy}

### Copyleft & Copyright {.slidy}

<div class=slidy> - wikipedia — “Sadegh 1990 hosseini”: - [CC-Attribution-ShareAlike (CC-BY-SA)](https://creativecommons.org/licenses/by-sa/4.0/deed.en) - [Directory Tree](https://commons.wikimedia.org/wiki/File:Linux_directories.jpg) - VSC — Lengyel Balazs: - No explicit license - Screenshots </div> –> </HTML>

  • pandoc/introduction-to-vsc/03_linux_primer/linuxprimer.1508326936.txt.gz
  • Last modified: 2017/10/18 11:42
  • by pandoc