Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
pandoc:introduction-to-vsc:03_linux_primer:linuxprimer [2018/01/31 11:10] – Pandoc Auto-commit pandoc | pandoc:introduction-to-vsc:03_linux_primer:linuxprimer [2020/10/20 09:13] (current) – Pandoc Auto-commit pandoc | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | |||
+ | ====== Linux Primer ====== | ||
+ | |||
+ | * Article written by Balázs Lengyel (VSC Team) < | ||
+ | |||
+ | ==== Note ==== | ||
+ | |||
+ | Please use the '' | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | ---- | ||
+ | |||
+ | |||
+ | |||
+ | ===== Filesystems ===== | ||
+ | |||
+ | ==== Filesystem 101 ==== | ||
+ | |||
+ | The job of filesystems is to keep data in a structured way. Every filesystem has a filesystem root, directories and files. The filesystem root is a special object which acts as an entry-point for using that filesystem. All other objects (directories and files) are structured below the root. | ||
+ | |||
+ | Two main concepts emerged for using more than one filesystem on a single machine: | ||
+ | |||
+ | < | ||
+ | < | ||
+ | < | ||
+ | < | ||
+ | < | ||
+ | |||
+ | Linux’ virtual filesystem tree represents all the files and directories that are reachable from the system. The nice part is that you can work on a Linux machine and not care about whether your file is on the network or on a local filesystem. The main difference for users is the performance delivered by different filesystems. | ||
+ | |||
+ | This is how the (virtual) filesystem looks on Linux: | ||
+ | |||
+ | |||
+ | |||
+ | - Everything starts at the root | ||
+ | * the root is a directory | ||
+ | * “'' | ||
+ | - the filesystem has different kinds of objects | ||
+ | - files | ||
+ | - directories | ||
+ | * containers for multiple objects | ||
+ | - links to objects, which either | ||
+ | * add a second name for the same object | ||
+ | * point to a position in the filesystem | ||
+ | - objects can be referenced by their path | ||
+ | * absolute: '' | ||
+ | * relative: '' | ||
+ | - special objects in directories: | ||
+ | * “'' | ||
+ | * “'' | ||
+ | - the system may consist of multiple filesystems | ||
+ | * filesystems may be mounted at any (empty) directory | ||
+ | |||
+ | |||
+ | ==== Further concepts ==== | ||
+ | |||
+ | Next to basic storage and organization of data filesystems have different properties and functionality. Most filesystems provide a way to store and access attributes, different kinds of special files and some filesystems provide various advanced features: | ||
+ | |||
+ | * Attributes | ||
+ | * Ownership | ||
+ | * Access rights | ||
+ | * Filesystem limits | ||
+ | * Size | ||
+ | * Timestamps | ||
+ | |||
+ | |||
+ | * Special files | ||
+ | * device | ||
+ | * fifo pipe | ||
+ | * socket | ||
+ | |||
+ | |||
+ | * Advanced FS features | ||
+ | * data integrity | ||
+ | * device management | ||
+ | * sub-volume support | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | ==== Filesystem tree ==== | ||
+ | |||
+ | {{.: | ||
+ | |||
+ | ==== Special filesystems used on VSC ==== | ||
+ | |||
+ | === NFS === | ||
+ | |||
+ | |||
+ | * old and reliable network filesystem | ||
+ | * much slower than any local filesystem | ||
+ | * simultaneous usage possible | ||
+ | |||
+ | === TMPFS === | ||
+ | |||
+ | |||
+ | * very fast filesystem | ||
+ | * uses RAM instead of other media | ||
+ | * lost at shutdown | ||
+ | |||
+ | |||
+ | The home directory of the user is located on an NFS filesystem, which ensures that all parts of the cluster have a consistent view of files. | ||
+ | |||
+ | The filesystem behind the '' | ||
+ | |||
+ | |||
+ | ===== Shell ===== | ||
+ | |||
+ | ==== Prompt ==== | ||
+ | |||
+ | > This is how the prompt looks by default: | ||
+ | |||
+ | < | ||
+ | <!-- ![](pictures/ | ||
+ | </ | ||
+ | <code bash> | ||
+ | [myname@l3_ ~]$ | ||
+ | </ | ||
+ | |||
+ | |||
+ | * tells you: | ||
+ | * who you are | ||
+ | * which computer you’re on | ||
+ | * which directory you’re in | ||
+ | * can be configured | ||
+ | * variable '' | ||
+ | * default: '' | ||
+ | |||
+ | > Ways to get help when you’re stuck: | ||
+ | |||
+ | |||
+ | Most of the time a command doesn’t act as expected, it shows an error message. From this point you have multiple approaches: | ||
+ | |||
+ | * Think about why the program failed - maybe you (un-)intentionally tried to force the program to do something it’s not intended for? | ||
+ | * Just copy that message into your favorite search engine and don’t forget to remove the parts that are specific to your environment (e.g. directory and user names). | ||
+ | * Most programs supports a '' | ||
+ | * '' | ||
+ | * An alternative to '' | ||
+ | * Ask colleagues for help | ||
+ | |||
+ | |||
+ | |||
+ | ==== Execution ==== | ||
+ | |||
+ | To execute a program, we call it: | ||
+ | |||
+ | <code python> | ||
+ | gcc FizzBuzz.c -o FizzBuzz | ||
+ | ./FizzBuzz | ||
+ | false | ||
+ | echo $? | ||
+ | </ | ||
+ | |||
+ | |||
+ | The examples show compiling a program, executing the result, trying to load a module on our cluster and checking if the previous command succeeded. | ||
+ | |||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | |||
+ | |||
+ | === History === | ||
+ | |||
+ | |||
+ | |||
+ | > Your shell keeps a log of all the commands you executed. | ||
+ | |||
+ | |||
+ | * the '' | ||
+ | * for fast reuse of commands try the '' | ||
+ | |||
+ | |||
+ | ==== Parameters ==== | ||
+ | |||
+ | > The default way to apply parameters to a program is to write a space separated list of parameters after the program when calling it. | ||
+ | |||
+ | These parameters are either | ||
+ | |||
+ | - Single-character | ||
+ | - Multi-character | ||
+ | - Strings | ||
+ | |||
+ | where some parameters also take additional arguments. | ||
+ | |||
+ | == Combining parameters == | ||
+ | |||
+ | For most commands you can combine multiple single-character parameters. This doesn’t change the meaning of the parameters, but is limited to single-character parameters which don’t take extra arguments. | ||
+ | |||
+ | <code bash> | ||
+ | COMMAND -j 2 -a -b -c | ||
+ | COMMAND -j 2 -abc | ||
+ | </ | ||
+ | == Ordering parameters == | ||
+ | |||
+ | One thing to look out for is the order of parameters. Most of the time no specific order is required, but you should look out for things like copying the target over the source file. Also watch out to keep parameters and their arguments together. | ||
+ | |||
+ | <code bash> | ||
+ | COMMAND <SRC> < | ||
+ | COMMAND < | ||
+ | COMMAND -j 2 --color auto # OK | ||
+ | COMMAND -j auto --color 2 # PROBABLY WRONG | ||
+ | </ | ||
+ | ==== Escapes & Quotes ==== | ||
+ | |||
+ | Whenever a parameter has to contain a character that is either unprintable or reserved for the shell, you can use: | ||
+ | |||
+ | - Backslash escape: | ||
+ | * Escapes a character, that would have a special meaning | ||
+ | * Can be used inside of quotes | ||
+ | - Double Quotes: | ||
+ | * Similar to escaping all whitespace characters | ||
+ | - Single Quotes: | ||
+ | * Additionally prevents expansion of variables | ||
+ | |||
+ | <code bash> | ||
+ | COMMAND This\ is\ a\ single\ parameter | ||
+ | COMMAND "This is a single parameter" | ||
+ | COMMAND 'This is a single parameter' | ||
+ | </ | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | ==== Aliases ==== | ||
+ | |||
+ | You can define aliases in your shell. These are usually used to shorten names for commands which are used often with a fixed set of parameters or where you have to be careful to get things right. These aliases are accessable as if they were commands. | ||
+ | |||
+ | |||
+ | |||
+ | < | ||
+ | alias ll='ls -alh' | ||
+ | alias rm='rm -i' | ||
+ | alias myProject=' | ||
+ | </ | ||
+ | |||
+ | |||
+ | After that, you can use the aliases synonymously. | ||
+ | |||
+ | <code python> | ||
+ | ll # Same as 'ls -alh' | ||
+ | rm # Same as 'rm -i' | ||
+ | myProject # Same as 'cd $ProjectDir; | ||
+ | </ | ||
+ | |||
+ | ==== Patterns ==== | ||
+ | |||
+ | |||
+ | Patterns are an easy way of defining multiple arguments, which are mostly the same. The pattern will match anything in it’s place. | ||
+ | |||
+ | The other concept is a expansion. In this case only defined patterns will be matched. | ||
+ | |||
+ | |||
+ | |||
+ | * the most important patterns are: | ||
+ | * **?** — matches one character | ||
+ | * ***** — matches any character sequence | ||
+ | * the most important expansions are: | ||
+ | * **A{1,9}Z** — expands to A1Z A9Z | ||
+ | * **A{1..9}Z** — expands to A1Z A2Z … A9Z | ||
+ | |||
+ | |||
+ | You can try these commands and see what they do. These are all totally safe, even if you modify the arguments. | ||
+ | |||
+ | |||
+ | <code bash> | ||
+ | ls file.??? | ||
+ | ls *.* | ||
+ | </ | ||
+ | <code bash> | ||
+ | echo {{A..Z}, | ||
+ | echo {{A, | ||
+ | </ | ||
+ | <code bash> | ||
+ | echo {A..Z}, | ||
+ | echo {A,B}{X,Y} | ||
+ | </ | ||
+ | |||
+ | |||
+ | ==== Regular Expressions ==== | ||
+ | |||
+ | Often you need to specify some string, but patterns and expansions aren’t enough, to cover all possibilities. In these cases you can use a regular expression also known as regex. These regexes are used by editors for search and replace, the '' | ||
+ | |||
+ | <code python> | ||
+ | .+ # Match any character, once or more | ||
+ | \. # match a dot | ||
+ | (A|a)p{2}le | ||
+ | ^[^aeiouAEIOU]+$ | ||
+ | </ | ||
+ | |||
+ | |||
+ | For a detailed explanation see [[https:// | ||
+ | |||
+ | If you want to challenge yourself, try [[https:// | ||
+ | |||
+ | ==== Control Flow ==== | ||
+ | |||
+ | In the shell language there are a few ways to organize the execution path. The most important ones are: | ||
+ | |||
+ | - chaining of commands | ||
+ | - looping constructs | ||
+ | - conditionals (if/case) | ||
+ | |||
+ | |||
+ | === Chaining Commands === | ||
+ | |||
+ | The simplest mechanism for control flow is to chain commands together in a simple '' | ||
+ | |||
+ | |||
+ | <code bash> | ||
+ | false ; echo " | ||
+ | false && echo " | ||
+ | false || echo " | ||
+ | </ | ||
+ | <code bash> | ||
+ | Should I be Printed? | ||
+ | |||
+ | Should I be Printed? | ||
+ | </ | ||
+ | |||
+ | === Loops === | ||
+ | |||
+ | The other way to execute commands conditionally are loops. You can loop over files, numeric arguments, until a either the loop condition is false or a break is encountered. | ||
+ | |||
+ | |||
+ | <code bash> | ||
+ | for i in * | ||
+ | do | ||
+ | mv $i{,.bak} | ||
+ | done | ||
+ | </ | ||
+ | <code bash> | ||
+ | while true | ||
+ | do | ||
+ | echo " | ||
+ | sleep 3 | ||
+ | done | ||
+ | </ | ||
+ | |||
+ | <code bash> | ||
+ | for i in *; do mv $i{,.bak}; done | ||
+ | while true; do echo " | ||
+ | </ | ||
+ | |||
+ | |||
+ | === Conditionals === | ||
+ | |||
+ | |||
+ | |||
+ | '' | ||
+ | |||
+ | |||
+ | <code bash> | ||
+ | if [ $VARIABLE1 ] | ||
+ | then | ||
+ | COMMAND1 | ||
+ | elif [ $VARIABLE2 ] | ||
+ | COMMAND2 | ||
+ | else | ||
+ | COMMAND3 | ||
+ | fi | ||
+ | </ | ||
+ | '' | ||
+ | |||
+ | |||
+ | <code bash> | ||
+ | case $VARIABLE in | ||
+ | [0-9] | [1-2][0-9]) | ||
+ | COMMAND1 | ||
+ | ;; | ||
+ | *) | ||
+ | COMMAND2 | ||
+ | ;; | ||
+ | esac | ||
+ | </ | ||
+ | |||
+ | ==== Streams ==== | ||
+ | |||
+ | === Redirects === | ||
+ | |||
+ | > Write **output** to a **file** or **file-descriptor** | ||
+ | |||
+ | ^Command^Redirect | ||
+ | |program|'' | ||
+ | |program|'' | ||
+ | |program|'' | ||
+ | |||
+ | |||
+ | === Pipes === | ||
+ | |||
+ | > Write **output** into the **input**-stream of another process | ||
+ | |||
+ | ^Command^Pipe | ||
+ | |program|'' | ||
+ | |program|'' | ||
+ | |program|'' | ||
+ | |||
+ | |||
+ | ===== Environment Variables ===== | ||
+ | |||
+ | ==== Setting, getting and unsetting ==== | ||
+ | |||
+ | === Set === | ||
+ | |||
+ | <code python> | ||
+ | LANG=en_US.UTF-8 bash | ||
+ | export LANG=en_US.UTF-8 | ||
+ | </ | ||
+ | |||
+ | === Get === | ||
+ | |||
+ | <code python> | ||
+ | env | ||
+ | echo ${LANG} | ||
+ | echo $PWD | ||
+ | </ | ||
+ | |||
+ | === Unset === | ||
+ | |||
+ | <code python> | ||
+ | unset LANG | ||
+ | env -u LANG | ||
+ | </ | ||
+ | |||
+ | ==== Use cases ==== | ||
+ | |||
+ | > Some variables that could affect you are: | ||
+ | |||
+ | <code python> | ||
+ | $EDITOR | ||
+ | $PAGER | ||
+ | $PATH # program paths, in priority order | ||
+ | </ | ||
+ | > if you’re aiming for programming, | ||
+ | |||
+ | <code python> | ||
+ | $LIBRARY_PATH | ||
+ | $LD_LIBRARY_PATH | ||
+ | $CC # sometimes used to set default C compiler | ||
+ | $CFLAGS | ||
+ | </ | ||
+ | |||
+ | < | ||
+ | <div class=" | ||
+ | <div> | ||
+ | <hr color=" | ||
+ | </ | ||
+ | > if you have a lot of self-compiled binaries: | ||
+ | |||
+ | <code python> | ||
+ | export PATH=" | ||
+ | </ | ||
+ | < | ||
+ | </ | ||
+ | </ | ||
+ | <!-- | ||
+ | ## Pause Please?< | ||
+ | --> | ||
+ | </ | ||
+ | ===== Scripting ===== | ||
+ | |||
+ | ==== Ownership and Permissions ==== | ||
+ | |||
+ | > Just to ensure that you are able to run your scripts | ||
+ | |||
+ | === chown === | ||
+ | |||
+ | > Change the owner of files and directories by: | ||
+ | |||
+ | <code bash> | ||
+ | # only works with root privileges | ||
+ | |||
+ | chown user file | ||
+ | chown -R user:group dirs files | ||
+ | </ | ||
+ | === chmod === | ||
+ | |||
+ | > Change the mode of files and directories by: | ||
+ | |||
+ | <code bash> | ||
+ | chmod -R u=rwx, | ||
+ | chmod 640 files | ||
+ | chmod 750 dirs | ||
+ | chmod 750 executables | ||
+ | </ | ||
+ | |||
+ | * there are three bits for: | ||
+ | * read (4) | ||
+ | * write (2) | ||
+ | * execute (1) | ||
+ | * three times for: | ||
+ | * user | ||
+ | * group | ||
+ | * other | ||
+ | |||
+ | |||
+ | ==== Shebang ==== | ||
+ | |||
+ | A little test program, which we mark as executable and hand it over to the corresponding interpreter: | ||
+ | |||
+ | <code bash> | ||
+ | cat << EOF > test.sh | ||
+ | |||
+ | echo " | ||
+ | echo " | ||
+ | EOF | ||
+ | </ | ||
+ | <code bash> | ||
+ | chmod +x test.sh | ||
+ | </ | ||
+ | <code bash> | ||
+ | bash test.sh | ||
+ | </ | ||
+ | |||
+ | > Don’t we have an OS, capable of executing everything it recognizes as an executable? | ||
+ | |||
+ | > Yes, we do! | ||
+ | |||
+ | <code bash> | ||
+ | cat << EOF > test.sh | ||
+ | #!/bin/bash | ||
+ | echo " | ||
+ | echo " | ||
+ | EOF | ||
+ | </ | ||
+ | <code bash> | ||
+ | chmod +x test.sh | ||
+ | </ | ||
+ | <code bash> | ||
+ | ./test.sh | ||
+ | </ | ||
+ | |||
+ | |||
+ | ==== Functions (more like procedures) ==== | ||
+ | |||
+ | Programming in bash would be cumbersome without functions, so here we go: | ||
+ | |||
+ | <code bash> | ||
+ | allNumbersFromTo () { | ||
+ | echo "1 2 3" | ||
+ | } | ||
+ | </ | ||
+ | > This isn’t good, as were only getting a fixed amount of numbers. Let’s try a recursive approach: | ||
+ | |||
+ | <code bash> | ||
+ | allNumbersFromTo () { | ||
+ | num=$1 | ||
+ | max=$2 | ||
+ | echo " | ||
+ | if [ $num -lt $max ]; then | ||
+ | allNumbersFromTo " | ||
+ | fi | ||
+ | } | ||
+ | </ | ||
+ | <code bash> | ||
+ | allNumbersFromTo () { | ||
+ | min=$1 | ||
+ | max=$2 | ||
+ | for num in $(seq $min $max) | ||
+ | do | ||
+ | echo " | ||
+ | done | ||
+ | } | ||
+ | </ | ||
+ | |||
+ | <code bash> | ||
+ | allNumbersFromTo 1 10 | ||
+ | </ | ||
+ | |||
+ | ===== .bashrc ===== | ||
+ | |||
+ | ==== .bashrc ==== | ||
+ | |||
+ | <code bash> | ||
+ | # .bashrc | ||
+ | |||
+ | # Source global definitions | ||
+ | if [ -f /etc/bashrc ]; then | ||
+ | . /etc/bashrc | ||
+ | fi | ||
+ | |||
+ | # User specific aliases and functions | ||
+ | alias sq=' | ||
+ | alias rm='rm -i' | ||
+ | |||
+ | export PATH=" | ||
+ | </ | ||
+ | |||
+ | |||
+ | < | ||
+ | <!-- | ||
+ | ## Legal {.slidy} | ||
+ | |||
+ | ### Copyleft & Copyright {.slidy} | ||
+ | |||
+ | <div class=slidy> | ||
+ | - wikipedia --- " | ||
+ | - [CC-Attribution-ShareAlike (CC-BY-SA)](https:// | ||
+ | - [Directory Tree](https:// | ||
+ | - VSC --- Lengyel Balazs: | ||
+ | - No explicit license | ||
+ | - Screenshots | ||
+ | </ | ||
+ | --> | ||
+ | </ | ||