Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revisionLast revisionBoth sides next revision | ||
pandoc:introduction-to-vsc:03_linux_primer:linuxprimer [2017/11/07 12:57] – Pandoc Auto-commit pandoc | pandoc:introduction-to-vsc:03_linux_primer:linuxprimer [2020/10/20 08:09] – Pandoc Auto-commit pandoc | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | |||
====== Linux Primer ====== | ====== Linux Primer ====== | ||
Line 5: | Line 6: | ||
==== Note ==== | ==== Note ==== | ||
- | Please use the '' | + | Please use the '' |
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | ---- | ||
Line 12: | Line 21: | ||
==== Filesystem 101 ==== | ==== Filesystem 101 ==== | ||
- | The job of filesystems is to keep data in a structured way. Every filesystem has a filesystem root, directories and files. The filesystem root is a special object which acts as an entrypoint | + | The job of filesystems is to keep data in a structured way. Every filesystem has a filesystem root, directories and files. The filesystem root is a special object which acts as an entry-point |
Two main concepts emerged for using more than one filesystem on a single machine: | Two main concepts emerged for using more than one filesystem on a single machine: | ||
Line 18: | Line 27: | ||
< | < | ||
< | < | ||
- | < | + | < |
< | < | ||
< | < | ||
- | Linux' | + | Linux’ virtual filesystem tree represents all the files and directories that are reachable from the system. The nice part is that you can work on a Linux machine and not care about whether your file is on the network or on a local filesystem. The main difference for users is the performance delivered by different filesystems. |
This is how the (virtual) filesystem looks on Linux: | This is how the (virtual) filesystem looks on Linux: | ||
Line 30: | Line 39: | ||
- Everything starts at the root | - Everything starts at the root | ||
* the root is a directory | * the root is a directory | ||
- | * "'' | + | * “'' |
- the filesystem has different kinds of objects | - the filesystem has different kinds of objects | ||
- files | - files | ||
Line 42: | Line 51: | ||
* relative: '' | * relative: '' | ||
- special objects in directories: | - special objects in directories: | ||
- | * '' | + | * “'' |
- | * '' | + | * “'' |
- the system may consist of multiple filesystems | - the system may consist of multiple filesystems | ||
* filesystems may be mounted at any (empty) directory | * filesystems may be mounted at any (empty) directory | ||
Line 51: | Line 60: | ||
Next to basic storage and organization of data filesystems have different properties and functionality. Most filesystems provide a way to store and access attributes, different kinds of special files and some filesystems provide various advanced features: | Next to basic storage and organization of data filesystems have different properties and functionality. Most filesystems provide a way to store and access attributes, different kinds of special files and some filesystems provide various advanced features: | ||
- | |||
* Attributes | * Attributes | ||
Line 69: | Line 77: | ||
* Advanced FS features | * Advanced FS features | ||
* data integrity | * data integrity | ||
- | * device | + | * device |
- | * subvolume | + | * sub-volume |
Line 76: | Line 85: | ||
==== Filesystem tree ==== | ==== Filesystem tree ==== | ||
- | {{pandoc: | + | {{..: |
- | < | ||
- | <!-- | ||
- | ### Mountpoints | ||
- | |||
- | > Linux presents only one tree for file access - but every filesystems has its own tree! | ||
- | |||
- | #### Attaching a filesystem to the current tree: {.incremental} | ||
- | |||
- | > - takes a filesystem | ||
- | > - **`FS`** usually a device file or a network address | ||
- | > - takes an empty directory in the current tree | ||
- | > - usually `/mnt/XXX` or `/mnt` | ||
- | > - makes the root of the new filesystem available in place of the empty directory | ||
- | > - **`FS`** `-> /mnt/` | ||
- | |||
- | <div class=incremental> | ||
- | > This process is called mounting. | ||
- | </ | ||
- | --> | ||
- | </ | ||
==== Special filesystems used on VSC ==== | ==== Special filesystems used on VSC ==== | ||
Line 117: | Line 106: | ||
The home directory of the user is located on an NFS filesystem, which ensures that all parts of the cluster have a consistent view of files. | The home directory of the user is located on an NFS filesystem, which ensures that all parts of the cluster have a consistent view of files. | ||
- | The filesystem behind the '' | + | The filesystem behind the '' |
Line 126: | Line 115: | ||
> This is how the prompt looks by default: | > This is how the prompt looks by default: | ||
+ | < | ||
+ | <!-- ![](pictures/ | ||
+ | </ | ||
<code bash> | <code bash> | ||
[myname@l3_ ~]$ | [myname@l3_ ~]$ | ||
Line 133: | Line 125: | ||
* tells you: | * tells you: | ||
* who you are | * who you are | ||
- | * which computer you're on | + | * which computer you’re on |
- | * which directory you're in | + | * which directory you’re in |
* can be configured | * can be configured | ||
* variable '' | * variable '' | ||
* default: '' | * default: '' | ||
- | > Ways to get help when you're stuck: | + | > Ways to get help when you’re stuck: |
- | Most of the time a command doesn' | ||
- | | + | Most of the time a command doesn’t act as expected, it shows an error message. From this point you have multiple approaches: |
- | * Just copy that message into your favourite | + | |
+ | | ||
+ | * Just copy that message into your favorite | ||
* Most programs supports a '' | * Most programs supports a '' | ||
- | * '' | + | * '' |
- | * An alternative to '' | + | * An alternative to '' |
* Ask colleagues for help | * Ask colleagues for help | ||
Line 159: | Line 152: | ||
gcc FizzBuzz.c -o FizzBuzz | gcc FizzBuzz.c -o FizzBuzz | ||
./FizzBuzz | ./FizzBuzz | ||
- | module load non-existent-module | + | false |
echo $? | echo $? | ||
</ | </ | ||
+ | |||
The examples show compiling a program, executing the result, trying to load a module on our cluster and checking if the previous command succeeded. | The examples show compiling a program, executing the result, trying to load a module on our cluster and checking if the previous command succeeded. | ||
Line 167: | Line 161: | ||
* '' | * '' | ||
* '' | * '' | ||
- | * '' | + | * '' |
* '' | * '' | ||
=== History === | === History === | ||
+ | |||
+ | |||
> Your shell keeps a log of all the commands you executed. | > Your shell keeps a log of all the commands you executed. | ||
Line 194: | Line 190: | ||
== Combining parameters == | == Combining parameters == | ||
- | For most commands you can combine multiple single-character parameters. This doesn't change the meaning of the parameters, but is limited to single-character parameters which don't take extra arguments. | + | For most commands you can combine multiple single-character parameters. This doesn’t change the meaning of the parameters, but is limited to single-character parameters which don’t take extra arguments. |
<code bash> | <code bash> | ||
Line 210: | Line 206: | ||
COMMAND -j auto --color 2 # PROBABLY WRONG | COMMAND -j auto --color 2 # PROBABLY WRONG | ||
</ | </ | ||
- | == Escapes & Quotes == | + | ==== Escapes & Quotes |
Whenever a parameter has to contain a character that is either unprintable or reserved for the shell, you can use: | Whenever a parameter has to contain a character that is either unprintable or reserved for the shell, you can use: | ||
Line 227: | Line 223: | ||
COMMAND 'This is a single parameter' | COMMAND 'This is a single parameter' | ||
</ | </ | ||
+ | |||
+ | |||
+ | |||
==== Aliases ==== | ==== Aliases ==== | ||
Line 233: | Line 232: | ||
- | < | + | |
+ | < | ||
alias ll='ls -alh' | alias ll='ls -alh' | ||
alias rm='rm -i' | alias rm='rm -i' | ||
alias myProject=' | alias myProject=' | ||
</ | </ | ||
+ | |||
After that, you can use the aliases synonymously. | After that, you can use the aliases synonymously. | ||
Line 249: | Line 250: | ||
==== Patterns ==== | ==== Patterns ==== | ||
- | Patterns are an easy way of defining multiple arguments, which are mostly the same. The pattern will match anything in it's place. | + | |
+ | Patterns are an easy way of defining multiple arguments, which are mostly the same. The pattern will match anything in it’s place. | ||
The other concept is a expansion. In this case only defined patterns will be matched. | The other concept is a expansion. In this case only defined patterns will be matched. | ||
Line 256: | Line 258: | ||
* the most important patterns are: | * the most important patterns are: | ||
- | * **?** --- matches one character | + | * **?** — matches one character |
- | * ***** --- matches any character sequence | + | * ***** — matches any character sequence |
* the most important expansions are: | * the most important expansions are: | ||
- | * **A{1, | + | * **A{1, |
- | * **A{1..9}Z** | + | * **A{1..9}Z** |
Line 282: | Line 284: | ||
==== Regular Expressions ==== | ==== Regular Expressions ==== | ||
- | Often you need to specify some string, but patterns and expansions aren't enough, to cover all possibilities. In these cases you can use a regular expression also known as regex. These regexes are used by editors for search and replace, the '' | + | Often you need to specify some string, but patterns and expansions aren’t enough, to cover all possibilities. In these cases you can use a regular expression also known as regex. These regexes are used by editors for search and replace, the '' |
<code python> | <code python> | ||
Line 290: | Line 292: | ||
^[^aeiouAEIOU]+$ | ^[^aeiouAEIOU]+$ | ||
</ | </ | ||
+ | |||
For a detailed explanation see [[https:// | For a detailed explanation see [[https:// | ||
Line 306: | Line 309: | ||
=== Chaining Commands === | === Chaining Commands === | ||
- | The simplest mechanism for control flow is to chain commands together in a simple '' | + | The simplest mechanism for control flow is to chain commands together in a simple '' |
Line 343: | Line 346: | ||
while true; do echo " | while true; do echo " | ||
</ | </ | ||
+ | |||
=== Conditionals === | === Conditionals === | ||
Line 432: | Line 436: | ||
$PATH # program paths, in priority order | $PATH # program paths, in priority order | ||
</ | </ | ||
- | > if you're aiming for programming, | + | > if you’re aiming for programming, |
<code python> | <code python> | ||
Line 454: | Line 458: | ||
</ | </ | ||
</ | </ | ||
- | </ | ||
- | ===== Basic command line programs ===== | ||
- | |||
- | ==== Looking around ==== | ||
- | |||
- | > Looking through files is done with the '' | ||
- | |||
- | By default (meaning without parameters) the '' | ||
- | |||
- | < | ||
- | $ ls # shows files and directories | ||
- | testdir | ||
- | |||
- | $ ls -a # includes hidden ones | ||
- | . .. testdir | ||
- | |||
- | $ ls -l # detailed view | ||
- | drwxr-xr-x 1 myuser p12345 0 Apr 13 2017 testdir | ||
- | -rw-r--r-- 1 myuser p12345 4 Apr 13 11:55 test | ||
- | |||
- | $ ls /tmp/ | ||
- | allinea-USERNAME | ||
- | ssh-6E553lWZCM | ||
- | ssh-C6a754pJ1d | ||
- | systemd-private-a4214393983d448fbdc689791806519c-ntpd.service-LrAgBP | ||
- | tmp7CJZRA | ||
- | yum_save_tx.2017-04-03.12-07.VCUowf.yumtx | ||
- | |||
- | $ ls -alh ~ | ||
- | drwxr-xr-x | ||
- | -rw------- | ||
- | -rw-r----- | ||
- | drwx------ | ||
- | drwx------ | ||
- | </ | ||
- | |||
- | ==== Moving around ==== | ||
- | |||
- | > Moving through directories is done with the '' | ||
- | |||
- | The '' | ||
- | |||
- | < | ||
- | $ cd /bin # go to an absolute directory | ||
- | |||
- | $ cd [~] # go home | ||
- | |||
- | $ cd - # go to previous directory | ||
- | </ | ||
- | |||
- | ==== Copying & moving files around ==== | ||
- | |||
- | > Copying and moving files and directories is done with the '' | ||
- | |||
- | Both commands take at least two parameters, which correspond to the '' | ||
- | |||
- | > Beware of the pitfalls! you can overwrite data and therefor lose it, without getting any confirmation prompt! If in doubt use '' | ||
- | |||
- | < | ||
- | $ mv old | ||
- | |||
- | $ mv old | ||
- | |||
- | $ mv file1 file2 # overwrite file2 with file1 | ||
- | # (BEWARE) | ||
- | </ | ||
- | < | ||
- | $ cp -i input input.bak | ||
- | |||
- | $ cp -i input backup/ | ||
- | |||
- | $ cp -a dir1/ dir2 # exact copy of dir1 | ||
- | |||
- | </ | ||
- | |||
- | ==== Creating and deleting ==== | ||
- | |||
- | === directories === | ||
- | |||
- | > creating directories is done with the '' | ||
- | |||
- | The '' | ||
- | |||
- | > deleting directories is done with the '' | ||
- | |||
- | This command removes the specified directory, but only if it's already empty. it can also take an optional '' | ||
- | |||
- | |||
- | ==== Finding stuff ==== | ||
- | |||
- | > To look at everything | ||
- | |||
- | |||
- | - in your home directory | ||
- | - and nested up to three levels deep inside it | ||
- | - that ends in '' | ||
- | - or starts with '' | ||
- | - and is an ordinary file | ||
- | - concatenated as one stream: | ||
- | |||
- | <code bash> | ||
- | find \ | ||
- | ~ \ | ||
- | -maxdepth 3 \ | ||
- | -iname " | ||
- | -or -iname " | ||
- | -type f \ | ||
- | -exec cat ' | ||
- | </ | ||
- | <code bash> | ||
- | find \ | ||
- | ~ \ | ||
- | -maxdepth 3 \ | ||
- | -iname " | ||
- | -or -iname " | ||
- | -type f \ | ||
- | -exec cat ' | ||
- | </ | ||
- | |||
- | ==== Contents of files ==== | ||
- | |||
- | > **viewing** is done by the **'' | ||
- | |||
- | > **concatenating** is done by the **'' | ||
- | |||
- | |||
- | < | ||
- | $ less file.txt | ||
- | |||
- | $ less -R file.txt | ||
- | |||
- | $ cat file1 file2 | less | ||
- | </ | ||
- | < | ||
- | $ cat file | ||
- | |||
- | $ cat -A printable | ||
- | |||
- | $ cat -n numbered | ||
- | </ | ||
- | |||
- | < | ||
- | $ echo "VSC is great" > file | ||
- | $ cat file | ||
- | VSC is great | ||
- | </ | ||
- | < | ||
- | $ echo "VSC is awesome" | ||
- | $ cat file | ||
- | VSC is great | ||
- | VSC is awesome | ||
- | </ | ||
- | < | ||
- | $ cat file | grep awesome | ||
- | VSC is awesome | ||
- | </ | ||
- | < | ||
- | $ grep awesome file | ||
- | VSC is awesome | ||
- | </ | ||
- | |||
- | ==== Space accounting ==== | ||
- | |||
- | > viewing **used space** is done by the **'' | ||
- | |||
- | > viewing **free space** is done by the **'' | ||
- | |||
- | |||
- | < | ||
- | $ du -h file1 file2 # human readable output | ||
- | |||
- | $ du -s dir # summarize | ||
- | </ | ||
- | < | ||
- | $ df -h # human readable output | ||
- | |||
- | $ df -t nfs # only list filesystems of a type | ||
- | </ | ||
- | |||
- | {{pandoc: | ||
- | |||
- | ==== Recap ==== | ||
- | |||
- | < | ||
- | $ mv space.log space.log.bak | ||
- | </ | ||
- | < | ||
- | $ df -h | grep " | ||
- | </ | ||
- | < | ||
- | $ cat space.log | ||
- | </ | ||
- | < | ||
- | nfs05.ib.cluster:/ | ||
- | nfs04.ib.cluster:/ | ||
- | </ | ||
- | > we do this often, let's wrap it up! | ||
- | |||
- | |||
- | ==== Recap++ ==== | ||
- | |||
- | < | ||
- | $ mv space.log space.log.bak | ||
- | </ | ||
- | < | ||
- | $ df -h | grep " | ||
- | </ | ||
- | < | ||
- | $ cat space.log | ||
- | </ | ||
- | < | ||
- | nfs05.ib.cluster:/ | ||
- | nfs04.ib.cluster:/ | ||
- | </ | ||
- | > we do this often, let's wrap it up! | ||
- | |||
- | |||
- | < | ||
- | $ echo '# | ||
- | |||
- | $ echo 'mv space.log space.log.bak' | ||
- | |||
- | $ echo 'df -h | grep " | ||
- | |||
- | $ echo 'cat space.log' | ||
- | </ | ||
- | < | ||
- | $ chmod +x spacelog.sh | ||
- | </ | ||
- | < | ||
- | $ ./ | ||
- | </ | ||
- | |||
- | ==== Sed and awk ==== | ||
- | |||
- | > **sed** (stream editor) and **awk** are powerful tools when working with the command line | ||
- | |||
- | < | ||
- | $ mycommand | sed " | ||
- | </ | ||
- | < | ||
- | $ mycommand | awk ' | ||
- | </ | ||
- | |||
- | > Using sed and awk in action | ||
- | |||
- | ^program | ||
- | |sed | '' | ||
- | |sed | '' | ||
- | |awk | '' | ||
- | |||
- | Example script: | ||
- | |||
- | |||
- | <code bash> | ||
- | #!/bin/bash | ||
- | |||
- | mv space.log space.log.bak | ||
- | df -h | grep " | ||
- | cat space.log | ||
- | |||
- | |||
- | </ | ||
- | <code bash> | ||
- | #!/bin/bash | ||
- | |||
- | mv space.log space.log.bak | ||
- | df -h | grep " | ||
- | cat space.log | sed " | ||
- | | awk ' | ||
- | | column -t | ||
- | </ | ||
- | |||
- | < | ||
<!-- | <!-- | ||
- | ### Sort and uniq | + | ## Pause Please?< |
- | + | ||
- | > **sort** and **uniq** (unique) are used to sort and uniquify adjacent lines | + | |
--> | --> | ||
</ | </ | ||
Line 745: | Line 473: | ||
<code bash> | <code bash> | ||
+ | # only works with root privileges | ||
+ | |||
+ | chown user file | ||
chown -R user:group dirs files | chown -R user:group dirs files | ||
- | # only works with root privilages | ||
</ | </ | ||
=== chmod === | === chmod === | ||
Line 758: | Line 488: | ||
chmod 750 executables | chmod 750 executables | ||
</ | </ | ||
+ | |||
+ | * there are three bits for: | ||
+ | * read (4) | ||
+ | * write (2) | ||
+ | * execute (1) | ||
+ | * three times for: | ||
+ | * user | ||
+ | * group | ||
+ | * other | ||
+ | |||
+ | |||
==== Shebang ==== | ==== Shebang ==== | ||
Line 776: | Line 517: | ||
</ | </ | ||
- | > Don't we have an OS, capable of executing everything it recognises | + | > Don’t we have an OS, capable of executing everything it recognizes |
> Yes, we do! | > Yes, we do! | ||
Line 804: | Line 545: | ||
} | } | ||
</ | </ | ||
- | > This isn't good, as were only getting a fixed amount of numbers. Let's try a recursive approach: | + | > This isn’t good, as were only getting a fixed amount of numbers. Let’s try a recursive approach: |
<code bash> | <code bash> | ||
Line 830: | Line 571: | ||
allNumbersFromTo 1 10 | allNumbersFromTo 1 10 | ||
</ | </ | ||
- | |||
- | ===== Editors ===== | ||
- | |||
- | ==== General ==== | ||
- | |||
- | |||
- | * Many different editors | ||
- | * Unique (dis-)advantages | ||
- | * Different look and feel | ||
- | * Editors should provide us with | ||
- | - Simple text editing | ||
- | - Copy and paste | ||
- | - Search and replace | ||
- | - Saving changes | ||
- | - Wide availability | ||
- | |||
- | === Two editors that satisfy our needs: === | ||
- | |||
- | |||
- | * nano | ||
- | * vim | ||
- | |||
- | === Common starting point === | ||
- | |||
- | <code bash> | ||
- | nano filename | ||
- | </ | ||
- | <code bash> | ||
- | vim filename | ||
- | </ | ||
- | |||
- | |||
- | ==== Nano ==== | ||
- | |||
- | {{pandoc: | ||
- | |||
- | ==== Nano explained ==== | ||
- | |||
- | > This editor is focused on being easy to use, but still providing every feature a user might need. | ||
- | |||
- | === Interface === | ||
- | |||
- | > The interface consists of four parts, namely from top to bottom: | ||
- | |||
- | |||
- | * Title bar | ||
- | * Text area | ||
- | * Command line | ||
- | * Key bindings | ||
- | |||
- | === Usage === | ||
- | |||
- | > Nothing special, key-bindings visible while editing | ||
- | |||
- | ^Feature | ||
- | |Navigation | ||
- | |Actual editing| | ||
- | |Cut/Paste line| '' | ||
- | |... | ||
- | |||
- | |||
- | |||
- | === Short === | ||
- | |||
- | > Use this editor if you are new to the command line.\\ | ||
- | > It is straight forward, but can be extended on the way. | ||
- | |||
- | * Auto-indentation | ||
- | * Syntax highlighting | ||
- | * Multi-buffer | ||
- | |||
- | |||
- | {{pandoc: | ||
- | |||
- | ==== Vi(m) ==== | ||
- | |||
- | {{pandoc: | ||
- | |||
- | ==== Vi(m) explained ==== | ||
- | |||
- | > This editor is focused on productivity and efficiency, providing everything a user might need. | ||
- | |||
- | === Interface === | ||
- | |||
- | > The simple interface consists of two parts: | ||
- | |||
- | |||
- | |||
- | * Text area | ||
- | * Command line | ||
- | |||
- | > Since this editor is very easy to extend, after setting up a few plugins, it will probably look quite different! | ||
- | |||
- | |||
- | === Usage === | ||
- | |||
- | > This is a multimode editor, you'll have to switch modes whenever you change what you want to do. | ||
- | |||
- | ^Feature | ||
- | |Navigation| | ||
- | |Writing | ||
- | |Commands | ||
- | |... | ||
- | |||
- | |||
- | |||
- | === Short === | ||
- | |||
- | > Use this editor if you like a challenge.\\ | ||
- | > It is fast and very nice --- but you'll sometimes get hurt on the way. | ||
- | |||
- | * Auto-indentation, | ||
- | * File/ | ||
- | * Use a plugin manager | ||
- | |||
- | |||
- | {{pandoc: | ||
- | |||
- | ==== Vi(m) modes and keys ==== | ||
- | |||
- | * any mode: | ||
- | * back to the **default mode**: **'' | ||
- | * command mode (followed by '' | ||
- | * **save** current file: **'' | ||
- | * **quit** the editor | ||
- | * after saving: **'' | ||
- | * without confirmation: | ||
- | * **help**: **'' | ||
- | * **search and replace**: **'' | ||
- | * //default mode//: | ||
- | * enter **input mode**: **'' | ||
- | * enter **command mode**: **'' | ||
- | * **mark** | ||
- | * **character**-wise: | ||
- | * **line**-wise: | ||
- | * **delete** | ||
- | * **character**-wise: | ||
- | * **line**-wise: | ||
- | * **marked content**: **'' | ||
- | * **search**: **'' | ||
- | |||
- | |||
- | [[http:// | ||
===== .bashrc ===== | ===== .bashrc ===== | ||
Line 992: | Line 590: | ||
export PATH=" | export PATH=" | ||
</ | </ | ||
+ | |||
+ | |||
< | < | ||
<!-- | <!-- |