Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
pandoc:introduction-to-vsc:04_module_environment:01_modules [2018/01/31 11:10] – Pandoc Auto-commit pandoc | pandoc:introduction-to-vsc:04_module_environment:01_modules [2020/10/20 08:09] (current) – Pandoc Auto-commit pandoc | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== Software management ====== | ||
+ | |||
+ | * Article written by Markus Stöhr (VSC Team) < | ||
+ | |||
+ | |||
+ | |||
+ | ===== Considerations ===== | ||
+ | |||
+ | - Find a suitable **location and directory structure** for serveral versions of the same library/ | ||
+ | - Manage the **user environment** to be able to access the correct version automatically | ||
+ | - Manage different software stacks, e.g. use different compiler(-version) or different library versions to build another library (–> 1.) | ||
+ | |||
+ | |||
+ | ---- | ||
+ | |||
+ | ===== Location of compiled software ===== | ||
+ | |||
+ | < | ||
+ | mountpoint | ||
+ | └── processor architecture | ||
+ | │ | ||
+ | └── glibc (C-library) of OS | ||
+ | │ | ||
+ | └── processor type | ||
+ | │ | ||
+ | └──program name | ||
+ | | ||
+ | | ||
+ | |||
+ | </ | ||
+ | |||
+ | < | ||
+ | /opt/sw | ||
+ | └── x86_64 | ||
+ | │ | ||
+ | └── glibc-2.17 | ||
+ | │ | ||
+ | └── ivybridge-ep | ||
+ | │ | ||
+ | └──hdf5 | ||
+ | | ||
+ | | ||
+ | </ | ||
+ | |||
+ | |||
+ | |||
+ | ---- | ||
+ | |||
+ | ==== Details of hdf5/ | ||
+ | |||
+ | < | ||
+ | tree -d -L 5 / | ||
+ | </ | ||
+ | < | ||
+ | / | ||
+ | ├── gcc | ||
+ | │ └── 5.3 | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | └── intel | ||
+ | ├── 15 | ||
+ | │ └── intel-mpi | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | ├── 16 | ||
+ | │ └── intel-mpi | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | └── 17 | ||
+ | └── intel-mpi | ||
+ | └── 2017 | ||
+ | ├── bin | ||
+ | ├── include | ||
+ | ├── lib | ||
+ | └── share | ||
+ | </ | ||
+ | Software used for compiling is reflected in the directory structure | ||
+ | |||
+ | ===== Environment Modules ===== | ||
+ | |||
+ | * **environment**: | ||
+ | * http:// | ||
+ | * **module**: all modifications (set and prepend to variables) in the user environment necessary for using a piece of software | ||
+ | * handles additional software, which is not installed to default system paths | ||
+ | * makes different versions of one piece of software available | ||
+ | * **prerequisites**: | ||
+ | |||
+ | |||
+ | ---- | ||
+ | |||
+ | < | ||
+ | module avail | ||
+ | </ | ||
+ | * shows available modules. | ||
+ | * searches paths given by '' | ||
+ | |||
+ | * own modules can be added by extending '' | ||
+ | * If multiple modules with the same name/ | ||
+ | |||
+ | * two parts: | ||
+ | * software name (directory) | ||
+ | * version (file with definitions) | ||
+ | |||
+ | |||
+ | < | ||
+ | ls / | ||
+ | 1.6.10 | ||
+ | 1.8.12 | ||
+ | </ | ||
+ | |||
+ | |||
+ | |||
+ | ---- | ||
+ | |||
+ | < | ||
+ | module load < | ||
+ | </ | ||
+ | < | ||
+ | [markus@l33 ~]$ module load hdf5/ | ||
+ | |||
+ | You are not meeting the prereq of this module. Currently loaded modules: | ||
+ | None | ||
+ | |||
+ | The following modules combinations can be loaded: | ||
+ | (1): module load intel/15 intel-mpi/5 hdf5/ | ||
+ | (2): module load intel/16 intel-mpi/5 hdf5/ | ||
+ | (3): module load gcc/5.3 intel-mpi/5 hdf5/ | ||
+ | (4): module load intel/17 intel-mpi/ | ||
+ | |||
+ | |||
+ | </ | ||
+ | Do copy and paste: | ||
+ | |||
+ | < | ||
+ | module load intel/16 intel-mpi/5 hdf5/ | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | |||
+ | ---- | ||
+ | |||
+ | < | ||
+ | module list | ||
+ | </ | ||
+ | * list currently loaded modules | ||
+ | |||
+ | |||
+ | ---- | ||
+ | |||
+ | < | ||
+ | module purge | ||
+ | </ | ||
+ | * remove all loaded modules; | ||
+ | * recommended in job scripts before loading any modules (reproducibility) | ||
+ | |||
+ | < | ||
+ | module remove < | ||
+ | </ | ||
+ | * removes a list of modules | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | ---- | ||
+ | |||
+ | < | ||
+ | module show < | ||
+ | </ | ||
+ | * shows the variables added to your environment by a module | ||
+ | |||
+ | < | ||
+ | module show hdf5/ | ||
+ | / | ||
+ | |||
+ | conflict | ||
+ | setenv | ||
+ | |||
+ | prepend-path | ||
+ | prepend-path | ||
+ | setenv | ||
+ | |||
+ | prepend-path | ||
+ | prepend-path | ||
+ | setenv | ||
+ | |||
+ | prepend-path | ||
+ | setenv | ||
+ | |||
+ | setenv | ||
+ | setenv | ||
+ | setenv | ||
+ | setenv | ||
+ | ------------------------------------------------------------------- | ||
+ | </ | ||
+ | ===== Exercises ===== | ||
+ | |||
+ | * look at your user environment: | ||
+ | |||
+ | < | ||
+ | env | ||
+ | env |grep PATH | ||
+ | echo $PATH |tr ':' | ||
+ | echo $LD_LIBRARY_PATH |tr ':' | ||
+ | echo $MODULEPATH |tr ':' | ||
+ | </ | ||
+ | * inspect the contents of the directories in MODULEPATH | ||
+ | * look at directory structure of installed software, e.g.: | ||
+ | |||
+ | < | ||
+ | tree -d / | ||
+ | </ | ||
+ | * try the module commands | ||
+ | |||
+ | < | ||
+ | module avail | ||
+ | module load hdf5/ | ||
+ | module list | ||
+ | module show hdf5/ | ||
+ | module purge | ||
+ | </ | ||
+ | * compare '' | ||
+ | |||
+ | ===== Adding own gnuplot module ===== | ||
+ | |||
+ | * setup your own modules directory and a directory for gnuplot | ||
+ | |||
+ | < | ||
+ | mkdir ${HOME}/ | ||
+ | mkdir ${HOME}/ | ||
+ | </ | ||
+ | * prepend own module directory to '' | ||
+ | |||
+ | < | ||
+ | export MODULEPATH=${HOME}/ | ||
+ | </ | ||
+ | |||
+ | * create module file: '' | ||
+ | |||
+ | < | ||
+ | #%Module | ||
+ | set inst_base $env(HOME)/ | ||
+ | |||
+ | prepend-path PATH $inst_base/ | ||
+ | prepend-path LD_LIBRARY_PATH $inst_base/ | ||
+ | append-path CPATH $inst_base/ | ||
+ | setenv MYVAR test1234 | ||
+ | </ | ||
+ | |||
+ | |||
+ | ===== Exercises ===== | ||
+ | |||
+ | * check how module is loaded by default VSC-3 modules | ||
+ | |||
+ | < | ||
+ | module avail 2>& | ||
+ | module load gnuplot/ | ||
+ | module show gnuplot/ | ||
+ | |||
+ | </ | ||
+ | * create own module environment as described on previous slide, create a directory within your training sub-folder | ||
+ | * check again, how the module is loaded. | ||
+ | |||
+ | |||
+ | ===== Simple example ===== | ||
+ | |||
+ | eg. from HDF5: | ||
+ | |||
+ | < | ||
+ | EXAMPLE=$VSC_HDF5_ROOT/ | ||
+ | </ | ||
+ | |||
+ | |||
+ | Compile using Intel C Compiler ([[examples/ | ||
+ | |||
+ | < | ||
+ | module purge | ||
+ | module load intel/16 \ | ||
+ | intel-mpi/5 hdf5/ | ||
+ | |||
+ | cp $EXAMPLE . | ||
+ | mpiicc -lhdf5 ph5example.c -o ph5example | ||
+ | </ | ||
+ | |||
+ | |||
+ | Compile using Gnu C Compiler ([[examples/ | ||
+ | |||
+ | < | ||
+ | module purge | ||
+ | module load gcc/5.3 \ | ||
+ | intel-mpi/5 hdf5/ | ||
+ | |||
+ | cp $EXAMPLE . | ||
+ | mpicc -lhdf5 ph5example.c -o ph5example | ||
+ | </ | ||
+ | |||
+ | |||
+ | |||
+ | If needed add e.g. include directory explicitly: | ||
+ | |||
+ | < | ||
+ | mpiicc -I$VSC_HDF5_INCLUDE -lhdf5 ph5example.c -o ph5example | ||
+ | </ | ||
+ | |||
+ | |||
+ | |||
+ | ---- | ||
+ | |||
+ | ===== Compiler names ===== | ||
+ | |||
+ | Hint: | ||
+ | |||
+ | * mpiifort | ||
+ | * mpifort | ||
+ | * mpicc | ||
+ | * mpiicc | ||
+ | |||