This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong. ====== SPACK - migration to setup without environment ====== ===== Motivation: problems of SPACK environments ===== Having worked with //spack environments// for some time, we have encountered several severe issues which have convinced us that we need to find a more practical way of maintaining software packages on VSC. In particular we came across the following drawbacks: * hard to read and lengthy output of ''spack find'' commands. * long response times of all ''spack'' commands when the number of installed packages grows. * difficult and for practical purposes impossible maintenance of a consistent database of software packages (''spack.yaml'', ''spack.lock'' files). * unresolvable discrepancies between the list of software packages found in spack as compared to those found in modules. * unreliable concretisation procedure (discrepancies between what is shown in ''spack spec -I ..'' and what is actually installed with ''spack install ...''). ==== New approach without environments ==== There are now three separate spack installation trees corresponding to the CPU/GPU architectures on VSC: * skylake - Intel CPUs; works on Intel Skylake and Cascadelake CPUs * zen - AMD CPUs; works on Zen 2 and 3 CPUs * cuda-zen - AMD CPUs + NVIDIA GPUs; works on all nodes equipped with graphics cards By default the spack installation tree suitable for the current compute/login node is activated and will be indicated by a **prefix** on the command line, e.g.: <code> zen [user@l51 ~]$ </code> <box 80% round blue|Info> Please note that the prefix (e.g. ''zen'') does **not** mean that a python virtual environment is loaded.</box> The installation trees can be found in: <code> /opt/sw/skylake/spack-0.19.0 /opt/sw/zen/spack-0.19.0 /opt/sw/cuda-zen/spack-0.19.0 </code> Only the software packages and modules of the currently active tree (denoted by the prefix) will be searched by ''spack find/load'' and ''module avail/load'' commands. It is easily possible to switch between installation trees with the short commands ''cuz'',''zen'', and ''sky'' defined as aliases: <code> alias cuz='spackup cuda-zen' alias sky='spackup skylake' alias zen='spackup zen' </code> Packages depending on GPU, like ''cuda'' are only installed in the ''cuda-zen'' tree. If you want to compile software intended to run on GPU nodes, you need to: - login on a VSC-5 login node (or a VSC-5 compute node). - type ''cuz'' switch to ''cuda-zen''. - compile your code. <code> zen [userl51 ~]$ cuz cuda-zen [userl51 ~]$ ... build your software ... </code> The commands ''cuz''/''zen''/''sky'' ultimately call the shell function ''spackup <arch>'' which will: * set PATH * set MODULEPATH * set PS1 prompt * source /path/to/spack/instance/share/spack/setup-env.sh You can view the shell function ''spackup'' with ''type spackup'', or take a look at the whole script at ''/etc/profile.d/spack.sh''. ---- ===== Migration script - "spack search" ===== If you need to find a list of packages or modules which correspond to the software you have been using until now, you can use the shell function ''spack search'' to do that: <code> zen [user@l51 ~] spack search <hash> </code> where ''<hash>'' is the 7 character hash of the package that you see at the end of the module name or in the output of ''spack find -l <package>''. ==== Usage example ==== Let's assume you have used this ''netcdf-c'' module: <code> netcdf-c/4.8.1-gcc-11.2.0-jsfjwaz </code> which can be found with ''spack find -l netcdf-c %gcc'' as: <code> jsfjwaz netcdf-c@4.8.1 </code> You can use the migration script to find ''netcdf-c'' installations with identical build options: <code> zen [user@l51 ~]$ spack search jsfjwaz ==> The hash 'jsfjwaz7qp52fjxfeg6mbhtt2lj3l573' refers to 'netcdf-c' from 'vsc5' with parameters: netcdf-c ~dap~fsync~hdf4~jna+mpi~parallel-netcdf+pic+shared ==> Searching similar 'netcdf-c' modules in installation 'skylake' ... -- /opt/sw/skylake/spack-0.19.0/share/spack/modules/linux-almalinux8-skylake_avx512 -- netcdf-c/4.9.0-gcc-12.2.0-xck6m4e netcdf-c/4.9.0-gcc-12.2.0-vcjclck netcdf-c/4.9.0-intel-2021.7.1-u6wt7yr netcdf-c/4.9.0-intel-2021.7.1-k2p5vx2 ==> Load any one of these packages with 'module load mypackage' e.g.: module load netcdf-c/4.9.0-intel-2021.7.1-k2p5vx2 ==> Get detailed package info with 'spack find -lvd mypackage' e.g.: spack find -lvd /k2p5vx </code> To search for packages in a different spack tree you can just change to that tree with ''cuz''/''sky''/''zen'', and then run ''spack search'' there. For example to search in the ''cuda-zen'' spack tree: <code> zen [user@l51 search]$ cuz cuda-zen [user@l51 search]$ spacksearch jsfjwaz ==> The package hash jsfjwaz refers to netcdf-c and belongs to vsc5 as: netcdf-c ~dap~fsync~hdf4~jna+mpi~parallel-netcdf+pic+shared ==> Searching similar netcdf-c modules at cuda-zen... ---/gpfs/opt/sw/cuda-zen/spack-0.19.0/share/spack/modules/linux-almalinux8-zen--- netcdf-c/4.9.0-gcc-9.5.0-fx6pjb6 netcdf-c/4.9.0-gcc-9.5.0-4gdf6vm netcdf-c/4.9.0-gcc-9.5.0-o5eb5rf netcdf-c/4.9.0-gcc-9.5.0-upkxxip </code> ---- ==== Deprecated: Continue working with old modules (from environments) ==== <box 80% round red|Warning>This approach is not recommended and we will no longer maintain the spack environments or install new packages there</box> You may continue to use modules from the spack environments skylake and zen3 by adjusting the ''MODULEPATH'' variable: === On zen3 === <code> export MODULEPATH=/opt/sw/vsc4/VSC/Modules/TUWien:/opt/sw/vsc4/VSC/Modules/Intel/oneAPI:/opt/sw/vsc4/VSC/Modules/Parallel-Environment:/opt/sw/vsc4/VSC/Modules/Libraries:/opt/sw/vsc4/VSC/Modules/Compiler:/opt/sw/vsc4/VSC/Modules/Debugging-and-Profiling:/opt/sw/vsc4/VSC/Modules/Applications:/opt/sw/vsc4/VSC/Modules/p71545::/opt/sw/spack-0.17.1/var/spack/environments/zen3/modules/linux-almalinux8-zen:/opt/sw/spack-0.17.1/var/spack/environments/zen3/modules/linux-almalinux8-zen2:/opt/sw/spack-0.17.1/var/spack/environments/zen3/modules/linux-almalinux8-zen3 </code> === On skylake === <code> export MODULEPATH=/opt/sw/vsc4/VSC/Modules/TUWien:/opt/sw/vsc4/VSC/Modules/Intel/oneAPI:/opt/sw/vsc4/VSC/Modules/Parallel-Environment:/opt/sw/vsc4/VSC/Modules/Libraries:/opt/sw/vsc4/VSC/Modules/Compiler:/opt/sw/vsc4/VSC/Modules/Debugging-and-Profiling:/opt/sw/vsc4/VSC/Modules/Applications:/opt/sw/vsc4/VSC/Modules/p71545:/opt/sw/vsc4/VSC/Modules/p71782::/opt/sw/spack-0.19.0/var/spack/environments/skylake/modules/linux-almalinux8-x86_64:/opt/sw/spack-0.19.0/var/spack/environments/skylake/modules/linux-almalinux8-skylake </code> ---- ==== Deprecated: Continue working with old spack environments ==== <box 80% round red|Warning>This approach is not recommended and we will no longer maintain the spack environments or install new packages there</box> You may also continue to work with the spack environments. If you wish to do so you need the following commands: === On zen3 === <code> export SPACK_ROOT=/opt/sw/spack-0.17.1 source /opt/sw/spack-0.17.1/share/spack/setup-env.sh spacktivate zen3 </code> === On skylake === <code> export SPACK_ROOT=/opt/sw/spack-0.19.0 source /opt/sw/spack-0.19.0/share/spack/setup-env.sh spacktivate skylake </code> ---- ===== Loading modules with prerequisites ===== Some module have prerequisites which need to be loaded in addition to the module. You can check if a module has prerequisites with ''module show <module>'', e.g.: <code> zen [user@l51 ~]$ module show py-numpy/1.23.4-gcc-12.2.0-xbac5zw |grep prereq prereq openblas/0.3.21-gcc-12.2.0-gcn6jxp prereq py-setuptools/59.4.0-gcc-12.2.0-qphisr6 prereq python/3.9.15-gcc-12.2.0-my6jxu2 </code> These prerequisites can be loaded together with the module automatically ''module load --auto <module>'', e.g.: <code> zen [user@l51 ~]$ module load --auto py-numpy/1.23.4-gcc-12.2.0-xbac5zw Loading py-numpy/1.23.4-gcc-12.2.0-xbac5zw Loading requirement: openblas/0.3.21-gcc-12.2.0-gcn6jxp python/3.9.15-gcc-12.2.0-my6jxu2 py-setuptools/59.4.0-gcc-12.2.0-qphisr6 </code> ---- ===== Setting LD_LIBRARY_PATH ===== Loading a module does no longer automatically set the ''LD_LIBRARY_PATH'' environment variable, as with some software packages this has lead to conflicts with system libraries. If you have to set ''LD_LIBRARY_PATH'' you may use: <code> export LD_LIBRARY_PATH=$LIBRARY_PATH </code> Alternatively you may have to set it to specific paths where the needed libraries can be found. ---- doku/spack-transition.txt Last modified: 2023/05/17 14:48by msiegel