Python

Python is comparatively fast evolving programming language, so different versions behave very differently. We provide multiple varieties of python installations, please always use spack to find and load them. On the more recent python versions the following packages are included (eg try import os in python):

• os
• sys
• pathlib
• subprocess

However, even the newer python modules do not contain other widespread packages like numpy. We installed many packages, pease always use spack to find and load them. Yet there are so many additional python packages, some with long dependency chains, that we cannot really install all of them for all the different python versions. All installed python packages are named as py-mypackagename, like py-numpy or py-scipy. If your desired package is not installed, there might be a package at the conda forge.

Sometimes pip is not enough for your use case e.g. when packages need to be compiled and not all packages are available in our infrastructure.

In this case you can use conda (see Anaconda) instead of pip to set up a consistent local python environment.

To load conda on our clusters search for the miniconda3 package. At the time of writing the miniconda3 package is available on both VSC-4 & VSC-5 and can be loaded via

spack load miniconda3@4.12.0

If you plan to use conda more frequently you can simple add the spack load statement to your ~/.bashrc file to load it automatically after logging in.

Also make sure that you run the following statements if you are setting up conda for the first time:

conda init bash --dry-run --verbose | grep "# >>> conda initialize" -A 100 | grep "# <<< conda initialize" -B 100 | sed 's/+//g' > ~/.bashrc
source ~/.bashrc

This will add some necessary startup code for conda to your ~/.bashrc file.

After executing these steps you will see that your prompt changed to (base) [myname@l51 ~]$ which signifies that conda is active and the base environment is active.

The default conda channel points to anacondas repository. However this repo does not always contain the latest packages. There is a community driven channel called conda-forge that has many more packages and most of the time the newer versions readily available.

To use conda-forge you need to specify –channel conda-forge when executing conda install commands.

If you always want to use conda-forge you can executed the following statement to make it the default for your user

conda config --add channels conda-forge

In order to create your own user environment you need to do the following steps. To also give a short example for a package which we do not provide via spack we will install phono3py (available onconda forge) into our conda environment (myenv) with conda:

# create conda env 'myenv', set conda-forge channel as default and use the latest python 3.11
conda create --name myenv --channel conda-forge python=3.11
conda activate myenv
conda install --channel conda-forge numpy phono3py

With the above statements conda will create a new environment, activate it and install the requested packages into it. You should see that your prompt now changed to (myenv) [myname@l51 ~]$'

The following commands provide a bit of introspection to make sure that everything is setup as expected:

(myenv) [myname@l51 ~]$ which python
~/.conda/envs/myenv/bin/python
(myenv) [myname@l51 ~]$ python --version
Python 3.11.0
(myenv) [myname@l51 ~]$ which phono3py
~/.conda/envs/myenv/bin/phono3py

Starting python in this conda environment (myenv) and loading the packages also works:

(myenv) myname@l51:~$ python
Python 3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:27:40) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import phono3py
>>> exit()

When using slurm it should be enough to include the line conda activate myenv in your slurm batch script. See slurm for detailed information about slurm in general.

See jupyterhub for an even cooler way to access VSC, especially when you are using mainly python.