Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
Next revisionBoth sides next revision
doku:python [2023/02/23 11:29] – [SLURM] katrindoku:python [2023/03/30 13:16] – [Create conda env from commandline] katrin
Line 38: Line 38:
 To load ''conda'' on our clusters search for the ''miniconda3'' package. At the time of writing the ''miniconda3'' package is available on both VSC-4 & VSC-5 and can be loaded via  To load ''conda'' on our clusters search for the ''miniconda3'' package. At the time of writing the ''miniconda3'' package is available on both VSC-4 & VSC-5 and can be loaded via 
 <code bash> <code bash>
-spack load miniconda3@4.12.0+# VSC-4 
 +module load miniconda3/4.12.0-gcc-12.2.0-pad7sa7 
 + 
 +# VSC-5 
 +module load miniconda3/4.12.0-gcc-11.2.0-ap65vga
 </code> </code>
  
-If you plan to use conda more frequently you can simple add the spack load statement to your ''~/.bashrc'' file to load it automatically after logging in.+If you plan to use conda more frequently you can simple add the load statement to your ''~/.bashrc'' file to have it loaded automatically after logging in. 
 + 
 +=== Optional: execute conda hooks on login === 
 + 
 +If you want to have every conda function available directly after logging in you can execute the following statements to add the conda startup code to your ''~/.bashrc'' file as well.
  
-Also make sure that you run the following statements if you are setting up conda for the **first time**: 
 <code bash> <code bash>
 conda init bash --dry-run --verbose | grep "# >>> conda initialize" -A 100 | grep "# <<< conda initialize" -B 100 | sed 's/+//g' > ~/.bashrc conda init bash --dry-run --verbose | grep "# >>> conda initialize" -A 100 | grep "# <<< conda initialize" -B 100 | sed 's/+//g' > ~/.bashrc
 source ~/.bashrc source ~/.bashrc
 </code> </code>
- 
-This will add some necessary startup code for conda to your ''~/.bashrc'' file. 
  
 After executing these steps you will see that your prompt changed to ''(base) [myname@l51 ~]$'' which signifies that conda is active and the ''base'' environment is active. After executing these steps you will see that your prompt changed to ''(base) [myname@l51 ~]$'' which signifies that conda is active and the ''base'' environment is active.
 +
 +If you already have an environment you can also add ''conda activate myenv'' to your ''~/.bashrc'' file
  
 ==== Channels ==== ==== Channels ====
Line 65: Line 72:
  
 ==== Create your own custom conda environment ==== ==== Create your own custom conda environment ====
 +
 +=== Create conda env using environment files ===
 +
 +This is the recommended method to create new conda environments since it makes environment creation reproducible. The file can be easily shared or e.g. added to your version control system
 +
 +First decide which python version and packages you need. In case you don't know the exact versions upfront you can also just create a first draft of the env-file without pinned version to get the latest libraries. With that information in mind we now write our environment file called "myenv.yml"
 +
 +<code>
 +name: myenv
 +channels:
 +  - conda-forge
 +dependencies:
 +  - python=3.10
 +  - pytorch=1.13.1
 +</code>
 +
 +After this we run the conda solver to create and install the new environment
 +
 +<code>
 +conda env create -f myenv.yml
 +</code>
 +
 +Conda will now take its time and solve the environment and then download and install the packages. After this has been done the environment can be activated with
 +
 +<code>
 +conda activate myenv
 +</code>
 +
 +=== Create conda env from commandline ===
 +
 +**Note:** this method is only for illustrating how conda works and is not recommended since it does not create a reproducible environment specification
  
 In order to create your own user environment you need to do the following steps. To also give a short example for a package which we do not provide via spack we will install ''phono3py'' (available on[[https://anaconda.org/conda-forge|conda forge]]) into our conda environment ''(myenv)'' with conda: In order to create your own user environment you need to do the following steps. To also give a short example for a package which we do not provide via spack we will install ''phono3py'' (available on[[https://anaconda.org/conda-forge|conda forge]]) into our conda environment ''(myenv)'' with conda:
Line 102: Line 140:
 ===== SLURM ===== ===== SLURM =====
  
-See the following minimal example to use conda with slurm in your batch script (the example assumes that you have the conda init code in your ''~/.bashrc'' file - see below for an alternative). See [[doku:slurm]] for detailed information about slurm in general. +See the following minimal example to use conda with slurm in your batch script. See [[doku:slurm]] for detailed information about slurm in general.
- +
-<code bash> +
-#!/bin/bash --login +
- +
-#SBATCH --job-name=slurm_conda_example +
-#SBATCH --time 00-00:05:00 +
-#SBATCH --ntasks=2 +
-#SBATCH --mem=2GB +
- +
-spack load miniconda3@4.12.0 +
-conda activate myenv +
-which python +
-python --version +
-</code> +
- +
-Please note the ''--login'' flag to the shell above. This way the shell in the job will be a login shell and thus load your ''~/.bashrc'' file before executing the script code. If cannot or don't want do this you can also just extract the conda init code from your bashrc file and put it into a separate file (e.g. ''conda-init.sh'') that is placed in your home directory and source this file manually in the batch script. +
- +
-See the following example:+
  
 <code bash> <code bash>
Line 130: Line 150:
 #SBATCH --mem=2GB #SBATCH --mem=2GB
  
-spack load miniconda3@4.12.0 +# modify SBATCH options according to needs 
-source ~/conda-init.sh+ 
 +# see "Setup Conda" above or consult "module avail miniconda3" to get the right package name 
 +module load MINICONDA3_PACKAGE_NAME 
 +eval "$(conda shell.bash hook)"
 conda activate myenv conda activate myenv
 +
 +# print out some info of the python executable in use
 +# this should point to the python version from "myenv"
 which python which python
 python --version python --version
  • doku/python.txt
  • Last modified: 2024/03/19 14:24
  • by katrin