Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
Next revisionBoth sides next revision
doku:python [2023/03/30 13:05] – [Create your own custom conda environment] katrindoku:python [2023/04/05 12:24] – [Channels] katrin
Line 38: Line 38:
 To load ''conda'' on our clusters search for the ''miniconda3'' package. At the time of writing the ''miniconda3'' package is available on both VSC-4 & VSC-5 and can be loaded via  To load ''conda'' on our clusters search for the ''miniconda3'' package. At the time of writing the ''miniconda3'' package is available on both VSC-4 & VSC-5 and can be loaded via 
 <code bash> <code bash>
-spack load miniconda3@4.12.0+# VSC-4 
 +module load miniconda3/4.12.0-gcc-12.2.0-pad7sa7 
 + 
 +# VSC-5 
 +module load miniconda3/4.12.0-gcc-11.2.0-ap65vga
 </code> </code>
  
-If you plan to use conda more frequently you can simple add the spack load statement to your ''~/.bashrc'' file to load it automatically after logging in.+If you plan to use conda more frequently you can simple add the load statement to your ''~/.bashrc'' file to have it loaded automatically after logging in. 
 + 
 +=== Optional: execute conda hooks on login === 
 + 
 +If you want to have every conda function available directly after logging in you can execute the following statements to add the conda startup code to your ''~/.bashrc'' file as well.
  
-Also make sure that you run the following statements if you are setting up conda for the **first time**: 
 <code bash> <code bash>
 conda init bash --dry-run --verbose | grep "# >>> conda initialize" -A 100 | grep "# <<< conda initialize" -B 100 | sed 's/+//g' > ~/.bashrc conda init bash --dry-run --verbose | grep "# >>> conda initialize" -A 100 | grep "# <<< conda initialize" -B 100 | sed 's/+//g' > ~/.bashrc
 source ~/.bashrc source ~/.bashrc
 </code> </code>
- 
-This will add some necessary startup code for conda to your ''~/.bashrc'' file. 
  
 After executing these steps you will see that your prompt changed to ''(base) [myname@l51 ~]$'' which signifies that conda is active and the ''base'' environment is active. After executing these steps you will see that your prompt changed to ''(base) [myname@l51 ~]$'' which signifies that conda is active and the ''base'' environment is active.
 +
 +If you already have an environment you can also add ''conda activate myenv'' to your ''~/.bashrc'' file
  
 ==== Channels ==== ==== Channels ====
  
-The ''default'' conda channel points to anacondas repository. However this repo does not always contain the latest packages. There is a community driven channel called ''conda-forge'' that has many more packages and most of the time the newer versions readily available.+The ''default'' conda channel points to anacondas repository. However this repo does not always contain the latest packages. There are a number of community driven channels (e.g. ''conda-forge'') that have many more packages and most of the time the newer versions readily available. 
 + 
 +Popular channels 
 +  - ''conda-forge'' - [[https://conda-forge.org/feedstock-outputs/]] 
 +  - ''bioconda'' - [[https://bioconda.github.io/conda-package_index.html]]
  
-To use ''conda-forge'' you need to specify ''--channel conda-forge'' when executing conda install commands.+To use e.g. ''conda-forge'' you need to specify ''--channel conda-forge'' when executing conda install commands.
  
-If you always want to use ''conda-forge'' you can executed the following statement to make it the default for your user+If you want to set e.g. ''conda-forge'' as default for your user you can achieve this by executing the following statement:
 <code bash> <code bash>
 conda config --add channels conda-forge conda config --add channels conda-forge
Line 95: Line 106:
 === Create conda env from commandline === === Create conda env from commandline ===
  
-Note: this method is not recommended since its not reproducible+**Note:** this method is only for illustrating how conda works and is not recommended since it does not create a reproducible environment specification
  
 In order to create your own user environment you need to do the following steps. To also give a short example for a package which we do not provide via spack we will install ''phono3py'' (available on[[https://anaconda.org/conda-forge|conda forge]]) into our conda environment ''(myenv)'' with conda: In order to create your own user environment you need to do the following steps. To also give a short example for a package which we do not provide via spack we will install ''phono3py'' (available on[[https://anaconda.org/conda-forge|conda forge]]) into our conda environment ''(myenv)'' with conda:
Line 133: Line 144:
 ===== SLURM ===== ===== SLURM =====
  
-See the following minimal example to use conda with slurm in your batch script (the example assumes that you have the conda init code in your ''~/.bashrc'' file - see below for an alternative). See [[doku:slurm]] for detailed information about slurm in general.+See the following minimal example to use conda with slurm in your batch script. See [[doku:slurm]] for detailed information about slurm in general.
  
 <code bash> <code bash>
-#!/bin/bash --login+#!/bin/bash
  
 #SBATCH --job-name=slurm_conda_example #SBATCH --job-name=slurm_conda_example
Line 143: Line 154:
 #SBATCH --mem=2GB #SBATCH --mem=2GB
  
-spack load miniconda3@4.12.0+# modify SBATCH options according to needs 
 + 
 +# see "Setup Conda" above or consult "module avail miniconda3" to get the right package name 
 +module load MINICONDA3_PACKAGE_NAME 
 +eval "$(conda shell.bash hook)"
 conda activate myenv conda activate myenv
-which python 
-python --version 
-</code> 
  
-Please note the ''--login'' flag to the shell above. This way the shell in the job will be a login shell and thus load your ''~/.bashrc'' file before executing the script code. If cannot or don't want do this you can also just extract the conda init code from your bashrc file and put it into a separate file (e.g. ''conda-init.sh'') that is placed in your home directory and source this file manually in the batch script. +# print out some info of the python executable in use 
- +# this should point to the python version from "myenv"
-See the following example: +
- +
-<code bash> +
-#!/bin/bash +
- +
-#SBATCH --job-name=slurm_conda_example +
-#SBATCH --time 00-00:05:00 +
-#SBATCH --ntasks=2 +
-#SBATCH --mem=2GB +
- +
-spack load miniconda3@4.12.0 +
-source ~/conda-init.sh +
-conda activate myenv+
 which python which python
 python --version python --version
  • doku/python.txt
  • Last modified: 2024/03/19 14:24
  • by katrin