Differences

This shows you the differences between two versions of the page.

--- doku:python [2023/03/30 13:16] – [Create conda env from commandline] katrin
+++ doku:python [2024/03/19 14:24] (current) – katrin
@@ Line 1: / Line 1: @@
-====== Python ======
+====== Python in VSC ======
+Previously we recommended using spack python packages, however we decided to deprecate this approach since it is much easier and user friendly to use '''conda''' which is already widely known.
+Have a look at the new **quickstart guide** below or the material from the python4hpc training: [[https://gitlab.tuwien.ac.at/vsc-public/training/python4hpc|python4hpc public repo]].
+The old approach is now //deprecated// but kept on this page for reference.
+===== Quickstart: Using Python with Conda =====
+To install a new environment, start with creating an **environment yaml file** and store it in a safe location (ideally in your source code repository).
+This step is optional but we really **recommend** it so you and everyone in your team is able to produce the same python environment.
+To find valid conda package names look at [[https://anaconda.org/search|Anaconda repo package search]].
+Example (add your own packages below dependencies):
+<code>
+name: my-env
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - python=3.10
+  - tensorflow=2.15.0
+</code>
+You can the install your environment by using the following commands:
+<code>
+# load the miniconda package
+$ module load miniconda3
+# executes bash hooks for the tool to function
+# and enters the "base" environment
+$ eval "$(conda shell.bash hook)"
+# create your environment via environment file
+# this will place the environment in your ~/.conda/envs folder
+# note:
+#  - make sure to use "conda >env< create" - "conda create" is a different command
+#  - the name is taken from the yaml file; a custom name can be specified with "-n my-custom-name"
+(base) $ conda env create --file my_env.yaml
+# after creation you can activate the environment to run code in it
+(base) $ conda activate my-env
+# test python version to make sure we have the right one
+(my-env) $ python --version
+Python 3.10.11
+</code>
+You can now start developing and even use the environment in a slurm script
+<code>
+#!/bin/bash
+#SBATCH --job-name=slurm_conda_example
+#SBATCH --time=00-00:05:00
+#SBATCH --ntasks=2
+#SBATCH --mem=2GB
+# modify SBATCH options according to needs
+# see "Setup Conda" above or consult "module avail miniconda3" to get the right package name
+module load miniconda3
+eval "$(conda shell.bash hook)"
+conda activate myenv
+# print out some info of the python executable in use
+# this should point to the python version from "myenv"
+which python
+python --version
+</code>
+===== More info about Conda =====
+For more information about conda check-out the conda notebook of the python4HPC training material: [[https://gitlab.tuwien.ac.at/vsc-public/training/python4hpc/-/blob/main/D1_02_env_03_conda.ipynb|python4HPC Conda Environments]]
+===== FAQ =====
+  * **'''eval "$(conda shell.bash hook)"''' fails with '''CommandNotFoundError ...'''**
+Make sure to use the currently provided '''miniconda3''' module via '''module load miniconda3'''
+  * **I dont get the right python version and packages in my slurm batch file environment** - **What is wrong>?**
+Make sure that one of the first things in your sbatch script is loading the miniconda3 package '''module load miniconda3''' and that you execute '''eval "$(conda shell.bash hook)"''' before using any other conda commands.
+  * **I don't get GPU/CUDA enabled packages when installing a conda environment** / **I get conda installation errors when i select GPU/CUDA enabled builds for my conda environment***
+In order to install packages from conda-forge that require CUDA on a machine that does not have GPUs you have to set an environment variable before creating the conda environment:
+<code>
+name: my-pytorch-gpu-env
+channels:
+  - pytorch
+  - conda-forge
+  - defaults
+dependencies:
+  - python=3.12
+  - pytorch=2.2.*=*cuda11.8*
+</code>
+<code>
+# for example if you use cuda 11.8
+CONDA_OVERRIDE_CUDA="11.8" conda env create -n my-pytorch-gpu-env --file my-pytorch-gpu-env.yaml
+</code>
+====== Deprecated Information (work in progress) ======
 ===== Python Installations =====
@@ Line 36: / Line 145: @@
 ==== Setup Conda ====
-To load ''conda'' on our clusters search for the ''miniconda3'' package. At the time of writing the ''miniconda3'' package is available on both VSC-4 & VSC-5 and can be loaded via
+To use the ''conda'' tool on our clusters search for the ''miniconda3'' package. At the time of writing the ''miniconda3'' package is available in all environments on both VSC-4 & VSC-5 and can be loaded with
 <code bash>
-# VSC-4
+module load miniconda3
-module load miniconda3/4.12.0-gcc-12.2.0-pad7sa7
-# VSC-5
-module load miniconda3/4.12.0-gcc-11.2.0-ap65vga
 </code>
 If you plan to use conda more frequently you can simple add the load statement to your ''~/.bashrc'' file to have it loaded automatically after logging in.
-=== Optional: execute conda hooks on login ===
+=== Optional: execute conda runtime hooks on login ===
-If you want to have every conda function available directly after logging in you can execute the following statements to add the conda startup code to your ''~/.bashrc'' file as well.
+To fully utilize the conda command it needs to load runtime hooks. If you want to have conda completely initialized directly after logging in you can execute the following statements to add the conda startup code to your ''~/.bashrc'' file as well.
 <code bash>
-conda init bash --dry-run --verbose | grep "# >>> conda initialize" -A 100 | grep "# <<< conda initialize" -B 100 | sed 's/+//g' > ~/.bashrc
+conda init bash --dry-run --verbose | grep "# >>> conda initialize" -A 100 | grep "# <<< conda initialize" -B 100 | sed 's/+//g' >> ~/.bashrc
 source ~/.bashrc
 </code>
@@ Line 62: / Line 167: @@
 ==== Channels ====
-The ''default'' conda channel points to anacondas repository. However this repo does not always contain the latest packages. There is a community driven channel called ''conda-forge'' that has many more packages and most of the time the newer versions readily available.
+The ''default'' conda channel points to anacondas repository. However this repo does not always contain the latest packages. There are a number of community driven channels (e.g. ''conda-forge'') that have many more packages and most of the time the newer versions readily available.
-To use ''conda-forge'' you need to specify ''--channel conda-forge'' when executing conda install commands.
+Popular channels
+  - ''conda-forge'' - [[https://conda-forge.org/feedstock-outputs/]]
+  - ''bioconda'' - [[https://bioconda.github.io/conda-package_index.html]]
-If you always want to use ''conda-forge'' you can executed the following statement to make it the default for your user
+To use e.g. ''conda-forge'' you need to specify ''--channel conda-forge'' when executing conda install commands.
+If you want to set e.g. ''conda-forge'' as default for your user you can achieve this by executing the following statement:
 <code bash>
 conda config --add channels conda-forge
@@ Line 136: / Line 245: @@
 >>> import phono3py
 >>> exit()
+</code>
+=== Pytorch ===
+To install the latest pytorch with conda please follow the instructions on this page:
+[[https://pytorch.org/get-started/locally/|Latest Pytorch installation]]
+For older and specific combinations of pytorch and cuda please have a look at this page:
+[[https://pytorch.org/get-started/previous-versions/#v182-with-lts-support|Previous Pytorch versions]]
+Eg.: to install v1.13.1 with cuda 11.6 use:
+<code>
+conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia
+</code>
+As of 2023-08-25: you can also install the current pytorch version with an older cuda version via:
+<code>
+conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c nvidia
 </code>