Differences

This shows you the differences between two versions of the page.

--- doku:python [2023/04/17 08:14] – [Optional: execute conda hooks on login] katrin
+++ doku:python [2024/03/19 14:24] (current) – katrin
@@ Line 1: / Line 1: @@
-====== Python ======
+====== Python in VSC ======
+Previously we recommended using spack python packages, however we decided to deprecate this approach since it is much easier and user friendly to use '''conda''' which is already widely known.
+Have a look at the new **quickstart guide** below or the material from the python4hpc training: [[https://gitlab.tuwien.ac.at/vsc-public/training/python4hpc|python4hpc public repo]].
+The old approach is now //deprecated// but kept on this page for reference.
+===== Quickstart: Using Python with Conda =====
+To install a new environment, start with creating an **environment yaml file** and store it in a safe location (ideally in your source code repository).
+This step is optional but we really **recommend** it so you and everyone in your team is able to produce the same python environment.
+To find valid conda package names look at [[https://anaconda.org/search|Anaconda repo package search]].
+Example (add your own packages below dependencies):
+<code>
+name: my-env
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - python=3.10
+  - tensorflow=2.15.0
+</code>
+You can the install your environment by using the following commands:
+<code>
+# load the miniconda package
+$ module load miniconda3
+# executes bash hooks for the tool to function
+# and enters the "base" environment
+$ eval "$(conda shell.bash hook)"
+# create your environment via environment file
+# this will place the environment in your ~/.conda/envs folder
+# note:
+#  - make sure to use "conda >env< create" - "conda create" is a different command
+#  - the name is taken from the yaml file; a custom name can be specified with "-n my-custom-name"
+(base) $ conda env create --file my_env.yaml
+# after creation you can activate the environment to run code in it
+(base) $ conda activate my-env
+# test python version to make sure we have the right one
+(my-env) $ python --version
+Python 3.10.11
+</code>
+You can now start developing and even use the environment in a slurm script
+<code>
+#!/bin/bash
+#SBATCH --job-name=slurm_conda_example
+#SBATCH --time=00-00:05:00
+#SBATCH --ntasks=2
+#SBATCH --mem=2GB
+# modify SBATCH options according to needs
+# see "Setup Conda" above or consult "module avail miniconda3" to get the right package name
+module load miniconda3
+eval "$(conda shell.bash hook)"
+conda activate myenv
+# print out some info of the python executable in use
+# this should point to the python version from "myenv"
+which python
+python --version
+</code>
+===== More info about Conda =====
+For more information about conda check-out the conda notebook of the python4HPC training material: [[https://gitlab.tuwien.ac.at/vsc-public/training/python4hpc/-/blob/main/D1_02_env_03_conda.ipynb|python4HPC Conda Environments]]
+===== FAQ =====
+  * **'''eval "$(conda shell.bash hook)"''' fails with '''CommandNotFoundError ...'''**
+Make sure to use the currently provided '''miniconda3''' module via '''module load miniconda3'''
+  * **I dont get the right python version and packages in my slurm batch file environment** - **What is wrong>?**
+Make sure that one of the first things in your sbatch script is loading the miniconda3 package '''module load miniconda3''' and that you execute '''eval "$(conda shell.bash hook)"''' before using any other conda commands.
+  * **I don't get GPU/CUDA enabled packages when installing a conda environment** / **I get conda installation errors when i select GPU/CUDA enabled builds for my conda environment***
+In order to install packages from conda-forge that require CUDA on a machine that does not have GPUs you have to set an environment variable before creating the conda environment:
+<code>
+name: my-pytorch-gpu-env
+channels:
+  - pytorch
+  - conda-forge
+  - defaults
+dependencies:
+  - python=3.12
+  - pytorch=2.2.*=*cuda11.8*
+</code>
+<code>
+# for example if you use cuda 11.8
+CONDA_OVERRIDE_CUDA="11.8" conda env create -n my-pytorch-gpu-env --file my-pytorch-gpu-env.yaml
+</code>
+====== Deprecated Information (work in progress) ======
 ===== Python Installations =====
@@ Line 36: / Line 145: @@
 ==== Setup Conda ====
-To load ''conda'' on our clusters search for the ''miniconda3'' package. At the time of writing the ''miniconda3'' package is available on both VSC-4 & VSC-5 and can be loaded via
+To use the ''conda'' tool on our clusters search for the ''miniconda3'' package. At the time of writing the ''miniconda3'' package is available in all environments on both VSC-4 & VSC-5 and can be loaded with
 <code bash>
-# VSC-4
+module load miniconda3
-module load miniconda3/4.12.0-gcc-12.2.0-pad7sa7
-# VSC-5
-module load miniconda3/4.12.0-gcc-11.2.0-ap65vga
 </code>
@@ Line 52: / Line 157: @@
 <code bash>
-conda init bash --dry-run --verbose | grep "# >>> conda initialize" -A 100 | grep "# <<< conda initialize" -B 100 | sed 's/+//g' > ~/.bashrc
+conda init bash --dry-run --verbose | grep "# >>> conda initialize" -A 100 | grep "# <<< conda initialize" -B 100 | sed 's/+//g' >> ~/.bashrc
 source ~/.bashrc
 </code>
@@ Line 140: / Line 245: @@
 >>> import phono3py
 >>> exit()
+</code>
+=== Pytorch ===
+To install the latest pytorch with conda please follow the instructions on this page:
+[[https://pytorch.org/get-started/locally/|Latest Pytorch installation]]
+For older and specific combinations of pytorch and cuda please have a look at this page:
+[[https://pytorch.org/get-started/previous-versions/#v182-with-lts-support|Previous Pytorch versions]]
+Eg.: to install v1.13.1 with cuda 11.6 use:
+<code>
+conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia
+</code>
+As of 2023-08-25: you can also install the current pytorch version with an older cuda version via:
+<code>
+conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c nvidia
 </code>