Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
doku:python [2023/06/19 08:41] – katrin | doku:python [2024/09/18 11:14] (current) – katrin | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Python | + | ====== Python |
- | Previously we recommended using spack python packages, however we decided to deprecate this approach since it is much easier and user friendly to use ''' | + | Previously we recommended using spack python packages, however we decided to **deprecate** this approach since it is much easier and user friendly to use ''' |
- | The documentation is currently in the process of being revised. For now please either use the old information that is still available OR use the new quickstart guide below. | + | Have a look at the new **quickstart guide** below or the material from the python4hpc training: [[https:// |
- | ===== (Conda) Quickstart ===== | + | The old approach is now // |
- | To install a new environment start with creating an environment yaml file and store it in a safe location (ideally in your source code repository). This step is optional but we really recommend it so you and everyone in your team is able to produce the same python environment. | + | ===== Quickstart: Using Python with Conda ===== |
+ | |||
+ | To install a new environment, start with creating an **environment yaml file** and store it in a safe location (ideally in your source code repository). | ||
+ | |||
+ | This step is optional but we really | ||
+ | |||
+ | To find valid conda package names look at [[https:// | ||
+ | |||
+ | Example (add your own packages below dependencies): | ||
< | < | ||
name: my-env | name: my-env | ||
+ | channels: | ||
+ | - conda-forge | ||
dependencies: | dependencies: | ||
- python=3.10 | - python=3.10 | ||
- | - tensorflow-gpu=2.6.0 | + | - tensorflow=2.15.0 |
</ | </ | ||
- | Afterwards you can then use conda like this: | + | You can the install your environment by using the following commands: |
< | < | ||
Line 27: | Line 37: | ||
# create your environment via environment file | # create your environment via environment file | ||
# this will place the environment in your ~/ | # this will place the environment in your ~/ | ||
- | (base) $ conda create -n my-env --file my_env.yaml | + | # note: |
+ | # - make sure to use "conda > | ||
+ | # - the name is taken from the yaml file; a custom name can be specified with "-n my-custom-name" | ||
+ | (base) $ conda env create | ||
# after creation you can activate the environment to run code in it | # after creation you can activate the environment to run code in it | ||
- | (base) $ conda activate | + | (base) $ conda activate |
# test python version to make sure we have the right one | # test python version to make sure we have the right one | ||
- | (my-env) $ python --version | + | (my_env) $ python --version |
Python 3.10.11 | Python 3.10.11 | ||
</ | </ | ||
- | You can now start developing and even just use the environment in a slurm script | + | You can now start developing and even use the environment in a slurm script |
< | < | ||
Line 43: | Line 56: | ||
#SBATCH --job-name=slurm_conda_example | #SBATCH --job-name=slurm_conda_example | ||
- | #SBATCH --time 00-00:05:00 | + | #SBATCH --time=00-00:05:00 |
#SBATCH --ntasks=2 | #SBATCH --ntasks=2 | ||
#SBATCH --mem=2GB | #SBATCH --mem=2GB | ||
Line 50: | Line 63: | ||
# see "Setup Conda" above or consult " | # see "Setup Conda" above or consult " | ||
- | module load MINICONDA3_PACKAGE_NAME | + | module load miniconda3 |
eval " | eval " | ||
conda activate myenv | conda activate myenv | ||
Line 60: | Line 73: | ||
</ | </ | ||
- | ==== FAQ ==== | + | ===== More info about Conda ===== |
- | **I see the error ''' | + | For more information about conda check-out the conda notebook of the python4HPC training material: [[https:// |
+ | |||
+ | ===== FAQ ===== | ||
+ | |||
+ | * **''' | ||
+ | |||
+ | Make sure to use the currently provided ''' | ||
+ | |||
+ | * **I dont get the right python version and packages | ||
+ | |||
+ | Make sure that one of the first things in your sbatch script is loading the miniconda3 package ''' | ||
+ | |||
+ | * **I don't get GPU/CUDA enabled packages when installing a conda environment** / **I get conda installation errors when i select GPU/CUDA enabled builds for my conda environment*** | ||
+ | |||
+ | In order to install packages from conda-forge that require CUDA on a machine that does not have GPUs you have to set an environment variable before creating the conda environment: | ||
+ | < | ||
+ | name: my-pytorch-gpu-env | ||
+ | channels: | ||
+ | - pytorch | ||
+ | - conda-forge | ||
+ | dependencies: | ||
+ | - python=3.12 | ||
+ | - pytorch=2.2.*=*cuda11.8* | ||
+ | </ | ||
+ | |||
+ | < | ||
+ | # for example if you use cuda 11.8 | ||
+ | CONDA_OVERRIDE_CUDA=" | ||
+ | </ | ||
- | The default for sbatch is to use the current user environment and start the job in the context of this environment. Since conda is depending on environment variables being setup correctly you need to ake sure to start the job script from a clean environment. Alternatively you can also use sbatch with the ''' | ||
====== Deprecated Information (work in progress) ====== | ====== Deprecated Information (work in progress) ====== | ||
Line 115: | Line 155: | ||
<code bash> | <code bash> | ||
- | conda init bash --dry-run --verbose | grep "# >>> | + | conda init bash --dry-run --verbose | grep "# >>> |
source ~/.bashrc | source ~/.bashrc | ||
</ | </ | ||
Line 125: | Line 165: | ||
==== Channels ==== | ==== Channels ==== | ||
- | The '' | + | The '' |
Popular channels | Popular channels | ||
Line 135: | Line 175: | ||
If you want to set e.g. '' | If you want to set e.g. '' | ||
<code bash> | <code bash> | ||
+ | conda config –remove channels defaults | ||
conda config --add channels conda-forge | conda config --add channels conda-forge | ||
</ | </ | ||
Line 203: | Line 244: | ||
>>> | >>> | ||
>>> | >>> | ||
+ | </ | ||
+ | |||
+ | === Pytorch === | ||
+ | |||
+ | To install the latest pytorch with conda please follow the instructions on this page: | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | For older and specific combinations of pytorch and cuda please have a look at this page: | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | Eg.: to install v1.13.1 with cuda 11.6 use: | ||
+ | < | ||
+ | conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia | ||
+ | </ | ||
+ | |||
+ | |||
+ | As of 2023-08-25: you can also install the current pytorch version with an older cuda version via: | ||
+ | < | ||
+ | conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c nvidia | ||
</ | </ | ||