Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision | ||
doku:python [2023/03/30 13:14] – [SLURM] katrin | doku:python [2024/03/19 14:24] – katrin | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Python ====== | + | ====== Python |
+ | |||
+ | Previously we recommended using spack python packages, however we decided to deprecate this approach since it is much easier and user friendly to use ''' | ||
+ | |||
+ | Have a look at the new **quickstart guide** below or the material from the python4hpc training: [[https:// | ||
+ | |||
+ | The old approach is now // | ||
+ | |||
+ | ===== Quickstart: Using Python with Conda ===== | ||
+ | |||
+ | To install a new environment, | ||
+ | |||
+ | This step is optional but we really **recommend** it so you and everyone in your team is able to produce the same python environment. | ||
+ | |||
+ | To find valid conda package names look at [[https:// | ||
+ | |||
+ | Example (add your own packages below dependencies): | ||
+ | < | ||
+ | name: my-env | ||
+ | channels: | ||
+ | - conda-forge | ||
+ | - defaults | ||
+ | dependencies: | ||
+ | - python=3.10 | ||
+ | - tensorflow=2.15.0 | ||
+ | </ | ||
+ | |||
+ | You can the install your environment by using the following commands: | ||
+ | |||
+ | < | ||
+ | # load the miniconda package | ||
+ | $ module load miniconda3 | ||
+ | |||
+ | # executes bash hooks for the tool to function | ||
+ | # and enters the " | ||
+ | $ eval " | ||
+ | |||
+ | # create your environment via environment file | ||
+ | # this will place the environment in your ~/ | ||
+ | # note: | ||
+ | # - make sure to use "conda >env< create" | ||
+ | # - the name is taken from the yaml file; a custom name can be specified with "-n my-custom-name" | ||
+ | (base) $ conda env create --file my_env.yaml | ||
+ | |||
+ | # after creation you can activate the environment to run code in it | ||
+ | (base) $ conda activate my-env | ||
+ | |||
+ | # test python version to make sure we have the right one | ||
+ | (my-env) $ python --version | ||
+ | Python 3.10.11 | ||
+ | </ | ||
+ | |||
+ | You can now start developing and even use the environment in a slurm script | ||
+ | |||
+ | < | ||
+ | # | ||
+ | |||
+ | #SBATCH --job-name=slurm_conda_example | ||
+ | #SBATCH --time=00-00: | ||
+ | #SBATCH --ntasks=2 | ||
+ | #SBATCH --mem=2GB | ||
+ | |||
+ | # modify SBATCH options according to needs | ||
+ | |||
+ | # see "Setup Conda" above or consult " | ||
+ | module load miniconda3 | ||
+ | eval " | ||
+ | conda activate myenv | ||
+ | |||
+ | # print out some info of the python executable in use | ||
+ | # this should point to the python version from " | ||
+ | which python | ||
+ | python --version | ||
+ | </ | ||
+ | |||
+ | ===== More info about Conda ===== | ||
+ | |||
+ | For more information about conda check-out the conda notebook of the python4HPC training material: [[https:// | ||
+ | |||
+ | ===== FAQ ===== | ||
+ | |||
+ | * **''' | ||
+ | |||
+ | Make sure to use the currently provided ''' | ||
+ | |||
+ | * **I dont get the right python version and packages in my slurm batch file environment** - **What is wrong>? | ||
+ | |||
+ | Make sure that one of the first things in your sbatch script is loading the miniconda3 package ''' | ||
+ | |||
+ | * **I don't get GPU/CUDA enabled packages when installing a conda environment** / **I get conda installation errors when i select GPU/CUDA enabled builds for my conda environment*** | ||
+ | |||
+ | In order to install packages from conda-forge that require CUDA on a machine that does not have GPUs you have to set an environment variable before creating the conda environment: | ||
+ | < | ||
+ | name: my-pytorch-gpu-env | ||
+ | channels: | ||
+ | - pytorch | ||
+ | - conda-forge | ||
+ | - defaults | ||
+ | dependencies: | ||
+ | - python=3.12 | ||
+ | - pytorch=2.2.*=*cuda11.8* | ||
+ | </ | ||
+ | |||
+ | < | ||
+ | # for example if you use cuda 11.8 | ||
+ | CONDA_OVERRIDE_CUDA=" | ||
+ | </ | ||
+ | |||
+ | |||
+ | ====== Deprecated Information (work in progress) | ||
===== Python Installations ===== | ===== Python Installations ===== | ||
Line 36: | Line 145: | ||
==== Setup Conda ==== | ==== Setup Conda ==== | ||
- | To load '' | + | To use the '' |
<code bash> | <code bash> | ||
- | # VSC-4 | + | module load miniconda3 |
- | module load miniconda3/ | + | |
- | + | ||
- | # VSC-5 | + | |
- | module load miniconda3/ | + | |
</ | </ | ||
If you plan to use conda more frequently you can simple add the load statement to your '' | If you plan to use conda more frequently you can simple add the load statement to your '' | ||
- | === Optional: execute conda hooks on login === | + | === Optional: execute conda runtime |
- | If you want to have every conda function available | + | To fully utilize the conda command it needs to load runtime hooks. |
<code bash> | <code bash> | ||
- | conda init bash --dry-run --verbose | grep "# >>> | + | conda init bash --dry-run --verbose | grep "# >>> |
source ~/.bashrc | source ~/.bashrc | ||
</ | </ | ||
Line 62: | Line 167: | ||
==== Channels ==== | ==== Channels ==== | ||
- | The '' | + | The '' |
- | To use '' | + | Popular channels |
+ | - '' | ||
+ | - '' | ||
- | If you always | + | To use e.g. '' |
+ | |||
+ | If you want to set e.g. '' | ||
<code bash> | <code bash> | ||
conda config --add channels conda-forge | conda config --add channels conda-forge | ||
Line 102: | Line 211: | ||
=== Create conda env from commandline === | === Create conda env from commandline === | ||
- | Note: this method is not recommended since its not reproducible | + | **Note:** this method |
In order to create your own user environment you need to do the following steps. To also give a short example for a package which we do not provide via spack we will install '' | In order to create your own user environment you need to do the following steps. To also give a short example for a package which we do not provide via spack we will install '' | ||
Line 136: | Line 245: | ||
>>> | >>> | ||
>>> | >>> | ||
+ | </ | ||
+ | |||
+ | === Pytorch === | ||
+ | |||
+ | To install the latest pytorch with conda please follow the instructions on this page: | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | For older and specific combinations of pytorch and cuda please have a look at this page: | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | Eg.: to install v1.13.1 with cuda 11.6 use: | ||
+ | < | ||
+ | conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia | ||
+ | </ | ||
+ | |||
+ | |||
+ | As of 2023-08-25: you can also install the current pytorch version with an older cuda version via: | ||
+ | < | ||
+ | conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c nvidia | ||
</ | </ | ||
Line 152: | Line 282: | ||
# modify SBATCH options according to needs | # modify SBATCH options according to needs | ||
- | # see above for the right package name | + | # see "Setup Conda" |
module load MINICONDA3_PACKAGE_NAME | module load MINICONDA3_PACKAGE_NAME | ||
eval " | eval " |