Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
doku:python [2023/02/23 11:11] – [Jupyterhub] katrin | doku:python [2024/03/19 14:24] (current) – katrin | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Python ====== | + | ====== Python |
+ | |||
+ | Previously we recommended using spack python packages, however we decided to deprecate this approach since it is much easier and user friendly to use ''' | ||
+ | |||
+ | Have a look at the new **quickstart guide** below or the material from the python4hpc training: [[https:// | ||
+ | |||
+ | The old approach is now // | ||
+ | |||
+ | ===== Quickstart: Using Python with Conda ===== | ||
+ | |||
+ | To install a new environment, | ||
+ | |||
+ | This step is optional but we really **recommend** it so you and everyone in your team is able to produce the same python environment. | ||
+ | |||
+ | To find valid conda package names look at [[https:// | ||
+ | |||
+ | Example (add your own packages below dependencies): | ||
+ | < | ||
+ | name: my-env | ||
+ | channels: | ||
+ | - conda-forge | ||
+ | - defaults | ||
+ | dependencies: | ||
+ | - python=3.10 | ||
+ | - tensorflow=2.15.0 | ||
+ | </ | ||
+ | |||
+ | You can the install your environment by using the following commands: | ||
+ | |||
+ | < | ||
+ | # load the miniconda package | ||
+ | $ module load miniconda3 | ||
+ | |||
+ | # executes bash hooks for the tool to function | ||
+ | # and enters the " | ||
+ | $ eval " | ||
+ | |||
+ | # create your environment via environment file | ||
+ | # this will place the environment in your ~/ | ||
+ | # note: | ||
+ | # - make sure to use "conda >env< create" | ||
+ | # - the name is taken from the yaml file; a custom name can be specified with "-n my-custom-name" | ||
+ | (base) $ conda env create --file my_env.yaml | ||
+ | |||
+ | # after creation you can activate the environment to run code in it | ||
+ | (base) $ conda activate my-env | ||
+ | |||
+ | # test python version to make sure we have the right one | ||
+ | (my-env) $ python --version | ||
+ | Python 3.10.11 | ||
+ | </ | ||
+ | |||
+ | You can now start developing and even use the environment in a slurm script | ||
+ | |||
+ | < | ||
+ | # | ||
+ | |||
+ | #SBATCH --job-name=slurm_conda_example | ||
+ | #SBATCH --time=00-00: | ||
+ | #SBATCH --ntasks=2 | ||
+ | #SBATCH --mem=2GB | ||
+ | |||
+ | # modify SBATCH options according to needs | ||
+ | |||
+ | # see "Setup Conda" above or consult " | ||
+ | module load miniconda3 | ||
+ | eval " | ||
+ | conda activate myenv | ||
+ | |||
+ | # print out some info of the python executable in use | ||
+ | # this should point to the python version from " | ||
+ | which python | ||
+ | python --version | ||
+ | </ | ||
+ | |||
+ | ===== More info about Conda ===== | ||
+ | |||
+ | For more information about conda check-out the conda notebook of the python4HPC training material: [[https:// | ||
+ | |||
+ | ===== FAQ ===== | ||
+ | |||
+ | * **''' | ||
+ | |||
+ | Make sure to use the currently provided ''' | ||
+ | |||
+ | * **I dont get the right python version and packages in my slurm batch file environment** - **What is wrong>? | ||
+ | |||
+ | Make sure that one of the first things in your sbatch script is loading the miniconda3 package ''' | ||
+ | |||
+ | * **I don't get GPU/CUDA enabled packages when installing a conda environment** / **I get conda installation errors when i select GPU/CUDA enabled builds for my conda environment*** | ||
+ | |||
+ | In order to install packages from conda-forge that require CUDA on a machine that does not have GPUs you have to set an environment variable before creating the conda environment: | ||
+ | < | ||
+ | name: my-pytorch-gpu-env | ||
+ | channels: | ||
+ | - pytorch | ||
+ | - conda-forge | ||
+ | - defaults | ||
+ | dependencies: | ||
+ | - python=3.12 | ||
+ | - pytorch=2.2.*=*cuda11.8* | ||
+ | </ | ||
+ | |||
+ | < | ||
+ | # for example if you use cuda 11.8 | ||
+ | CONDA_OVERRIDE_CUDA=" | ||
+ | </ | ||
+ | |||
+ | |||
+ | ====== Deprecated Information (work in progress) | ||
===== Python Installations ===== | ===== Python Installations ===== | ||
Line 36: | Line 145: | ||
==== Setup Conda ==== | ==== Setup Conda ==== | ||
- | To load '' | + | To use the '' |
<code bash> | <code bash> | ||
- | spack load miniconda3@4.12.0 | + | module |
</ | </ | ||
- | If you plan to use conda more frequently you can simple add the spack load statement to your '' | + | If you plan to use conda more frequently you can simple add the load statement to your '' |
+ | |||
+ | === Optional: execute conda runtime hooks on login === | ||
+ | |||
+ | To fully utilize the conda command it needs to load runtime hooks. If you want to have conda completely initialized directly after logging in you can execute the following statements to add the conda startup code to your '' | ||
- | Also make sure that you run the following statements if you are setting up conda for the **first time**: | ||
<code bash> | <code bash> | ||
- | conda init bash --dry-run --verbose | grep "# >>> | + | conda init bash --dry-run --verbose | grep "# >>> |
source ~/.bashrc | source ~/.bashrc | ||
</ | </ | ||
- | |||
- | This will add some necessary startup code for conda to your '' | ||
After executing these steps you will see that your prompt changed to '' | After executing these steps you will see that your prompt changed to '' | ||
+ | |||
+ | If you already have an environment you can also add '' | ||
==== Channels ==== | ==== Channels ==== | ||
- | The '' | + | The '' |
- | To use '' | + | Popular channels |
+ | - '' | ||
+ | - '' | ||
- | If you always | + | To use e.g. '' |
+ | |||
+ | If you want to set e.g. '' | ||
<code bash> | <code bash> | ||
conda config --add channels conda-forge | conda config --add channels conda-forge | ||
Line 65: | Line 181: | ||
==== Create your own custom conda environment ==== | ==== Create your own custom conda environment ==== | ||
+ | |||
+ | === Create conda env using environment files === | ||
+ | |||
+ | This is the recommended method to create new conda environments since it makes environment creation reproducible. The file can be easily shared or e.g. added to your version control system | ||
+ | |||
+ | First decide which python version and packages you need. In case you don't know the exact versions upfront you can also just create a first draft of the env-file without pinned version to get the latest libraries. With that information in mind we now write our environment file called " | ||
+ | |||
+ | < | ||
+ | name: myenv | ||
+ | channels: | ||
+ | - conda-forge | ||
+ | dependencies: | ||
+ | - python=3.10 | ||
+ | - pytorch=1.13.1 | ||
+ | </ | ||
+ | |||
+ | After this we run the conda solver to create and install the new environment | ||
+ | |||
+ | < | ||
+ | conda env create -f myenv.yml | ||
+ | </ | ||
+ | |||
+ | Conda will now take its time and solve the environment and then download and install the packages. After this has been done the environment can be activated with | ||
+ | |||
+ | < | ||
+ | conda activate myenv | ||
+ | </ | ||
+ | |||
+ | === Create conda env from commandline === | ||
+ | |||
+ | **Note:** this method is only for illustrating how conda works and is not recommended since it does not create a reproducible environment specification | ||
In order to create your own user environment you need to do the following steps. To also give a short example for a package which we do not provide via spack we will install '' | In order to create your own user environment you need to do the following steps. To also give a short example for a package which we do not provide via spack we will install '' | ||
Line 98: | Line 245: | ||
>>> | >>> | ||
>>> | >>> | ||
+ | </ | ||
+ | |||
+ | === Pytorch === | ||
+ | |||
+ | To install the latest pytorch with conda please follow the instructions on this page: | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | For older and specific combinations of pytorch and cuda please have a look at this page: | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | Eg.: to install v1.13.1 with cuda 11.6 use: | ||
+ | < | ||
+ | conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia | ||
+ | </ | ||
+ | |||
+ | |||
+ | As of 2023-08-25: you can also install the current pytorch version with an older cuda version via: | ||
+ | < | ||
+ | conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c nvidia | ||
</ | </ | ||
===== SLURM ===== | ===== SLURM ===== | ||
- | When using slurm it should be enough to include | + | See the following |
<code bash> | <code bash> | ||
- | spack load miniconda3@4.12.0 | + | # |
+ | |||
+ | #SBATCH --job-name=slurm_conda_example | ||
+ | #SBATCH --time 00-00: | ||
+ | #SBATCH --ntasks=2 | ||
+ | #SBATCH --mem=2GB | ||
+ | |||
+ | # modify SBATCH options according to needs | ||
+ | |||
+ | # see "Setup Conda" above or consult " | ||
+ | module | ||
+ | eval " | ||
conda activate myenv | conda activate myenv | ||
- | < | + | |
+ | # print out some info of the python executable in use | ||
+ | # this should point to the python version from " | ||
+ | which python | ||
+ | python --version | ||
+ | </code> | ||
===== JupyterHub ===== | ===== JupyterHub ===== |