Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision | ||
doku:python [2023/04/05 12:24] – [Channels] katrin | doku:python [2024/02/22 08:59] – katrin | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Python ====== | + | ====== Python |
+ | |||
+ | Previously we recommended using spack python packages, however we decided to deprecate this approach since it is much easier and user friendly to use ''' | ||
+ | |||
+ | Have a look at the new **quickstart guide** below or the material from the python4hpc training: [[https:// | ||
+ | |||
+ | The old approach is now // | ||
+ | |||
+ | ===== Quickstart: Using Python with Conda ===== | ||
+ | |||
+ | To install a new environment, | ||
+ | |||
+ | This step is optional but we really **recommend** it so you and everyone in your team is able to produce the same python environment. | ||
+ | |||
+ | To find valid conda package names look at [[https:// | ||
+ | |||
+ | Example (add your own packages below dependencies): | ||
+ | < | ||
+ | name: my-env | ||
+ | channels: | ||
+ | - conda-forge | ||
+ | - defaults | ||
+ | dependencies: | ||
+ | - python=3.10 | ||
+ | - tensorflow=2.15.0 | ||
+ | </ | ||
+ | |||
+ | You can the install your environment by using the following commands: | ||
+ | |||
+ | < | ||
+ | # load the miniconda package | ||
+ | $ module load miniconda3 | ||
+ | |||
+ | # executes bash hooks for the tool to function | ||
+ | # and enters the " | ||
+ | $ eval " | ||
+ | |||
+ | # create your environment via environment file | ||
+ | # this will place the environment in your ~/ | ||
+ | # note: make sure to use "conda >env< create" | ||
+ | (base) $ conda env create -n my-env --file my_env.yaml | ||
+ | |||
+ | # after creation you can activate the environment to run code in it | ||
+ | (base) $ conda activate my-env | ||
+ | |||
+ | # test python version to make sure we have the right one | ||
+ | (my-env) $ python --version | ||
+ | Python 3.10.11 | ||
+ | </ | ||
+ | |||
+ | You can now start developing and even use the environment in a slurm script | ||
+ | |||
+ | < | ||
+ | # | ||
+ | |||
+ | #SBATCH --job-name=slurm_conda_example | ||
+ | #SBATCH --time 00-00: | ||
+ | #SBATCH --ntasks=2 | ||
+ | #SBATCH --mem=2GB | ||
+ | |||
+ | # modify SBATCH options according to needs | ||
+ | |||
+ | # see "Setup Conda" above or consult " | ||
+ | module load miniconda3 | ||
+ | eval " | ||
+ | conda activate myenv | ||
+ | |||
+ | # print out some info of the python executable in use | ||
+ | # this should point to the python version from " | ||
+ | which python | ||
+ | python --version | ||
+ | </ | ||
+ | |||
+ | ==== More info about Conda ==== | ||
+ | |||
+ | For more information about conda check-out the conda notebook of the python4HPC training material: [[https:// | ||
+ | |||
+ | ==== FAQ ==== | ||
+ | |||
+ | * **''' | ||
+ | |||
+ | Make sure to use the currently provided ''' | ||
+ | |||
+ | * **I dont get the right python version and packages** / **I see the error ''' | ||
+ | |||
+ | The default for sbatch is to use the current user environment and start the job in the context of this environment. Since conda is depending on environment variables being setup correctly you need to make sure to start the job script from a clean environment. | ||
+ | |||
+ | Alternatively you can also use sbatch with the ''' | ||
+ | |||
+ | ====== Deprecated Information (work in progress) | ||
===== Python Installations ===== | ===== Python Installations ===== | ||
Line 36: | Line 125: | ||
==== Setup Conda ==== | ==== Setup Conda ==== | ||
- | To load '' | + | To use the '' |
<code bash> | <code bash> | ||
- | # VSC-4 | + | module load miniconda3 |
- | module load miniconda3/ | + | |
- | + | ||
- | # VSC-5 | + | |
- | module load miniconda3/ | + | |
</ | </ | ||
If you plan to use conda more frequently you can simple add the load statement to your '' | If you plan to use conda more frequently you can simple add the load statement to your '' | ||
- | === Optional: execute conda hooks on login === | + | === Optional: execute conda runtime |
- | If you want to have every conda function available | + | To fully utilize the conda command it needs to load runtime hooks. |
<code bash> | <code bash> | ||
- | conda init bash --dry-run --verbose | grep "# >>> | + | conda init bash --dry-run --verbose | grep "# >>> |
source ~/.bashrc | source ~/.bashrc | ||
</ | </ | ||
Line 140: | Line 225: | ||
>>> | >>> | ||
>>> | >>> | ||
+ | </ | ||
+ | |||
+ | === Pytorch === | ||
+ | |||
+ | To install the latest pytorch with conda please follow the instructions on this page: | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | For older and specific combinations of pytorch and cuda please have a look at this page: | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | Eg.: to install v1.13.1 with cuda 11.6 use: | ||
+ | < | ||
+ | conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia | ||
+ | </ | ||
+ | |||
+ | |||
+ | As of 2023-08-25: you can also install the current pytorch version with an older cuda version via: | ||
+ | < | ||
+ | conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c nvidia | ||
</ | </ | ||