Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
Next revisionBoth sides next revision
doku:python [2023/06/19 08:37] – katrindoku:python [2024/02/22 16:38] – katrin
Line 3: Line 3:
 Previously we recommended using spack python packages, however we decided to deprecate this approach since it is much easier and user friendly to use '''conda''' which is already widely known.  Previously we recommended using spack python packages, however we decided to deprecate this approach since it is much easier and user friendly to use '''conda''' which is already widely known. 
  
-The documentation is currently in the process of being revised. For now please either use the old information that is still available OR use the new quickstart guide below.+Have a look at the new **quickstart guide** below or the material from the python4hpc training: [[https://gitlab.tuwien.ac.at/vsc-public/training/python4hpc|python4hpc public repo]].
  
-===== (Conda) Quickstart =====+The old approach is now //deprecated// but kept on this page for reference.
  
-To install a new environment start with creating an environment yaml file and store it in a safe location (ideally in your source code repository). This step is optional but we really recommend it so you and everyone in your team is able to produce the same python environment.+===== Quickstart: Using Python with Conda ===== 
 + 
 +To install a new environment, start with creating an **environment yaml file** and store it in a safe location (ideally in your source code repository).  
 + 
 +This step is optional but we really **recommend** it so you and everyone in your team is able to produce the same python environment. 
 + 
 +To find valid conda package names look at [[https://anaconda.org/search|Anaconda repo package search]]. 
 + 
 +Example (add your own packages below dependencies):
 <code> <code>
 name: my-env name: my-env
 +channels:
 +  - conda-forge
 +  - defaults
 dependencies: dependencies:
   - python=3.10   - python=3.10
-  - tensorflow-gpu=2.6.0+  - tensorflow=2.15.0
 </code> </code>
  
-Afterwards you can then use conda like this:+You can the install your environment by using the following commands:
  
 <code> <code>
Line 26: Line 37:
  
 # create your environment via environment file # create your environment via environment file
-(base) $ conda create -n my-env --file my_env.yaml+# this will place the environment in your ~/.conda/envs folder 
 +# note: make sure to use "conda >env< create" - "conda create" is a different command 
 +(base) $ conda env create -n my-env --file my_env.yaml
  
 # after creation you can activate the environment to run code in it # after creation you can activate the environment to run code in it
Line 36: Line 49:
 </code> </code>
  
-You can now start developing and even just use the environment in a slurm script+You can now start developing and even use the environment in a slurm script
  
 <code> <code>
Line 42: Line 55:
  
 #SBATCH --job-name=slurm_conda_example #SBATCH --job-name=slurm_conda_example
-#SBATCH --time 00-00:05:00+#SBATCH --time=00-00:05:00
 #SBATCH --ntasks=2 #SBATCH --ntasks=2
 #SBATCH --mem=2GB #SBATCH --mem=2GB
Line 49: Line 62:
  
 # see "Setup Conda" above or consult "module avail miniconda3" to get the right package name # see "Setup Conda" above or consult "module avail miniconda3" to get the right package name
-module load MINICONDA3_PACKAGE_NAME+module load miniconda3
 eval "$(conda shell.bash hook)" eval "$(conda shell.bash hook)"
 conda activate myenv conda activate myenv
Line 58: Line 71:
 python --version python --version
 </code> </code>
 +
 +==== More info about Conda ====
 +
 +For more information about conda check-out the conda notebook of the python4HPC training material: [[https://gitlab.tuwien.ac.at/vsc-public/training/python4hpc/-/blob/main/D1_02_env_03_conda.ipynb|python4HPC Conda Environments]]
  
 ==== FAQ ==== ==== FAQ ====
  
-**I see the error '''ModuleNotFoundError: No module named "conda"''' in my slurm job logfile. What is wrong>?**+  * **'''eval "$(conda shell.bash hook)"''' fails with '''CommandNotFoundError ...'''**
  
-The default for sbatch is to use the current user environment and start the job in the context of this environment. Since conda is depending on environment variables being setup correctly you need to ake sure to start the job script from a clean environment. Alternatively you can also use sbatch with the '''--export=NONE''' argument.+Make sure to use the currently provided '''miniconda3''' module via '''module load miniconda3''' 
 + 
 +  * **I dont get the right python version and packages** / **I see the error '''ModuleNotFoundError: No module named "conda"''' in my slurm job logfile** - **What is wrong>?** 
 + 
 +The default for sbatch is to use the current user environment and start the job in the context of this environment. Since conda is depending on environment variables being setup correctly you need to make sure to start the job script from a clean environment.  
 + 
 +Alternatively you can also use sbatch with the '''--export=NONE''' argument from commandline or add an sbatch directive in your job script: "#SBATCH --export=NONE"
  
 ====== Deprecated Information (work in progress) ====== ====== Deprecated Information (work in progress) ======
Line 114: Line 137:
  
 <code bash> <code bash>
-conda init bash --dry-run --verbose | grep "# >>> conda initialize" -A 100 | grep "# <<< conda initialize" -B 100 | sed 's/+//g' > ~/.bashrc+conda init bash --dry-run --verbose | grep "# >>> conda initialize" -A 100 | grep "# <<< conda initialize" -B 100 | sed 's/+//g' >> ~/.bashrc
 source ~/.bashrc source ~/.bashrc
 </code> </code>
Line 202: Line 225:
 >>> import phono3py >>> import phono3py
 >>> exit() >>> exit()
 +</code>
 +
 +=== Pytorch ===
 +
 +To install the latest pytorch with conda please follow the instructions on this page:
 +
 +[[https://pytorch.org/get-started/locally/|Latest Pytorch installation]]
 +
 +For older and specific combinations of pytorch and cuda please have a look at this page:
 +
 +[[https://pytorch.org/get-started/previous-versions/#v182-with-lts-support|Previous Pytorch versions]]
 +
 +Eg.: to install v1.13.1 with cuda 11.6 use:
 +<code>
 +conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia
 +</code>
 +
 +
 +As of 2023-08-25: you can also install the current pytorch version with an older cuda version via:
 +<code>
 +conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c nvidia
 </code> </code>
  
  • doku/python.txt
  • Last modified: 2024/03/19 14:24
  • by katrin