This version is outdated by a newer approved version.This version (2021/05/14 18:29) is a draft.
Approvals: 0/1
Approvals: 0/1
This is an old revision of the document!
Special hardware (GPUs, binfs) available & how to use it
- Article written by Siegfried Höfinger (VSC Team) <html><br></html>(last update 2020-10-04 by sh).
TOP500 List June 2020
<HTML> <!–slide 1–> <!–for nations flags see https://www.free-country-flags.com–> </HTML>
<HTML> <!–slide 2–> </HTML>
Components on VSC-3
<HTML> <!–slide 3–> </HTML>
Working on GPU nodes
Interactive mode
1. VSC-3 > salloc -N 1 -p gpu_gtx1080single --qos gpu_gtx1080single 2. VSC-3 > squeue -u $USER 3. VSC-3 > srun -n 1 hostname (...while still on the login node !) 4. VSC-3 > ssh n372-012 (...or whatever else node had been assigned) 5. VSC-3 > module load cuda/9.1.85 cd ~/examples/09_special_hardware/gpu_gtx1080/matrixMul nvcc ./matrixMul.cu ./a.out cd ~/examples/09_special_hardware/gpu_gtx1080/matrixMulCUBLAS nvcc matrixMulCUBLAS.cu -lcublas ./a.out 6. VSC-3 > nvidia-smi 7. VSC-3 > /opt/sw/x86_64/glibc-2.17/ivybridge-ep/cuda/9.1.85/NVIDIA_CUDA-9.1_Samples/1_Utilities/deviceQuery/deviceQuery
<HTML> <!–slide 4–> </HTML>
Working on GPU nodes cont.
SLURM submission gpu_test.scrpt
#!/bin/bash # # usage: sbatch ./gpu_test.scrpt # #SBATCH -J gtx1080 #SBATCH -N 1 #SBATCH --partition gpu_gtx1080single #SBATCH --qos gpu_gtx1080single module purge module load cuda/9.1.85 nvidia-smi /opt/sw/x86_64/glibc-2.17/ivybridge-ep/cuda/9.1.85/NVIDIA_CUDA-9.1_Samples/1_Utilities/deviceQuery/deviceQuery
<html><font color=“navy”></html>Exercise/Example/Problem:<html></font></html> <html><br/></html> Using interactive mode or batch submission, figure out whether we have ECC enabled on GPUs of type gtx1080 ?
<HTML> <!–slide 5–> </HTML>
Working on binf nodes
Interactive mode
1. VSC-3 > salloc -N 1 -p binf --qos normal_binf -C binf -L intel@vsc (... add --nodelist binf-13 to request a specific node) 2. VSC-3 > squeue -u $USER 3. VSC-3 > srun -n 4 hostname (... while still on the login node !) 4. VSC-3 > ssh binf-11 (... or whatever else node had been assigned) 5. VSC-3 > module purge 6. VSC-3 > module load intel/17 cd examples/09_special_hardware/binf icc -xHost -qopenmp sample.c export OMP_NUM_THREADS=8 ./a.out
<HTML> <!–slide 8–> </HTML>
Working on binf nodes cont.
SLURM submission slrm.sbmt.scrpt
#!/bin/bash # # usage: sbatch ./slrm.sbmt.scrpt # #SBATCH -J gmxbinfs #SBATCH -N 2 #SBATCH --partition binf #SBATCH --qos normal_binf #SBATCH -C binf #SBATCH --ntasks-per-node 24 #SBATCH --ntasks-per-core 1 module purge module load intel/17 intel-mkl/2017 intel-mpi/2017 gromacs/5.1.4_binf export I_MPI_PIN=1 export I_MPI_PIN_PROCESSOR_LIST=0-23 export I_MPI_FABRICS=shm:tmi export I_MPI_TMI_PROVIDER=psm2 export OMP_NUM_THREADS=1 export MDRUN_ARGS=" -dd 0 0 0 -rdd 0 -rcon 0 -dlb yes -dds 0.8 -tunepme -v -nsteps 10000 " mpirun -np $SLURM_NTASKS gmx_mpi mdrun ${MDRUN_ARGS} -s hSERT_5HT_PROD.0.tpr -deffnm hSERT_5HT_PROD.0 -px hSERT_5HT_PROD.0_px.xvg -pf hSERT_5HT_PROD.0_pf.xvg -swap hSERT_5HT_PROD.0.xvg
<HTML> <!–slide 9–> </HTML>