The most recent version of this page is a draft.
This version (2021/05/19 20:59) is a draft.
Approvals: 0/1
This version (2021/05/19 20:59) is a draft.Approvals: 0/1
This is an old revision of the document!
Special hardware (GPUs, binfs) available & how to use it
- Article written by Siegfried Höfinger (VSC Team) <html><br></html>(last update 2020-10-04 by sh).
TOP500 List June 2020
<HTML> <!–slide 1–> <!–for nations flags see https://www.free-country-flags.com–> </HTML>
| Rank | Nation | Machine | Performance | Accelerators |
|---|---|---|---|---|
| 1. |  | Fugaku | 416 PFLOPs/s | |
| 2. |  | Summit | 149 PFLOPs/s | <html><font color=“navy”></html>NVIDIA V100<html></font></html> |
| 3. | | Sierra | 95 PFLOPs/s | <html><font color=“navy”></html>NVIDIA V100<html></font></html> |
| 4. |  | Sunway TaihuLight | 93 PFLOPs/s | |
| 5. | | Tianhe-2A | 62 PFLOPs/s | |
| 6. |  | HPC5 | 36 PFLOPs/s | <html><font color=“navy”></html>NVIDIA V100<html></font></html> |
| 7. | | Selene | 28 PFLOPs/s | <html><font color=“navy”></html>NVIDIA A100<html></font></html> |
| 8. | | Frontera | 24 PFLOPs/s | <html><font color=“navy”></html>NVIDIA RTX5000/V100<html></font></html> |
| 9. | | Marconi-100 | 22 PFLOPs/s | <html><font color=“navy”></html>NVIDIA V100<html></font></html> |
| 10. |  | Piz Daint | 21 PFLOPs/s | <html><font color=“navy”></html>NVIDIA P100<html></font></html> |
<HTML> <!–slide 2–> </HTML>
Components on VSC-3+
| Model | #cores | Clock Freq (GHz) | Memory (GB) | Bandwidth (GB/s) | TDP (Watt) | FP32/FP64 (GFLOPs/s) |
|---|---|---|---|---|---|---|
| <html><font color=“navy”></html>42+51x GeForce GTX-1080 n37[1,2,3]-[001-006,001-022,001-026]<html></font></html> | ||||||
 | 2560 | 1.61 | 8 | 320 | 180 | 8228/257 |
| <html><font color=“navy”></html>4x Tesla k20m n372-02[4,5]<html></font></html> | ||||||
 | 2496 | 0.71 | 5 | 208 | 195 | 3520/1175 |
| <html><font color=“navy”></html>1x Tesla V100 n372-023<html></font></html> | ||||||
 | 5120/644 | 1.31 | 32 | 900 | 250 | 14000/7000 |
| <html><font color=“navy”></html>19x GeForce RTX-2080Ti n375-[001-019]<html></font></html> | ||||||
 | 4352 | 1.35 | 11 | 616 | 250 | 13450/420 |
| <html><font color=“navy”></html>Avalable soon: 42×2 nVidia A40<html></font></html> | ||||||
| 10752 | 1.305 | 48 | 696 | 300 | 37400/1169 |
<HTML> <!–slide 3–> </HTML>
Working on GPU nodes
Interactive mode
1. VSC-3 > salloc -N 1 -p gpu_gtx1080single --qos gpu_gtx1080single
2. VSC-3 > squeue -u $USER
3. VSC-3 > srun -n 1 hostname (...while still on the login node !)
4. VSC-3 > ssh n372-012 (...or whatever else node had been assigned)
5. VSC-3 > module load cuda/9.1.85
cd ~/examples/09_special_hardware/gpu_gtx1080/matrixMul
nvcc ./matrixMul.cu
./a.out
cd ~/examples/09_special_hardware/gpu_gtx1080/matrixMulCUBLAS
nvcc matrixMulCUBLAS.cu -lcublas
./a.out
6. VSC-3 > nvidia-smi
7. VSC-3 > /opt/sw/x86_64/glibc-2.17/ivybridge-ep/cuda/9.1.85/NVIDIA_CUDA-9.1_Samples/1_Utilities/deviceQuery/deviceQuery
<HTML> <!–slide 4–> </HTML>
Working on GPU nodes cont.
SLURM submission gpu_test.scrpt
#!/bin/bash # # usage: sbatch ./gpu_test.scrpt # #SBATCH -J gtx1080 #SBATCH -N 1 #SBATCH --partition gpu_gtx1080single #SBATCH --qos gpu_gtx1080single module purge module load cuda/9.1.85 nvidia-smi /opt/sw/x86_64/glibc-2.17/ivybridge-ep/cuda/9.1.85/NVIDIA_CUDA-9.1_Samples/1_Utilities/deviceQuery/deviceQuery
<html><font color=“navy”></html>Exercise/Example/Problem:<html></font></html> <html><br/></html> Using interactive mode or batch submission, figure out whether we have ECC enabled on GPUs of type gtx1080 ?
<HTML> <!–slide 5–> </HTML>
Real-World Example, AMBER-16
| Performance | Power Efficiency |
|---|---|
 |  |