User Tools

Site Tools


doku:gpu

GPU Nodes on VSC-1

On VSC-1 currently two different kind of GPU Nodes are available:

  • Four Nodes with two NVIDIA Tesla C2050 GPU cards each. One Tesla GPU features 448 CUDA Cores and the card provides 3GB of GDDR5 memory. The Nodes are equipped with two Intel X5650 Westmere 2.67 GHz CPUs and 24 GB RAM.
  • Two Nodes with two NVIDIA Kepler K20m GPU cards each. One K20m GPU features 2496 CUDA cores and provides 5GB of GDDR5 memory. The Nodes are equipped with two Intel Xeon E5-2680 2.70 GHz CPUs and 256 GB RAM.

Access to nodes is managed via a dedicated queue called 'fermi' / 'kepler'. If you would like to have access to this queue please contact system administration and specify the user who wants to access the GPU nodes. The username needs to be one of an already existing VSC user. After a user has been added, the nodes can be access interactively via:

#switch user group first, then login
sg fermi             
qrsh -q fermi 

or

#switch user group first, then login
sg fermi             
qrsh -q kepler

Alternatively you can submit a job script which has in its header the parameters:

#$ -q fermi
#$ -P fermi

or

#$ -q kepler
#$ -P fermi

The usage of the additional -pe smp 6 parameter is currently not mandatory, but strongly encouraged. In order to avoid more than two jobs requesting GPU's on one node (each node has two GPU units) half of the 12 CPU cores of the node should be requested by the job.

Job submission has to be done using the 'qsub.py' wrapper script:

qsub.py job.sh

The runtime consumed on the GPU nodes will not be deducted from your VSC account.

GPU CUDA Documentation

Available software tools are the nvcc compiler, cuda libraries, and cula tools. The following variables are defined (automatically set in your environment) in order to use these:

CULA_ROOT="/opt/sw/cula"
CULA_INC_PATH="$CULA_ROOT/include"
CULA_BIN_PATH_64="$CULA_ROOT/bin64"
CULA_LIB_PATH_64="$CULA_ROOT/lib64"
PATH=/opt/sw/cuda/bin:$PATH
LD_LIBRARY_PATH=/opt/sw/cuda/lib64:/opt/sw/cuda/computeprof/bin:$CULA_LIB_PATH_64:$LD_LIBRARY_PATH

Example programs can be found in '/opt/sw/cula/examples/'.
Extensive documentation here: '/opt/sw/cuda/doc/'.

NVIDIA System Management Interface program - nvidia-smi

can be used to monitor the GPU usage, e.g.

[@r18n45 ~]# nvidia-smi -i 1 -q -d UTILIZATION,MEMORY -l

==============NVSMI LOG==============

Timestamp                       : Tue Jun 28 14:09:58 2011

Driver Version                  : 270.40

Attached GPUs                   : 2

GPU 0:84:0
    Memory Usage
        Total                   : 2687 Mb
        Used                    : 101 Mb
        Free                    : 2586 Mb
    Utilization
        Gpu                     : 99 %
        Memory                  : 0 %

for more options see man nvidia-smi

Code examples and exercises

Examples of C and fortran code can be found here:

/opt/sw/gpu-doc/exercises

There are exercises with templates and also possible solutions in the subdirectories C and fortran and the respective solutions directory.

CUDA C References

CUDA Libraries References

CUDA Fermi References

CUDA Fortran Refernce

CUDA PTX Reference

Release Notes

doku/gpu.txt · Last modified: 2014/10/02 13:21 by ir