===== GPU Nodes on VSC-1 =====

On VSC-1 currently two different kind of GPU Nodes are available:

  * Four Nodes with two NVIDIA Tesla C2050 GPU cards each. One Tesla GPU features 448 CUDA Cores and the card provides 3GB of GDDR5 memory. The Nodes are equipped with two Intel X5650 Westmere 2.67 GHz CPUs and 24 GB RAM.
  * Two Nodes with two NVIDIA Kepler K20m GPU cards each. One K20m GPU features 2496 CUDA cores and provides 5GB of GDDR5 memory. The Nodes are equipped with two Intel Xeon E5-2680 2.70 GHz CPUs and 256 GB RAM.

Access to nodes is managed via a dedicated queue called 'fermi' / 'kepler'. If you would like to have access to this queue please 
**[[doku:contact|contact system administration]]**
and specify the user who wants to access the GPU nodes. 
The username needs to be one of an already existing VSC user. After a user has been added, the nodes can be access interactively via:

<code>
#switch user group first, then login
sg fermi             
qrsh -q fermi 
</code>

or

<code>
#switch user group first, then login
sg fermi             
qrsh -q kepler
</code>



Alternatively you can submit a job script which has in its header the parameters:
<code>
#$ -q fermi
#$ -P fermi
</code>

or 

<code>
#$ -q kepler
#$ -P fermi
</code>


The usage of the additional -pe smp 6 parameter is currently not mandatory, but strongly encouraged. 
In order to avoid more than two jobs requesting GPU's on one node (each node has two GPU units) half of the 12
CPU cores of the node should be requested by the job.

Job submission has to be done using the 'qsub.py' wrapper script:

<code>
qsub.py job.sh
</code>


The runtime consumed on the GPU nodes will not be deducted from your VSC account.



===== GPU CUDA Documentation =====


Available software tools are the nvcc compiler, cuda libraries, and cula tools.
The following variables are defined (automatically set in your environment) in order to use these:
<code>
CULA_ROOT="/opt/sw/cula"
CULA_INC_PATH="$CULA_ROOT/include"
CULA_BIN_PATH_64="$CULA_ROOT/bin64"
CULA_LIB_PATH_64="$CULA_ROOT/lib64"
PATH=/opt/sw/cuda/bin:$PATH
LD_LIBRARY_PATH=/opt/sw/cuda/lib64:/opt/sw/cuda/computeprof/bin:$CULA_LIB_PATH_64:$LD_LIBRARY_PATH
</code>

Example programs can be found in '/opt/sw/cula/examples/'.\\
Extensive documentation here: '/opt/sw/cuda/doc/'.


====  NVIDIA System Management Interface program - nvidia-smi ====
can be used to monitor the GPU usage, e.g.
<code>[@r18n45 ~]# nvidia-smi -i 1 -q -d UTILIZATION,MEMORY -l

==============NVSMI LOG==============

Timestamp                       : Tue Jun 28 14:09:58 2011

Driver Version                  : 270.40

Attached GPUs                   : 2

GPU 0:84:0
    Memory Usage
        Total                   : 2687 Mb
        Used                    : 101 Mb
        Free                    : 2586 Mb
    Utilization
        Gpu                     : 99 %
        Memory                  : 0 %</code>

for more options see ''man nvidia-smi''

==== Code examples and exercises ====

Examples of C and fortran code can be found here:
<code>
/opt/sw/gpu-doc/exercises
</code>
There are exercises with templates and also possible solutions in the subdirectories ''C'' and ''fortran'' and the respective ''solutions'' directory.


==== CUDA C References ====
{{:doku:cuda-docu:cuda_c_best_practices_guide_3.2.pdf|CUDA C best practices}}\\
{{:doku:cuda-docu:cuda_c_programming_guide_3.2.pdf|CUDA C programming guide}}\\
{{:doku:cuda-docu:cuda_gdb_user_manual_3.2.pdf|CUDA GDB user manual}}\\
{{:doku:cuda-docu:cuda_toolkit_reference_manual_3.2beta.pdf|CUDA toolkit reference}}\\

==== CUDA Libraries References ====
{{:doku:cuda-docu:cublas_library_1008.pdf|CUBLAS library}}\\
{{:doku:cuda-docu:cufft_library_1008.pdf|CUFFT library}}\\
{{:doku:cuda-docu:curand_library_1008.pdf|CURAN library}}\\
{{:doku:cuda-docu:cusparse_library_1008.pdf|CUSPARSE library}}\\

==== CUDA Fermi References ====
{{:doku:cuda-docu:fermi_compatibility_guide_1.3.pdf|Fermi compatibility guide}}\\
{{:doku:cuda-docu:fermi_tuning_guide_1.3.pdf|Fermi tuning guide}}\\
{{:doku:cuda-docu:nvidia_fermi_compute_architecture_whitepaper.pdf|NVIDIA fermi compute archtiecture - whitepaper}}\\

==== CUDA Fortran Refernce ====
{{:doku:cuda-docu:pgi_cuda_fortran_users_guide_1002.pdf|PGI CUDA fortran user guide}}\\

==== CUDA PTX Reference ====
{{:doku:cuda-docu:ptx_isa_2.2.pdf|PTX documentation}}

==== Release Notes ====
{{:doku:cuda-docu:compute_visual_profiler_release_notes_linux.txt|}}\\
{{:doku:cuda-docu:cuda_profiler_3.0.txt|}}