===== Scratch Directories on VSC_2 =====

In addition to $HOME, which is fine to use for standard jobs with rather few small files (<1000 files, overall size <1G), there are a number of specialized scratch directories.

The [[http://www.fhgfs.com/cms/documentation|Fraunhofer parallel cluster file system (FhGFS)/BeeGFS]] is used in $GLOBAL and $SCRATCH.

=== Global Personal Scratch Directories $GLOBAL ===

Please use the environment variable ''$GLOBAL'' to access your personal scratch space. Access is available from the compute and login nodes. The variable expands as e.g.:
<code>
$ echo $GLOBAL
/global/lv70999/username
</code>
The directory is writeable as user and readable by the group members. It is advisable to make use of these directories in particular for jobs with heavy I/O operations. In addition it will reduce the load on the fileserver holding the $HOME directories.

The Fraunhofer parallel file system is shared by all users and by all nodes.
Single jobs producing heavy load (>>1000 requests per second) have been observed to reduce responsiveness for all jobs and all users.

Lifetime of data is limited, see table below.

=== Per-node Scratch Directories $SCRATCH ===

Local scratch directories on each node are provided as a link to the Fraunhofer parallel file system and can thus be viewed also via the login nodes as '''/fhgfs/rXXnXX/'''. 
The parallel file system (and thus the performance) is identical between $SCRATCH and $GLOBAL.
The variable ''$SCRATCH'' expands as:
<code>
$ echo $SCRATCH
/scratch
</code>
These directories are purged after job execution.

=== Local temporary ram disk $TMPDIR ===

For smaller files and very fast access, restricted to single nodes, the variables ''$TMP'' or ''$TMPDIR'' may be used which expand equally to
<code>
$ echo $TMP -- $TMPDIR
/tmp/123456.789.queue.q -- /tmp/123456.789.queue.q
</code>
These directories are purged after job execution.

Please refrain from writing directly to '''/tmp'''!

=== Joblocal scratch directory $JOBLOCAL ===

The newest, still experimental, scratch file system $JOBLOCAL is a common temporary storage within a user job.
The '''joblocal''' file system may be requested with
<code>
-v JOBLOCAL_FILESYSTEM=TRUE
</code>
All nodes within a job access the same files under '''/joblocal''', which is purged after job execution.

This method scales very well up to several hundred similar jobs.
Although the file system has 32GB, it is recommended to use only a few GB.

To save files at the job end, use, e.g.,
<code>
cd /joblocal; tar czf ${HOME}/joblocal_${JOB_NAME}_${JOB_ID}.tgz myfiles
</code>
in your [[prolog|user epilog]] script.

If there are many files (>>1000), please refrain from plain copying to $HOME or $GLOBAL at the job end.

Implementation details: ''$JOBLOCAL'' is implemented via [[http://en.wikipedia.org/wiki/SCSI_RDMA_Protocol| SCSI RDMA Protocol (SRP)]] and [[http://en.wikipedia.org/wiki/Network_File_System|NFS]].
Very high performance for small files is achieved by extensive caching on the jobs master node, which acts as (job internal) NFS server.

=== Comparison of scratch directories ===

|                      || $GLOBAL                        || $SCRATCH               || $TMPDIR                 || $JOBLOCAL (experimental) ||
| Recommended file size || large                         || large                  || small                   || small ||
| Lifetime             || files older than 180 days deleted if $GLOBAL space is running low     || job                    || job                     || job ||
| Size                 || x00 TB (for all users)         || x00 TB (for all users) || a few GB (within memory) || about 5 GB (hard limit: 32GB) ||
| Scaling              || does not fit very large number of small file IO || does not fit very large number of small file IO  || no problem (local) || no problem (local) ||
| Visibility           || global                         || node (see above)      || node                    || job ||
| Recommended usage     || large files, available temporarily after job life || large files    || small files, or many seek-operations within a file || many small files (>1000), or many seek-operations within a file ||