In addition to $HOME, which is fine to use for standard jobs with rather few small files (<1000 files, overall size <1G), there are a number of specialized scratch directories.

The Fraunhofer parallel cluster file system (FhGFS)/BeeGFS is used in $GLOBAL and $SCRATCH.

Please use the environment variable $GLOBAL to access your personal scratch space. Access is available from the compute and login nodes. The variable expands as e.g.:

$ echo $GLOBAL
/global/lv70999/username

The directory is writeable as user and readable by the group members. It is advisable to make use of these directories in particular for jobs with heavy I/O operations. In addition it will reduce the load on the fileserver holding the $HOME directories.

The Fraunhofer parallel file system is shared by all users and by all nodes. Single jobs producing heavy load (»1000 requests per second) have been observed to reduce responsiveness for all jobs and all users.

Lifetime of data is limited, see table below.

Local scratch directories on each node are provided as a link to the Fraunhofer parallel file system and can thus be viewed also via the login nodes as '/fhgfs/rXXnXX/'. The parallel file system (and thus the performance) is identical between $SCRATCH and $GLOBAL. The variable $SCRATCH expands as:

$ echo $SCRATCH
/scratch

These directories are purged after job execution.

For smaller files and very fast access, restricted to single nodes, the variables $TMP or $TMPDIR may be used which expand equally to

$ echo $TMP -- $TMPDIR
/tmp/123456.789.queue.q -- /tmp/123456.789.queue.q

These directories are purged after job execution.

Please refrain from writing directly to '/tmp'!

The newest, still experimental, scratch file system $JOBLOCAL is a common temporary storage within a user job. The 'joblocal' file system may be requested with

-v JOBLOCAL_FILESYSTEM=TRUE

All nodes within a job access the same files under '/joblocal', which is purged after job execution.

This method scales very well up to several hundred similar jobs. Although the file system has 32GB, it is recommended to use only a few GB.

To save files at the job end, use, e.g.,

cd /joblocal; tar czf ${HOME}/joblocal_${JOB_NAME}_${JOB_ID}.tgz myfiles

in your user epilog script.

If there are many files (»1000), please refrain from plain copying to $HOME or $GLOBAL at the job end.

Implementation details: $JOBLOCAL is implemented via SCSI RDMA Protocol (SRP) and NFS. Very high performance for small files is achieved by extensive caching on the jobs master node, which acts as (job internal) NFS server.