In addition to $HOME, which is fine to use for standard jobs with rather few small files (<1000 files, overall size <1G), there are a number of specialized scratch directories.

The Fraunhofer parallel cluster file system (FhGFS)/BeeGFS is used in $GLOBAL and $SCRATCH.

Global Personal Scratch Directories $GLOBAL

Please use the environment variable $GLOBAL to access your personal scratch space. Access is available from the compute and login nodes. The variable expands as e.g.:

$ echo $GLOBAL
/global/lv70999/username

The directory is writeable as user and readable by the group members. It is advisable to make use of these directories in particular for jobs with heavy I/O operations. In addition it will reduce the load on the fileserver holding the $HOME directories.

The Fraunhofer parallel file system is shared by all users and by all nodes. Single jobs producing heavy load (»1000 requests per second) have been observed to reduce responsiveness for all jobs and all users.

Lifetime of data is limited, see table below.

Per-node Scratch Directories $SCRATCH

Local scratch directories on each node are provided as a link to the Fraunhofer parallel file system and can thus be viewed also via the login nodes as '/fhgfs/rXXnXX/'. The parallel file system (and thus the performance) is identical between $SCRATCH and $GLOBAL. The variable $SCRATCH expands as:

$ echo $SCRATCH
/scratch

These directories are purged after job execution.

Local temporary ram disk $TMPDIR

For smaller files and very fast access, restricted to single nodes, the variables $TMP or $TMPDIR may be used which expand equally to

$ echo $TMP -- $TMPDIR
/tmp/123456.789.queue.q -- /tmp/123456.789.queue.q

These directories are purged after job execution.

Please refrain from writing directly to '/tmp'!

Joblocal scratch directory $JOBLOCAL

The newest, still experimental, scratch file system $JOBLOCAL is a common temporary storage within a user job. The 'joblocal' file system may be requested with

-v JOBLOCAL_FILESYSTEM=TRUE

All nodes within a job access the same files under '/joblocal', which is purged after job execution.

This method scales very well up to several hundred similar jobs. Although the file system has 32GB, it is recommended to use only a few GB.

To save files at the job end, use, e.g.,

cd /joblocal; tar czf ${HOME}/joblocal_${JOB_NAME}_${JOB_ID}.tgz myfiles

in your user epilog script.

If there are many files (»1000), please refrain from plain copying to $HOME or $GLOBAL at the job end.

Implementation details: $JOBLOCAL is implemented via SCSI RDMA Protocol (SRP) and NFS. Very high performance for small files is achieved by extensive caching on the jobs master node, which acts as (job internal) NFS server.

Comparison of scratch directories

$GLOBAL || $SCRATCH $TMPDIR || $JOBLOCAL (experimental)
Recommended file size large large small small
Lifetime files older than 180 days deleted if $GLOBAL space is running low job job job
Size x00 TB (for all users) x00 TB (for all users) a few GB (within memory) about 5 GB (hard limit: 32GB)
Scaling does not fit very large number of small file IO does not fit very large number of small file IO no problem (local) no problem (local)
Visibility global node (see above) node job
Recommended usage large files, available temporarily after job life large files small files, or many seek-operations within a file many small files (>1000), or many seek-operations within a file
  • doku/scratch.txt
  • Last modified: 2015/03/23 09:31
  • by dieter