======= Storage infrastructure =======


====== Storage targets ======

  * Several Storage Targets available
    * ''%%$HOME%%''
    * ''%%$TMPDIR%%''
    * ''%%$BINFS%%''
    * ''%%$BINFL%%''
    * ''%%$DATA%%''
  * For different purposes
    * Random I/O
    * Small Files
    * Huge Files / Streaming Data


====== The HOME Filesystem (VSC-4) ======

  * Use for software and job scripts
  * Default quota: 100GB
  * Accessible with the ''%%$HOME%%'' environment variable (VSC-4)
    * /home/fs70XXX/username
  * Also available on VSC-3
    * /gpfs/home/fs70XXX/username
  * Check quota

<code>
mmlsquota --block-size auto -j home_fs70XXX home
</code>
<code>
VSC-4 > mmlsquota --block-size auto -j home_fs70824 home
                         Block Limits                                    |     File Limits
Filesystem type         blocks      quota      limit   in_doubt    grace |    files   quota    limit in_doubt
home       FILESET       63.7M       100G       100G          0     none |     3822 1000000  1000000        0 

</code>


====== The TMP filesystem ======

  * Use for
    * Random I/O
    * Many small files
  * Size is up to 50% of main memory
  * Data gets deleted after the job
    * Write Results to ''%%$HOME%%'' or ''%%$GLOBAL%%''
  * Disadvantages
    * Space is consumed from main memory <html><!--* Alternatively the mmap() system call can be used
  * Keep in mind, that mmap() uses lazy loading
  * Very small files waste main memory (memory mapped files are aligned to page-size)--></html>
  * Accessible with the ''%%$TMPDIR%%'' environment variable


====== The DATA Filesystem ======

  * Use for all kind of I/O
  * Default quota: 10TB
    * Extansion can be requested
  * Accessible with the ''%%$DATA%%'' environment variable
    * /gpfs/data/fs70XXX/username
  * Check quota

<code>
mmlsquota --block-size auto -j data_fs70XXX data
</code>
<code>
VSC-4 > mmlsquota --block-size auto -j data_fs70824 data
                         Block Limits                                    |     File Limits
Filesystem type         blocks      quota      limit   in_doubt    grace |    files   quota    limit in_doubt 
data       FILESET           0     9.766T     9.766T          0     none |       14 1000000  1000000        0 

</code>
====== Backup policy ======

  * Backup of user files is **solely the responsibility of each user**
    * [[https://service.vsc.ac.at/slides/introduction-to-vsc/02_connecting_to_VSC/connecting_to_VSC.html#(21)|How to back up my files]]
  * Backed up filesystems:
    * ''%%$HOME%%'' (VSC-4)
    * ''%%$DATA%%'' (VSC-4)
  * Backups are performed on best effort basis
    * Full backup run: ~3 days
  * Backups are used for **disaster recovery only**
  * Project manager can exclude $DATA filesystem from backup
    * [[https://service.vsc.ac.at/|service.vsc.ac.at]]