User Tools

Site Tools


doku:storage

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

doku:storage [2022/01/04 13:31] (current)
jz created
Line 1: Line 1:
 +======= Storage infrastructure =======
 +
 +
 +
 +====== Storage targets ======
 +
 +  * Several Storage Targets available
 +    * ''%%$HOME%%''
 +    * ''%%$TMPDIR%%''
 +    * ''%%$BINFS%%''
 +    * ''%%$BINFL%%''
 +    * ''%%$DATA%%''
 +  * For different purposes
 +    * Random I/O
 +    * Small Files
 +    * Huge Files / Streaming Data
 +
 +
 +====== The HOME Filesystem (VSC-3) ======
 +
 +  * Use for non I/O intensive jobs
 +  * Basically NFS Exports over infiniband (no RDMA)
 +  * Logical volumes of projects are distributed among the servers
 +    * Each logical volume belongs to 1 NFS server
 +  * Accessible with the ''%%$HOME%%'' environment variable
 +    * /home/lv70XXX/username
 +
 +====== The HOME Filesystem (VSC-4) ======
 +
 +  * Use for software and job scripts
 +  * Default quota: 100GB
 +  * Accessible with the ''%%$HOME%%'' environment variable (VSC-4)
 +    * /home/fs70XXX/username
 +  * Also available on VSC-3
 +    * /gpfs/home/fs70XXX/username
 +  * Check quota
 +
 +<code>
 +mmlsquota --block-size auto -j home_fs70XXX home
 +</code>
 +<code>
 +VSC-4 > mmlsquota --block-size auto -j home_fs70824 home
 +                         Block Limits                                    |     File Limits
 +Filesystem type         blocks      quota      limit   in_doubt    grace |    files   quota    limit in_doubt
 +home       FILESET       63.7M       100G       100G          0     none |     3822 1000000  1000000        0 
 +
 +</code>
 +
 +====== The BINFL filesystem ======
 +
 +  * Specifically designed for Bioinformatics applications
 +  * Use for I/O intensive jobs
 +  * ~ 1 PB Space (default quota is 10GB/project)
 +    * Can be increased on request (subject to availability)
 +  * BeeGFS Filesystem
 +  * Accessible via ''%%$BINFL%%'' environment variable
 +    * ''%%$BINFL%%'' … ///binfl/lv70XXX/username//
 +  * Also available on VSC-4
 +  * Check quota
 +
 +<code>
 +    beegfs-ctl --getquota --cfgFile=/etc/beegfs/hdd_storage.d/beegfs-client.conf --gid 70XXX
 +</code>
 +<code>
 +VSC-3 > beegfs-ctl --getquota --cfgFile=/etc/beegfs/hdd_storage.d/beegfs-client.conf --gid 70824
 +      user/group     ||           size          ||    chunk files    
 +     name      id  ||    used    |    hard    ||  used    hard   
 +--------------|------||------------|------------||---------|---------
 +        p70824| 70824||    5.93 MiB|   10.00 GiB||      574|  1000000
 +
 +</code>
 +====== The BINFS filesystem ======
 +
 +  * Specifically designed for Bioinformatics applications
 +  * Use for very I/O intensive jobs
 +  * ~ 100 TB Space (default quota is 2GB/project)
 +    * Can be increased on request (subject to availability)
 +  * BeeGFS Filesystem
 +  * Accessible via ''%%$BINFS%%'' environment variable
 +    * ''%%$BINFS%%'' … ///binfs/lv70XXX/username//
 +  * Also available on VSC-4
 +  * Check quota
 +
 +<code>
 +    beegfs-ctl --getquota --cfgFile=/etc/beegfs/nvme_storage.d/beegfs-client.conf --gid 70XXX
 +</code>
 +<code>
 +VSC-3 > beegfs-ctl --getquota --cfgFile=/etc/beegfs/nvme_storage.d/beegfs-client.conf --gid 70824
 +      user/group     ||           size          ||    chunk files    
 +     name      id  ||    used    |    hard    ||  used    hard   
 +--------------|------||------------|------------||---------|---------
 +        p70824| 70824||      0 Byte|    2.00 GiB||        0|     2000
 +
 +</code>
 +
 +====== The TMP filesystem ======
 +
 +  * Use for
 +    * Random I/O
 +    * Many small files
 +  * Size is up to 50% of main memory
 +  * Data gets deleted after the job
 +    * Write Results to ''%%$HOME%%'' or ''%%$GLOBAL%%''
 +  * Disadvantages
 +    * Space is consumed from main memory <html><!--* Alternatively the mmap() system call can be used
 +  * Keep in mind, that mmap() uses lazy loading
 +  * Very small files waste main memory (memory mapped files are aligned to page-size)--></html>
 +  * Accessible with the ''%%$TMPDIR%%'' environment variable
 +
 +
 +
 +====== The DATA Filesystem ======
 +
 +  * Use for all kind of I/O
 +  * Default quota: 10TB
 +    * Extansion can be requested
 +  * Accessible with the ''%%$DATA%%'' environment variable (VSC-4)
 +    * /data/fs70XXX/username
 +  * Also available on VSC-3
 +    * /gpfs/data/fs70XXX/username
 +  * Check quota
 +
 +<code>
 +mmlsquota --block-size auto -j data_fs70XXX data
 +</code>
 +<code>
 +VSC-4 > mmlsquota --block-size auto -j data_fs70824 data
 +                         Block Limits                                    |     File Limits
 +Filesystem type         blocks      quota      limit   in_doubt    grace |    files   quota    limit in_doubt 
 +data       FILESET               9.766T     9.766T          0     none |       14 1000000  1000000        0 
 +
 +</code>
 +====== Backup policy ======
 +
 +  * Backup of user files is **solely the responsibility of each user**
 +    * [[https://service.vsc.ac.at/slides/introduction-to-vsc/02_connecting_to_VSC/connecting_to_VSC.html#(21)|How to back up my files]]
 +  * Backed up filesystems:
 +    * ''%%$HOME%%'' (VSC-3)
 +    * ''%%$HOME%%'' (VSC-4)
 +    * ''%%$DATA%%'' (VSC-4)
 +  * Backups are performed on best effort basis
 +    * Full backup run: ~3 days
 +  * Backups are used for **disaster recovery only**
 +  * Project manager can exclude $DATA filesystem from backup
 +    * [[https://service.vsc.ac.at/|service.vsc.ac.at]]
  
doku/storage.txt · Last modified: 2022/01/04 13:31 by jz