This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong. ======= Storage infrastructure ======= ====== Storage targets ====== * Several Storage Targets available * ''%%$HOME%%'' * ''%%$TMPDIR%%'' * ''%%$BINFS%%'' * ''%%$BINFL%%'' * ''%%$DATA%%'' * For different purposes * Random I/O * Small Files * Huge Files / Streaming Data ====== The HOME Filesystem (VSC-4) ====== * Use for software and job scripts * Default quota: 100GB * Accessible with the ''%%$HOME%%'' environment variable (VSC-4) * /home/fs70XXX/username * Also available on VSC-3 * /gpfs/home/fs70XXX/username * Check quota <code> mmlsquota --block-size auto -j home_fs70XXX home </code> <code> VSC-4 > mmlsquota --block-size auto -j home_fs70824 home Block Limits | File Limits Filesystem type blocks quota limit in_doubt grace | files quota limit in_doubt home FILESET 63.7M 100G 100G 0 none | 3822 1000000 1000000 0 </code> ====== The TMP filesystem ====== * Use for * Random I/O * Many small files * Size is up to 50% of main memory * Data gets deleted after the job * Write Results to ''%%$HOME%%'' or ''%%$GLOBAL%%'' * Disadvantages * Space is consumed from main memory <html><!--* Alternatively the mmap() system call can be used * Keep in mind, that mmap() uses lazy loading * Very small files waste main memory (memory mapped files are aligned to page-size)--></html> * Accessible with the ''%%$TMPDIR%%'' environment variable ====== The DATA Filesystem ====== * Use for all kind of I/O * Default quota: 10TB * Extansion can be requested * Accessible with the ''%%$DATA%%'' environment variable * /gpfs/data/fs70XXX/username * Check quota <code> mmlsquota --block-size auto -j data_fs70XXX data </code> <code> VSC-4 > mmlsquota --block-size auto -j data_fs70824 data Block Limits | File Limits Filesystem type blocks quota limit in_doubt grace | files quota limit in_doubt data FILESET 0 9.766T 9.766T 0 none | 14 1000000 1000000 0 </code> ====== Backup policy ====== * Backup of user files is **solely the responsibility of each user** * [[https://service.vsc.ac.at/slides/introduction-to-vsc/02_connecting_to_VSC/connecting_to_VSC.html#(21)|How to back up my files]] * Backed up filesystems: * ''%%$HOME%%'' (VSC-4) * ''%%$DATA%%'' (VSC-4) * Backups are performed on best effort basis * Full backup run: ~3 days * Backups are used for **disaster recovery only** * Project manager can exclude $DATA filesystem from backup * [[https://service.vsc.ac.at/|service.vsc.ac.at]] doku/storage.txt Last modified: 2023/06/26 08:06by goldenberg