Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
pandoc:introduction-to-vsc:08_storage_infrastructure:storage_infrastructure [2019/01/15 15:39] – Pandoc Auto-commit pandocpandoc:introduction-to-vsc:08_storage_infrastructure:storage_infrastructure [2020/10/20 09:13] (current) – Pandoc Auto-commit pandoc
Line 5: Line 5:
  
  
-====== Storage hardware ======+====== Storage hardware VSC-3 ======
  
   * Storage on VSC-3   * Storage on VSC-3
Line 30: Line 30:
 ====== Storage performance ====== ====== Storage performance ======
  
-{{:pandoc:introduction-to-vsc:08_storage_infrastructure:storage_infrastructure:vsc3_storage_performance.png}}+{{.:vsc3_storage_performance.png}}
  
-====== The HOME Filesystem ======+====== The HOME Filesystem (VSC-3) ======
  
   * Use for non I/O intensive jobs   * Use for non I/O intensive jobs
   * Basically NFS Exports over infiniband (no RDMA)   * Basically NFS Exports over infiniband (no RDMA)
-  * Targets with up to 24 Disks (RAID-6 on VSC-3) 
-  * Up to 2 Gigabyte/second write speed 
   * Logical volumes of projects are distributed among the servers   * Logical volumes of projects are distributed among the servers
     * Each logical volume belongs to 1 NFS server     * Each logical volume belongs to 1 NFS server
Line 49: Line 47:
     * Can be increased on request (subject to availability)     * Can be increased on request (subject to availability)
   * BeeGFS Filesystem   * BeeGFS Filesystem
-  * Metadata Servers 
-    * Metadata on SSDs (RAID-1) 
-    * 8 Metadata Targets for VSC-3 
-  * Object Storages 
-    * Disk Storages (RAID-6 on VSC-3) 
-    * VSC-3: 12 Disks per Target / 4 Targets per Server / 8 Servers total 
-  * Up to 20 Gigabyte/second write speed 
   * Accessible via the ''%%$GLOBAL%%'' and ''%%$SCRATCH%%'' environment variables   * Accessible via the ''%%$GLOBAL%%'' and ''%%$SCRATCH%%'' environment variables
     * ''%%$GLOBAL%%'' … ///global/lv70XXX/username//     * ''%%$GLOBAL%%'' … ///global/lv70XXX/username//
     * ''%%$SCRATCH%%'' … ///scratch//     * ''%%$SCRATCH%%'' … ///scratch//
 +  * Check quota
  
 +<code>
 +    beegfs-ctl --getquota --cfgFile=/etc/beegfs/global3.d/beegfs-client.conf --gid 70XXX
 +</code>
 +<code>
 +VSC-3 > beegfs-ctl --getquota --cfgFile=/etc/beegfs/global3.d/beegfs-client.conf --gid 70824
 +      user/group     ||           size          ||    chunk files    
 +     name     |  id  ||    used    |    hard    ||  used   |  hard   
 +--------------|------||------------|------------||---------|---------
 +        p70824| 70824||      0 Byte|  500.00 GiB||        0|   100000
 +
 +</code>
 ====== The BINFL filesystem ====== ====== The BINFL filesystem ======
  
Line 67: Line 70:
     * Can be increased on request (subject to availability)     * Can be increased on request (subject to availability)
   * BeeGFS Filesystem   * BeeGFS Filesystem
-  * Metadata Servers 
-    * Metadata on Datacenter SSDs (RAID-10) 
-    * 8 Metadata Servers 
-  * Object Storages 
-    * Disk Storages configured as RAID-6 
-    * 12 Disks per Target / 1 Target per Server / 16 Servers total 
-  * Up to 40 Gigabyte/second write speed 
   * Accessible via ''%%$BINFL%%'' environment variable   * Accessible via ''%%$BINFL%%'' environment variable
     * ''%%$BINFL%%'' … ///binfl/lv70XXX/username//     * ''%%$BINFL%%'' … ///binfl/lv70XXX/username//
 +  * Also available on VSC-4
 +  * Check quota
  
 +<code>
 +    beegfs-ctl --getquota --cfgFile=/etc/beegfs/hdd_storage.d/beegfs-client.conf --gid 70XXX
 +</code>
 +<code>
 +VSC-3 > beegfs-ctl --getquota --cfgFile=/etc/beegfs/hdd_storage.d/beegfs-client.conf --gid 70824
 +      user/group     ||           size          ||    chunk files    
 +     name     |  id  ||    used    |    hard    ||  used   |  hard   
 +--------------|------||------------|------------||---------|---------
 +        p70824| 70824||    5.93 MiB|   10.00 GiB||      574|  1000000
 +
 +</code>
 ====== The BINFS filesystem ====== ====== The BINFS filesystem ======
  
Line 84: Line 93:
     * Can be increased on request (subject to availability)     * Can be increased on request (subject to availability)
   * BeeGFS Filesystem   * BeeGFS Filesystem
-  * Metadata Servers 
-    * Metadata on Datacenter SSDs (RAID-10) 
-    * 8 Metadata Servers 
-  * Object Storages 
-    * Datacenter SSDs are used instead of traditional disks. 
-      * No redundancy. See it as (very) fast and low-latency scratch space. Data may be lost after a hardware failure. 
-    * 4x Intel P3600 2TB Datacenter SSDs per Server 
-    * 16 Storage Servers 
-  * Up to 80 Gigabyte/second via OmniPath Interconnect 
   * Accessible via ''%%$BINFS%%'' environment variable   * Accessible via ''%%$BINFS%%'' environment variable
     * ''%%$BINFS%%'' … ///binfs/lv70XXX/username//     * ''%%$BINFS%%'' … ///binfs/lv70XXX/username//
 +  * Also available on VSC-4
 +  * Check quota
  
 +<code>
 +    beegfs-ctl --getquota --cfgFile=/etc/beegfs/nvme_storage.d/beegfs-client.conf --gid 70XXX
 +</code>
 +<code>
 +VSC-3 > beegfs-ctl --getquota --cfgFile=/etc/beegfs/nvme_storage.d/beegfs-client.conf --gid 70824
 +      user/group     ||           size          ||    chunk files    
 +     name     |  id  ||    used    |    hard    ||  used   |  hard   
 +--------------|------||------------|------------||---------|---------
 +        p70824| 70824||      0 Byte|    2.00 GiB||        0|     2000
 +
 +</code>
 ====== The TMP filesystem ====== ====== The TMP filesystem ======
  
Line 109: Line 122:
   * Very small files waste main memory (memory mapped files are aligned to page-size)--></html>   * Very small files waste main memory (memory mapped files are aligned to page-size)--></html>
   * Accessible with the ''%%$TMPDIR%%'' environment variable   * Accessible with the ''%%$TMPDIR%%'' environment variable
 +
 +====== Storage hardware VSC-4 ======
 +
 +  * Storage on VSC-4
 +    * 1 Server for ''%%$HOME%%''
 +    * 6 Servers for ''%%$DATA%%''
 +    * 720 spinning disks
 +    * 16 NVMEs flash drives
 +
 +====== The HOME Filesystem (VSC-4) ======
 +
 +  * Use for software and job scripts
 +  * Default quota: 100GB
 +  * Accessible with the ''%%$HOME%%'' environment variable (VSC-4)
 +    * /home/fs70XXX/username
 +  * Also available on VSC-3
 +    * /gpfs/home/fs70XXX/username
 +  * Check quota
 +
 +<code>
 +mmlsquota --block-size auto -j home_fs70XXX home
 +</code>
 +<code>
 +VSC-4 > mmlsquota --block-size auto -j home_fs70824 home
 +                         Block Limits                                    |     File Limits
 +Filesystem type         blocks      quota      limit   in_doubt    grace |    files   quota    limit in_doubt
 +home       FILESET       63.7M       100G       100G          0     none |     3822 1000000  1000000        0 
 +
 +</code>
 +====== The DATA Filesystem ======
 +
 +  * Use for all kind of I/O
 +  * Default quota: 10TB
 +    * Extansion can be requested
 +  * Accessible with the ''%%$DATA%%'' environment variable (VSC-4)
 +    * /data/fs70XXX/username
 +  * Also available on VSC-3
 +    * /gpfs/data/fs70XXX/username
 +  * Check quota
 +
 +<code>
 +mmlsquota --block-size auto -j data_fs70XXX data
 +</code>
 +<code>
 +VSC-4 > mmlsquota --block-size auto -j data_fs70824 data
 +                         Block Limits                                    |     File Limits
 +Filesystem type         blocks      quota      limit   in_doubt    grace |    files   quota    limit in_doubt 
 +data       FILESET           0     9.766T     9.766T          0     none |       14 1000000  1000000        0 
 +
 +</code>
 +====== Backup policy ======
 +
 +  * Backup of user files is **solely the responsibility of each user**
 +    * [[https://service.vsc.ac.at/slides/introduction-to-vsc/02_connecting_to_VSC/connecting_to_VSC.html#(21)|How to back up my files]]
 +  * Backed up filesystems:
 +    * ''%%$HOME%%'' (VSC-3)
 +    * ''%%$HOME%%'' (VSC-4)
 +    * ''%%$DATA%%'' (VSC-4)
 +  * Backups are performed on best effort basis
 +    * Full backup run: ~3 days
 +  * Backups are used for **disaster recovery only**
 +  * Project manager can exclude $DATA filesystem from backup
 +    * [[https://service.vsc.ac.at/|service.vsc.ac.at]]
  
 ====== Storage exercises ====== ====== Storage exercises ======
Line 133: Line 209:
 We will now measure the sequential performance of the different storage targets on VSC-3. We will now measure the sequential performance of the different storage targets on VSC-3.
  
-<HTML><ol style="list-style-type: lower-alpha;"></HTML> +  - With one process
-<HTML><li></HTML>With one process<HTML></li></HTML><HTML></ol></HTML>+
  
 <code> <code>
Line 190: Line 265:
 We will now measure the storage performance for tiny 4kilobyte random writes. We will now measure the storage performance for tiny 4kilobyte random writes.
  
-<HTML><ol style="list-style-type: lower-alpha;"></HTML> +  - With one process
-<HTML><li></HTML>With one process<HTML></li></HTML><HTML></ol></HTML>+
  
 <code> <code>
Line 246: Line 320:
  
 ---- ----
 +
  
  • pandoc/introduction-to-vsc/08_storage_infrastructure/storage_infrastructure.1547566778.txt.gz
  • Last modified: 2019/01/15 15:39
  • by pandoc