Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
pandoc:introduction-to-vsc:08_storage_infrastructure:storage_infrastructure [2018/10/16 13:29] – Pandoc Auto-commit pandocpandoc:introduction-to-vsc:08_storage_infrastructure:storage_infrastructure [2020/10/20 09:13] (current) – Pandoc Auto-commit pandoc
Line 1: Line 1:
 ====== Storage infrastructure ====== ====== Storage infrastructure ======
  
-  * Article written by Siegfried Reinwald (VSC Team) <html><br></html>(last update 2017-04-27 by sr).+  * Article written by Siegfried Reinwald (VSC Team) <html><br></html>(last update 2019-01-15 by sh).
  
  
  
-====== Storage hardware ======+====== Storage hardware VSC-3 ======
  
   * Storage on VSC-3   * Storage on VSC-3
Line 30: Line 30:
 ====== Storage performance ====== ====== Storage performance ======
  
-{{:pandoc:introduction-to-vsc:08_storage_infrastructure:storage_infrastructure:vsc3_storage_performance.png}}+{{.:vsc3_storage_performance.png}}
  
-====== The HOME Filesystem ======+====== The HOME Filesystem (VSC-3) ======
  
   * Use for non I/O intensive jobs   * Use for non I/O intensive jobs
   * Basically NFS Exports over infiniband (no RDMA)   * Basically NFS Exports over infiniband (no RDMA)
-  * Targets with up to 24 Disks (RAID-6 on VSC-3) 
-  * Up to 2 Gigabyte/second write speed 
   * Logical volumes of projects are distributed among the servers   * Logical volumes of projects are distributed among the servers
     * Each logical volume belongs to 1 NFS server     * Each logical volume belongs to 1 NFS server
Line 49: Line 47:
     * Can be increased on request (subject to availability)     * Can be increased on request (subject to availability)
   * BeeGFS Filesystem   * BeeGFS Filesystem
-  * Metadata Servers 
-    * Metadata on SSDs (RAID-1) 
-    * 8 Metadata Targets for VSC-3 
-  * Object Storages 
-    * Disk Storages (RAID-6 on VSC-3) 
-    * VSC-3: 12 Disks per Target / 4 Targets per Server / 8 Servers total 
-  * Up to 20 Gigabyte/second write speed 
   * Accessible via the ''%%$GLOBAL%%'' and ''%%$SCRATCH%%'' environment variables   * Accessible via the ''%%$GLOBAL%%'' and ''%%$SCRATCH%%'' environment variables
     * ''%%$GLOBAL%%'' … ///global/lv70XXX/username//     * ''%%$GLOBAL%%'' … ///global/lv70XXX/username//
     * ''%%$SCRATCH%%'' … ///scratch//     * ''%%$SCRATCH%%'' … ///scratch//
 +  * Check quota
  
 +<code>
 +    beegfs-ctl --getquota --cfgFile=/etc/beegfs/global3.d/beegfs-client.conf --gid 70XXX
 +</code>
 +<code>
 +VSC-3 > beegfs-ctl --getquota --cfgFile=/etc/beegfs/global3.d/beegfs-client.conf --gid 70824
 +      user/group     ||           size          ||    chunk files    
 +     name     |  id  ||    used    |    hard    ||  used   |  hard   
 +--------------|------||------------|------------||---------|---------
 +        p70824| 70824||      0 Byte|  500.00 GiB||        0|   100000
 +
 +</code>
 ====== The BINFL filesystem ====== ====== The BINFL filesystem ======
  
Line 67: Line 70:
     * Can be increased on request (subject to availability)     * Can be increased on request (subject to availability)
   * BeeGFS Filesystem   * BeeGFS Filesystem
-  * Metadata Servers 
-    * Metadata on Datacenter SSDs (RAID-10) 
-    * 8 Metadata Servers 
-  * Object Storages 
-    * Disk Storages configured as RAID-6 
-    * 12 Disks per Target / 1 Target per Server / 16 Servers total 
-  * Up to 40 Gigabyte/second write speed 
   * Accessible via ''%%$BINFL%%'' environment variable   * Accessible via ''%%$BINFL%%'' environment variable
     * ''%%$BINFL%%'' … ///binfl/lv70XXX/username//     * ''%%$BINFL%%'' … ///binfl/lv70XXX/username//
 +  * Also available on VSC-4
 +  * Check quota
  
 +<code>
 +    beegfs-ctl --getquota --cfgFile=/etc/beegfs/hdd_storage.d/beegfs-client.conf --gid 70XXX
 +</code>
 +<code>
 +VSC-3 > beegfs-ctl --getquota --cfgFile=/etc/beegfs/hdd_storage.d/beegfs-client.conf --gid 70824
 +      user/group     ||           size          ||    chunk files    
 +     name     |  id  ||    used    |    hard    ||  used   |  hard   
 +--------------|------||------------|------------||---------|---------
 +        p70824| 70824||    5.93 MiB|   10.00 GiB||      574|  1000000
 +
 +</code>
 ====== The BINFS filesystem ====== ====== The BINFS filesystem ======
  
Line 84: Line 93:
     * Can be increased on request (subject to availability)     * Can be increased on request (subject to availability)
   * BeeGFS Filesystem   * BeeGFS Filesystem
-  * Metadata Servers 
-    * Metadata on Datacenter SSDs (RAID-10) 
-    * 8 Metadata Servers 
-  * Object Storages 
-    * Datacenter SSDs are used instead of traditional disks. 
-      * No redundancy. See it as (very) fast and low-latency scratch space. Data may be lost after a hardware failure. 
-    * 4x Intel P3600 2TB Datacenter SSDs per Server 
-    * 16 Storage Servers 
-  * Up to 80 Gigabyte/second via OmniPath Interconnect 
   * Accessible via ''%%$BINFS%%'' environment variable   * Accessible via ''%%$BINFS%%'' environment variable
     * ''%%$BINFS%%'' … ///binfs/lv70XXX/username//     * ''%%$BINFS%%'' … ///binfs/lv70XXX/username//
 +  * Also available on VSC-4
 +  * Check quota
  
 +<code>
 +    beegfs-ctl --getquota --cfgFile=/etc/beegfs/nvme_storage.d/beegfs-client.conf --gid 70XXX
 +</code>
 +<code>
 +VSC-3 > beegfs-ctl --getquota --cfgFile=/etc/beegfs/nvme_storage.d/beegfs-client.conf --gid 70824
 +      user/group     ||           size          ||    chunk files    
 +     name     |  id  ||    used    |    hard    ||  used   |  hard   
 +--------------|------||------------|------------||---------|---------
 +        p70824| 70824||      0 Byte|    2.00 GiB||        0|     2000
 +
 +</code>
 ====== The TMP filesystem ====== ====== The TMP filesystem ======
  
Line 109: Line 122:
   * Very small files waste main memory (memory mapped files are aligned to page-size)--></html>   * Very small files waste main memory (memory mapped files are aligned to page-size)--></html>
   * Accessible with the ''%%$TMPDIR%%'' environment variable   * Accessible with the ''%%$TMPDIR%%'' environment variable
 +
 +====== Storage hardware VSC-4 ======
 +
 +  * Storage on VSC-4
 +    * 1 Server for ''%%$HOME%%''
 +    * 6 Servers for ''%%$DATA%%''
 +    * 720 spinning disks
 +    * 16 NVMEs flash drives
 +
 +====== The HOME Filesystem (VSC-4) ======
 +
 +  * Use for software and job scripts
 +  * Default quota: 100GB
 +  * Accessible with the ''%%$HOME%%'' environment variable (VSC-4)
 +    * /home/fs70XXX/username
 +  * Also available on VSC-3
 +    * /gpfs/home/fs70XXX/username
 +  * Check quota
 +
 +<code>
 +mmlsquota --block-size auto -j home_fs70XXX home
 +</code>
 +<code>
 +VSC-4 > mmlsquota --block-size auto -j home_fs70824 home
 +                         Block Limits                                    |     File Limits
 +Filesystem type         blocks      quota      limit   in_doubt    grace |    files   quota    limit in_doubt
 +home       FILESET       63.7M       100G       100G          0     none |     3822 1000000  1000000        0 
 +
 +</code>
 +====== The DATA Filesystem ======
 +
 +  * Use for all kind of I/O
 +  * Default quota: 10TB
 +    * Extansion can be requested
 +  * Accessible with the ''%%$DATA%%'' environment variable (VSC-4)
 +    * /data/fs70XXX/username
 +  * Also available on VSC-3
 +    * /gpfs/data/fs70XXX/username
 +  * Check quota
 +
 +<code>
 +mmlsquota --block-size auto -j data_fs70XXX data
 +</code>
 +<code>
 +VSC-4 > mmlsquota --block-size auto -j data_fs70824 data
 +                         Block Limits                                    |     File Limits
 +Filesystem type         blocks      quota      limit   in_doubt    grace |    files   quota    limit in_doubt 
 +data       FILESET           0     9.766T     9.766T          0     none |       14 1000000  1000000        0 
 +
 +</code>
 +====== Backup policy ======
 +
 +  * Backup of user files is **solely the responsibility of each user**
 +    * [[https://service.vsc.ac.at/slides/introduction-to-vsc/02_connecting_to_VSC/connecting_to_VSC.html#(21)|How to back up my files]]
 +  * Backed up filesystems:
 +    * ''%%$HOME%%'' (VSC-3)
 +    * ''%%$HOME%%'' (VSC-4)
 +    * ''%%$DATA%%'' (VSC-4)
 +  * Backups are performed on best effort basis
 +    * Full backup run: ~3 days
 +  * Backups are used for **disaster recovery only**
 +  * Project manager can exclude $DATA filesystem from backup
 +    * [[https://service.vsc.ac.at/|service.vsc.ac.at]]
  
 ====== Storage exercises ====== ====== Storage exercises ======
Line 125: Line 201:
 mkdir my_directory_name mkdir my_directory_name
 cd my_directory_name cd my_directory_name
-cp -r ~training/08_storage_infrastructure/*Benchmark ./+cp -r ~training/examples/08_storage_infrastructure/*Benchmark ./
 </code> </code>
 Keep in mind that the results will vary, because there are other users working on the storage targets. Keep in mind that the results will vary, because there are other users working on the storage targets.
Line 133: Line 209:
 We will now measure the sequential performance of the different storage targets on VSC-3. We will now measure the sequential performance of the different storage targets on VSC-3.
  
-<HTML><ol style="list-style-type: lower-alpha;"></HTML> +  - With one process
-<HTML><li></HTML>With one process<HTML></li></HTML><HTML></ol></HTML>+
  
 <code> <code>
Line 158: Line 233:
   * The performance of which storage targets improves with the number of processes? Why?   * The performance of which storage targets improves with the number of processes? Why?
   * What could you do to further improve the performance of the sequential write throughput? What could be a problem with that?   * What could you do to further improve the performance of the sequential write throughput? What could be a problem with that?
-  * Bonus Question: ''%%$TMPDIR%%'' seems to scale pretty well with the number of processes although its an in-memory filesystem. Why is that happening?+  * Bonus Question: ''%%$TMPDIR%%'' seems to scale pretty well with the number of processes although it is an in-memory filesystem. Why is that happening?
  
  
Line 188: Line 263:
 ====== Exercise 2 - Random I/O ====== ====== Exercise 2 - Random I/O ======
  
-We will now measure the storage performance when confronted with tiny 4kilobyte random writes.+We will now measure the storage performance for tiny 4kilobyte random writes.
  
-<HTML><ol style="list-style-type: lower-alpha;"></HTML> +  - With one process
-<HTML><li></HTML>With one process<HTML></li></HTML><HTML></ol></HTML>+
  
 <code> <code>
Line 246: Line 320:
  
 ---- ----
 +
  
  • pandoc/introduction-to-vsc/08_storage_infrastructure/storage_infrastructure.1539696572.txt.gz
  • Last modified: 2018/10/16 13:29
  • by pandoc