Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
pandoc:introduction-to-vsc:08_storage_infrastructure:storage_infrastructure [2019/01/15 15:39] – Pandoc Auto-commit pandoc | pandoc:introduction-to-vsc:08_storage_infrastructure:storage_infrastructure [2020/10/20 09:13] (current) – Pandoc Auto-commit pandoc | ||
---|---|---|---|
Line 5: | Line 5: | ||
- | ====== Storage hardware ====== | + | ====== Storage hardware |
* Storage on VSC-3 | * Storage on VSC-3 | ||
Line 30: | Line 30: | ||
====== Storage performance ====== | ====== Storage performance ====== | ||
- | {{: | + | {{.: |
- | ====== The HOME Filesystem ====== | + | ====== The HOME Filesystem |
* Use for non I/O intensive jobs | * Use for non I/O intensive jobs | ||
* Basically NFS Exports over infiniband (no RDMA) | * Basically NFS Exports over infiniband (no RDMA) | ||
- | * Targets with up to 24 Disks (RAID-6 on VSC-3) | ||
- | * Up to 2 Gigabyte/ | ||
* Logical volumes of projects are distributed among the servers | * Logical volumes of projects are distributed among the servers | ||
* Each logical volume belongs to 1 NFS server | * Each logical volume belongs to 1 NFS server | ||
Line 49: | Line 47: | ||
* Can be increased on request (subject to availability) | * Can be increased on request (subject to availability) | ||
* BeeGFS Filesystem | * BeeGFS Filesystem | ||
- | * Metadata Servers | ||
- | * Metadata on SSDs (RAID-1) | ||
- | * 8 Metadata Targets for VSC-3 | ||
- | * Object Storages | ||
- | * Disk Storages (RAID-6 on VSC-3) | ||
- | * VSC-3: 12 Disks per Target / 4 Targets per Server / 8 Servers total | ||
- | * Up to 20 Gigabyte/ | ||
* Accessible via the '' | * Accessible via the '' | ||
* '' | * '' | ||
* '' | * '' | ||
+ | * Check quota | ||
+ | < | ||
+ | beegfs-ctl --getquota --cfgFile=/ | ||
+ | </ | ||
+ | < | ||
+ | VSC-3 > beegfs-ctl --getquota --cfgFile=/ | ||
+ | user/ | ||
+ | | ||
+ | --------------|------||------------|------------||---------|--------- | ||
+ | p70824| 70824|| | ||
+ | |||
+ | </ | ||
====== The BINFL filesystem ====== | ====== The BINFL filesystem ====== | ||
Line 67: | Line 70: | ||
* Can be increased on request (subject to availability) | * Can be increased on request (subject to availability) | ||
* BeeGFS Filesystem | * BeeGFS Filesystem | ||
- | * Metadata Servers | ||
- | * Metadata on Datacenter SSDs (RAID-10) | ||
- | * 8 Metadata Servers | ||
- | * Object Storages | ||
- | * Disk Storages configured as RAID-6 | ||
- | * 12 Disks per Target / 1 Target per Server / 16 Servers total | ||
- | * Up to 40 Gigabyte/ | ||
* Accessible via '' | * Accessible via '' | ||
* '' | * '' | ||
+ | * Also available on VSC-4 | ||
+ | * Check quota | ||
+ | < | ||
+ | beegfs-ctl --getquota --cfgFile=/ | ||
+ | </ | ||
+ | < | ||
+ | VSC-3 > beegfs-ctl --getquota --cfgFile=/ | ||
+ | user/ | ||
+ | | ||
+ | --------------|------||------------|------------||---------|--------- | ||
+ | p70824| 70824|| | ||
+ | |||
+ | </ | ||
====== The BINFS filesystem ====== | ====== The BINFS filesystem ====== | ||
Line 84: | Line 93: | ||
* Can be increased on request (subject to availability) | * Can be increased on request (subject to availability) | ||
* BeeGFS Filesystem | * BeeGFS Filesystem | ||
- | * Metadata Servers | ||
- | * Metadata on Datacenter SSDs (RAID-10) | ||
- | * 8 Metadata Servers | ||
- | * Object Storages | ||
- | * Datacenter SSDs are used instead of traditional disks. | ||
- | * No redundancy. See it as (very) fast and low-latency scratch space. Data may be lost after a hardware failure. | ||
- | * 4x Intel P3600 2TB Datacenter SSDs per Server | ||
- | * 16 Storage Servers | ||
- | * Up to 80 Gigabyte/ | ||
* Accessible via '' | * Accessible via '' | ||
* '' | * '' | ||
+ | * Also available on VSC-4 | ||
+ | * Check quota | ||
+ | < | ||
+ | beegfs-ctl --getquota --cfgFile=/ | ||
+ | </ | ||
+ | < | ||
+ | VSC-3 > beegfs-ctl --getquota --cfgFile=/ | ||
+ | user/ | ||
+ | | ||
+ | --------------|------||------------|------------||---------|--------- | ||
+ | p70824| 70824|| | ||
+ | |||
+ | </ | ||
====== The TMP filesystem ====== | ====== The TMP filesystem ====== | ||
Line 109: | Line 122: | ||
* Very small files waste main memory (memory mapped files are aligned to page-size)--></ | * Very small files waste main memory (memory mapped files are aligned to page-size)--></ | ||
* Accessible with the '' | * Accessible with the '' | ||
+ | |||
+ | ====== Storage hardware VSC-4 ====== | ||
+ | |||
+ | * Storage on VSC-4 | ||
+ | * 1 Server for '' | ||
+ | * 6 Servers for '' | ||
+ | * 720 spinning disks | ||
+ | * 16 NVMEs flash drives | ||
+ | |||
+ | ====== The HOME Filesystem (VSC-4) ====== | ||
+ | |||
+ | * Use for software and job scripts | ||
+ | * Default quota: 100GB | ||
+ | * Accessible with the '' | ||
+ | * / | ||
+ | * Also available on VSC-3 | ||
+ | * / | ||
+ | * Check quota | ||
+ | |||
+ | < | ||
+ | mmlsquota --block-size auto -j home_fs70XXX home | ||
+ | </ | ||
+ | < | ||
+ | VSC-4 > mmlsquota --block-size auto -j home_fs70824 home | ||
+ | Block Limits | ||
+ | Filesystem type | ||
+ | home | ||
+ | |||
+ | </ | ||
+ | ====== The DATA Filesystem ====== | ||
+ | |||
+ | * Use for all kind of I/O | ||
+ | * Default quota: 10TB | ||
+ | * Extansion can be requested | ||
+ | * Accessible with the '' | ||
+ | * / | ||
+ | * Also available on VSC-3 | ||
+ | * / | ||
+ | * Check quota | ||
+ | |||
+ | < | ||
+ | mmlsquota --block-size auto -j data_fs70XXX data | ||
+ | </ | ||
+ | < | ||
+ | VSC-4 > mmlsquota --block-size auto -j data_fs70824 data | ||
+ | Block Limits | ||
+ | Filesystem type | ||
+ | data | ||
+ | |||
+ | </ | ||
+ | ====== Backup policy ====== | ||
+ | |||
+ | * Backup of user files is **solely the responsibility of each user** | ||
+ | * [[https:// | ||
+ | * Backed up filesystems: | ||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | * Backups are performed on best effort basis | ||
+ | * Full backup run: ~3 days | ||
+ | * Backups are used for **disaster recovery only** | ||
+ | * Project manager can exclude $DATA filesystem from backup | ||
+ | * [[https:// | ||
====== Storage exercises ====== | ====== Storage exercises ====== | ||
Line 133: | Line 209: | ||
We will now measure the sequential performance of the different storage targets on VSC-3. | We will now measure the sequential performance of the different storage targets on VSC-3. | ||
- | < | + | |
- | < | + | |
< | < | ||
Line 190: | Line 265: | ||
We will now measure the storage performance for tiny 4kilobyte random writes. | We will now measure the storage performance for tiny 4kilobyte random writes. | ||
- | < | + | |
- | < | + | |
< | < | ||
Line 246: | Line 320: | ||
---- | ---- | ||
+ | |||