This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong. ====== Storage Technologies - Welcome ====== “Computers are like Old Testament gods; lots of rules and no mercy.” (Joseph Campbell) <!– “Simple things should be simple. Complicated things should be possible.” (Alan Kay) –!> <!– “Computer sciene is not about machines, in the same way that astronomy is not about telescopes. […] Science is not about tools, it is about how we use them and what we find out when we do.” (Michael R. Fellows) –!> ====== Storage Technologies - Contents ====== * Hardware Basics * Memory Hierarchy * Storage Devices * Flash-Memories * Harddisks * Magnetic Tapes * Multi Tiered Storages / Hierarchical Storage Systems * File Systems * Local * Network * Distributed ====== Storage Technologies - Hardware Basics ====== {{.:node.jpg}} ====== Storage Technologies - Hardware Basics ====== * CPU * Central Processing Unit * Controls Operation and performs data processing * Registers and Caches for fast data access * Main Memory * Stores data and programs * Typically volatile * I/O Modules * Move data between CPU/Memory and external devices ====== Storage Technologies - Hardware Basics (II) ====== * System Bus * Provides Communication among processor, memory and I/O modules * ISA, PCI, AGP, PCI-Express, … * External Devices * I/O Controllers (HBA / RAID) * Network Controllers (Ethernet, Fibre Channel, Infiniband) * Human Interface Devices (Keyboard, Mouse, Screen, etc.) ====== Storage Technologies - Direct Memory Access ====== * Traditionally copying of data from/to devices was done with cpu registers * Costs a lot of clock cycles that could be used for processing (up to 40 clock cycles per word) * DMA - Direct Memory Access * DMA-C is integrated in modern chipsets * Moves data from external devices to memory and vice versa * Informs the system when copying is done (interrupt) * Still some cycles needed for communication with the controller * Performance gets better the more data is transferred. Byte-wise DMA is a bad idea. ====== Memory Hierarchy (I) ====== {{.:memoryhierarchypyramid.png}} (Operating Systems, 7th Edition, W. Stallings, Chapter 1) ====== Memory Hierarchy (II) ====== * Inboard Memory * Registers * Caches * RAM * Outboard Storage * Flash-Drives * Disks * Blu-Ray / DVD / CDRom * Off-Line Storage * Magnetic Tapes ====== Memory Hierarchy (III) ====== * Why? * SRAM is expensive * Trade-off among speed, cost, size, and power consumption * Strategies * Caching\\ * Prefetching * Need data locality ====== Memory Hierarchy (IV) ====== * As one goes down the hierarchy… * Decreasing cost per bit * Increasing capacity * Decreasing frequency of access * Increasing access time ====== Memory Hierarchy (V) ====== * Memory Access Times on Intel Haswell Architecture 1 Clock Cycle = 250 picoseconds @ 4 Ghz (Movement at speed of light = 7.5 cm per 250 picoseconds) * Processor Registers * 1 Clock Cycle (250 ps) * Cache (L1) * Access Time ~4-5 Clock Cycles (~ 1 ns) * Cache (L2) * Access Time ~12 Clock Cycles (~ 3 ns) * Cache (L3) * Access Time ~36-58 Clock Cycles ( ~ 12 ns) * RAM * Access Time ~30-60 Clock Cycles + 50-100 ns ( ~ 70 - 120 ns) ====== Memory Access Times ====== {{.:haswell_access_times.png}} ====== Memory Access Times (II) ====== {{.:haswell_access_times_with_ram.png}} ====== Memory Hierarchy (VI) ====== * Storage Devices Typical Access Times * NVMe Flash Memory ~ 6 µs (@ 150’000 IOPS) * SAS Magnetical Disk ~ 2-3 ms (@ 350 IOPS) * Magnetical Tape milliseconds up to many seconds ====== Caching ====== {{.:memoryhierarchypyramid.png}} (Operating Systems, 7th Edition, W. Stallings, Chapter 1) ====== Storage Technologies - Cache Memory ====== * Acts as a buffer between 2 memory tiers * Modern CPUs utilize 3 levels of caches * Level 1 split into instruction and data cache. Separate for each core. * Level 2 data and instruction cache. Separate for each core. * Level 3 data and instruction cache. Shared among all cores on the die. * Benefits both throughput and latency * Different Caching Strategies for different purposes ====== Caching Strategies ====== * Cache Hit vs. Cache Miss * Cache Hit Ratio * Replacement Strategies / Associativity * Direct Mapped * N-Way Set Associative (But: Increased N means more potential locations need to be searched) * Fully Associative * Write Strategies * Write Back * Write Through * Prefetching / Readahead ====== Caching Problems ====== * Larger Caches have higher hit-rate but also higher latency * Multi Level Caches * Data may become incoherent particularly in multiprocessor systems –> Cache Coherency Protocol * Write Propagation * Transaction Serialization (Reads/Writes to a memory location must be seen in the same order by all processors) ====== Storage Technologies - Why Caching Works –> Locality of reference ====== * Memory References tend to cluster. * Program execution is sequential * Instructions tend to be localized to a few procedures * Iterative constructs consist of a small number of instructions * Processing of data structures, like arrays or sequences * Spatial locality * Execution involves a number of memory locations that are clustered * Acessing data locations sequentially * Spatial locality can be exploited by using large cache blocks and prefetching * Temporal locality * Execution involves memory locations that have been accessed recently * Temporal locality can be exploited by keeping recently used values in cache * Locality of reference enables us to do caching and prefetching * It also enables us to do branch prediction and other things, but this goes beyond the scope of this course ====== Caching - Example ====== <code c> int sizeX = 2000; int sizeY = 1000; int array[sizeY][sizeX]; // Fill the array with some data fill_buffer(&array); // Now run through the array and do something with the elements // This runs slow in C for (int x=0; x<sizeX, x++) { for (int y=0; y<sizeY; y++) { array[y][x] = x+2000*y; } } // This runs fast in C for (int y=0; y<sizeY, y++) { for (int x=0; x<sizeX; x++) { array[y][x] = x+2000*y; } } </code> ====== Memory Hierarchy - Recap ====== * Problem: A CPU waiting for data can’t do any work * Low Memory Access Times are crucial * Solution: Caching/Prefetching algorithms. * Works well with sequential data / local data access patterns * Won’t work with totally random access patterns (Locality) * As one goes down the hierarchy * Increasing access time * Decreasing throughput * Also known as “Von Neumann Bottleneck” ====== Storage Technologies (I) ====== * I/O Devices * Human Readable * Machine Readable * Communication * Differences in * Data Rate * Application * Complexity of Control ====== Storage Technologies (II) ====== * Across the different storage technologies, these relationships seem to hold: * Faster access time, greater cost per bit * Greater capacity, smaller cost per bit * Greater capacity, slower access speed ====== Storage Technologies - Device Characteristics (I) ====== * Performance * Access Time / Latency * Throughput * Capacity * Raw capacity * Storage density * Volatility / Differentiation * Dynamic Memory (periodic refreshes required) * Static Memory ====== Storage Technologies - Device Characteristics (II) ====== * Mutability * Read/Write * Read Only * Slow Write, Fast Read (e.g. SMR Discs) * Accessibility * Random Access * Sequential Access * Adressability * Location addressable * File addressable * Content addressable ====== Sequential I/O vs. Random I/O ====== * Sequential I/O * Writing / Reading contiguous large chunks of data (Chunk Size >= 10E6 Bytes) * Usually the fastest way to read data from storage devices * Not always easily applicable to a problem * Random I/O * Writing / Reading small chunks of data to / from random locations (Chunk Size <= 10E4 Byte) * Slowest way to read data from storage devices * Magnitude of the slow-down depends on the underlying hard- and software (e.g. Tape-Drives vs. Flash-Drives) ====== Hard-Drives Overview (I) ====== * Invented in the mid 50s by IBM * The first IBM drive stored 3.75 MB on a stack of 50 discs * Had the size of two refrigerators (1.9 m³) * Became cheap / mainstream in the late 80s * Today one 3.5" drive can hold up to 14TB of data * Density ~ 1.5 Tbit / in² * With future technologies even higer data densities may become possible * Filling disks with helium * Shingled Magnetic Recording * Heat Assisted Magnetic Recording * Interface * Serial Attached SCSI (SAS) * SATA * NVMe ====== Hard-Drives Overview (II) ====== {{.:harddisk_description_wikipedia.png}} * Video of operating harddisk on wikipedia: https:%%//%%upload.wikimedia.org/wikipedia/commons/c/c5/HardDisk1.ogv ====== Hard-Drives Characteristics ====== * Seek Time * The time it takes to move the head to the correct track * Rotational Delay * The time it takes until the platter rotates to the correct position ( ~2ms @ 15’000 rpm) * r… rotation speed in revolutions per second * $T_{rdelay} = 1/2r$ * Transfer time calculation * T… Transfer Time * b… Number of bytes to be transferred * N… Number of bytes per track * r… rotation speed in revolutions per second * $T = b/rN$ * Average Access Time * $T_{access} = T_{seek} + T_{rdelay} + T_{transfer}$ ====== Hard-Drives Sequential Access Example ====== * Do not rely on average values! * Given a Harddisk with * Rotational speed of 7’500 rpm * 512 byte sectors * 500 sectors per track * We read a file that is stored on 5 adjacent tracks in 2500 sectors (1.28 MBytes) * Average seek… 4ms * Rotational delay… 4ms * Read 500 sectors… 8ms * We need to do this 5 times (1 for each track), but because of sequential organization we can skip the seek time for the consecutive tracks * $T_{total} = 16 + (4 * 12) = 64 ms$ ====== Hard-Drives Random Access Example ====== * Given a Harddisk with * Rotational speed of 7’500 rpm * 512 byte sectors * 500 sectors per track * We read a file that is distributed randomly over 2500 sectors of the disk * Average seek… 4ms * Rotational delay… 4ms * Read 1 sector… 0.016ms * We need to do this 2500 times with a seek after each sector * $T_{total} = 2'500 * 8.016 = 20'040 ms = 20.04s$ * Slowdown of nearly 3 orders of magnitude with the same average values * Do not rely on average values! ====== Quo vadis Hard-Drives ====== * Current/Future Trends * Helium filled drives * Shingled Magnetic Recording * Heat assisted recording ====== (NVMe) Solid State Drives (I) ====== * NAND Flash Memory * Lower cost compared to DRAM * No refresh cycles / external PSU needed to retain the data compared to DRAM * Uses less space compared to memory based on NOR gates * No random access (Cells are connected in series to sectors) * Key components * Controller * Memory * New Interfaces * SATA Express * NVM Express ====== (NVMe) Solid State Drives (II) ====== * NAND Flash Memory * Single Level Cell (SLC) * Multi Level Cell (MLC) * SLC * 1 bit per cell * ~10 times longer life than MLC * Lower latency than MLC * MLC * multiple bits per cell * Lower production cost ====== (NVMe) Solid State Drives - Memory (I) ====== * NAND Flash Memory * Is composed of one or more chips * Chips are segmented into planes * Planes are segmented into thousands (e.g. 2048) of blocks * And 1 or 2 registers of the page size for buffering * A Block usually contains 64 to 128 pages * Each page has a data part (few KBytes) * and a small metadata area (e.g. 128 bytes) for Error Correcting Code * Exact specification varies across different memory packages ====== (NVMe) Solid State Drives - Memory (II) ====== * Read * Is performed in units of pages * 1 page takes approx. 10 µs (SLC) up to 100 µs (MLC) to read * Write * Is performed in units of pages * Some controllers support sub-page operations * Pages in one block must be written sequentially * A page write takes approximately 100 µs (SLC) up to 900µs (MLC) * Block must be erased before being reprogrammed ====== (NVMe) Solid State Drives - Memory (III) ====== * Erase * Must be performed in block granularity (hence the term erase-block) * Can take up to 5 ms * Limited number of erase cycles * SLC: 100’000 * MLC: 10’000 * Some flash memory is reserved to replace bad blocks * Controller takes care of wear leveling ====== (NVMe) Solid State Drives - Controller (I) ====== * Higher complexity of access compared to hard-disks * High performance SSDs consists of hundreds of dies with parallel communication paths * Data cells may wear out and become bad * Erase size != Page size * We need a controller to handle this: * Does the initial setup * Format the memory * Mark bad blocks * Moves data and picks blocks so that single cells don’t wear out * Parallelizes read/write/erase operations on many dies * Adresses in Blocks/Pages * Flash Transition Layer and Logical to Physical Mapping * Garbage Collection ====== (NVMe) Solid State Drives - Speed (I) ====== * Besides the additional complexity, SSDs are much faster than traditional disks * Read Latency is in the order of few microseconds (We had milliseconds with HDDs) * No need to move the head around * But because of the arrangement in blocks and pages, sequential access is still faster * More IOPS because of the lower latency * Needs more CPU utilization to get full throughput * Needs some pressure (multiple calls) to fully parallelize read/write calls * This is also called ‘Queue Depth’ ====== (NVMe) Solid State Drives - IOPS vs. Throughput ====== * Traditional discs have been measured in terms of throughput * SSDs are sometimes measured in terms of IOPS (Input- Output- Operations per Second) * The following equation holds (if there is no controller overhead like erasing, garbage collection, etc.): * $Throughput = IOPS * Blocksize$ * Where blocksize can be chosen freely * So if we know the blocksize that was used in benchmarking… * We can calculate IOPS from throughput and vice versa * (If we assume that the disk was empty at the beginning of the benchmark and no additional controller overhead was involved) * We now even have a base to calculate which blocksize is at least necessary to get full write throughput ====== (NVMe) Solid State Drives - Speed (II) ====== * Given an Intel DC P3700 SSD with a capacity of 2 TB, specification says: * Sequential Read 450’000 IOPS with 2800MB/s of max throughput * $2'800'000'000 Bytes = 450'000 * x$ * $x = 6222 Bytes ~ 6KByte$ * So a blocksize of 8 KByte is a good starting point to try to get full throughput * Random Reads with a page size of 8KByte should work well on this device and we should get the full throughput of 2800MB/s * How would a traditional HDD perform under these conditions? ====== (NVMe) Comparison SSD vs. HDD ====== * In the slide before, we proposed that on our SSD with a block size of 8 KByte will lead to a throughput of 2800MB/s * Let’s try this with a HDD * Rotational speed of 7’500 rpm * 512 byte sectors * 500 sectors per track ====== (NVMe) Comparison SSD vs. HDD ====== * We read a file that is distributed randomly over 2500 sectors of the disk in blocks of 8KByte (Blocks of 16 sectors) * Average seek… 4ms * Rotational delay… 4ms * Read 16 sectors… 0.256ms * We need to do this 156.25 times with a seek after every block of 16 sectors * $T_{total} = 156.25 * 8.256 = 1290 ms = 1.290 s$ * We transferred 156.25 * 8 KBytes = 1250 KBytes in 1.290 s * This equals ~ 1 MByte per second (SSD 2800 MByte per second) * With the same block size our benchmark will run ~2800 times faster on SSD than on HDD * When utilised in sequential form, HDDs are still 20-30 times slower than high performance SSDs ====== (NVMe) Solid State Drives - Speed (II) ====== * $Transfertime = T_{controller} + T_{transfer}$ ====== Quo vadis Solid State Drives ====== * Current / Future Trends * Flash Memory Market surpassed DRAM Market in 2012 * Density of Flash Memory surpassed Hard Drive density in 2016 * In Q4 2016 45 million SSDs with a total capacity of 16 Exabyte were delivered to customers * Market for HDDs is significantly bigger than for SSDs * New memory technologies (e.g. Intel/Micron 3DXPoint) * Intel Optane Memories * Things to consider * Will SSDs replace all HDDs in the future? * All Flash SDS * Storage as Memory * Everything except archive * What use cases remain for HDDs? ====== Magnetic Tapes (I) ====== * Have been in use for data storage since the 50’s * Main storage medium in some early computers * Capacity: * 1950’s: ~ 1 MByte * 1980’s: ~ 100 MByte - 1 GByte * 1990’s: ~ 10 - 100 GByte * 2000’s: ~ 100 GByte - 1 TByte * Now: >10 TByte * Future: Going up to 200 TByte per Tape seems possible * (https:%%//%%www.sony.net/SonyInfo/News/Press/201404/14-044E/index.html) * Real tape capacity is usually 1/2 of the given capacity * A compression ratio of 2:1 is quietly assumed ====== Magnetic Tapes - Characteristics (I) ====== * Tape width * Half inch has been the most common width * Recording Method * Linear * Linear Serpentine * Scanning (writing across the width of the tape) * Helical Scan (Short, dense, diagonal tracks) * Block Layout * Data is written in blocks, each block is a single operation ====== Magnetic Tapes - Characteristics (II) ====== * Access * Sequential Access * Index database or Tape Marks * Compression * Data is usually compressed when written on tape * Encryption * Data encryption is possible and standardised * Encryption needs to be done after compression * Why? ====== Magnetic Tapes - Characteristics (II) ====== * Access * Sequential Access * Index database or Tape Marks * Compression * Data is usually compressed when written on tape * Encryption * Data encryption is possible and standardised * Encryption needs to be done after compression * Encrypted data should look like random noise * If your encrypted data is compressible then your encryption is flawed ====== Magnetic Tapes - LTO ====== * Linear Tape Open * Open standard to store data on magnetic tapes * Developed in the 90’s * Eigth Generation LTO was released in 2017 * LTO-8 State of the art * 12 TByte raw capacity (uncompressed) * 360 MByte/s max uncompressed speed * Compression ratio 2.5:1 * Supports encryption * Supports WORM (Write once read many) ====== Magnetic Tapes - No random I/O ====== * The loading and winding of the tape takes a long time (up to many seconds) * Tape moves at a constant speed * Modern tape machines can move at 1/2 or 1/4 speed * Shoe shining happens when there is not enough data to write * Wears out the tape * Doing random reads wears out the tape * Expected durability of a tape is about 10^4 end-to-end passes ====== Memory Hierarchy - cont’d ====== * With the given storage technologies (Flash, HDD and Tape) we can refine our Memory hierarchy * Using Flash Memories for Caching and Metadata * Using HDDs as online data storage * Using Tapes as offline data storage * With the concept of locality and the right applications this will speed up our storage stack * Recently random accessed files and Metadata that fits on the SSDs will be stored on the SSDs * Recently sequential acessed files will be moved to the HDDs (They are fast enough when used in a sequential way) * Rarely acessed files go to the tape machine * We could even link these files transparently into the file system, so the user doesn’t have to know anything about where his data is located ====== Multi Tiered Storage Systems ====== * Different Tiers of Storage e.g. * Tier 1: SSDs * Tier 2: SATA HDDs * Tier 3: Tape Storage * But there’s no free meal * We need to store data about where our data is stored (More on metadata later on) –> Memory Overhead * If a user accesses a file, we need to check where the file is –> Processing Overhead * In the worst case we have to copy it from tape to the disk * which means loading the right tape and winding to the correct position * which also means that the user might think, that the storage is very slow today ====== Gaining speed and redundancy with RAID ====== * RAID - Redundant Array of Independent (Inexpensive) Discs * Distribute data across several drives * Add parity calculation or mirroring for redundancy * Different Techniques / RAID Levels available * RAID 0, 1, 5, 6, 10, 01, … * Several Discs in a RAID are called a RAID array ====== RAID - Striping and Mirroring ====== * RAID Level 0 * Data is striped across drives without redundancy * Failure of one disc leads to loss of all data in the array * Speedup is proportional to the number of discs * RAID Level 1 * Data is mirrored across discs (usually 2, but more are possible) * Failure of a disc involves no loss of data * Read calls can be parallellized to multiple discs * No speedup for write calls ====== RAID - Parity ====== * RAID Level 5 * Data is striped across drives, parity is calculated (XOR) and stored * Block level striping * Failure of one disc can be compensated * Failure of more than one discs leads to data loss * Considered obsolete nowadays, because of long rebuild times * Use RAID-6 * RAID Level 6 * Data is striped across drives, parity is calculated and stored * 2 different syndromes are computed to allow the loss of up to two drives * Additional computation needed (no simple XOR for the second syndrome) ====== RAID - Other Levels ====== * “Hybrid”-RAID * Mirroring between SSD and HDD (special controller needed) * Nested-RAID * RAID-10 * Stripe of Mirrors * RAID-01 * Mirror of Stripes * Most probably RAID-10 is the better choice than RAID-01 * Improved fault tolerance ====== But where are the levels 2 - 4 ? ====== * RAID2 - RAID4 are obsolete * RAID2 * Bit level striping with dedicated parity disc * RAID3 * Byte level striping with dedicated parity disc * RAID4 * Block level striping with dedicated parity disc ====== File Management ====== * Abstraction of the secondary storage * After all we don’t want to cope with the internals of storage devices * Abstraction to Files / Directories –> File System * Files * Long-term existence * Sharable between processes * Structure * Files can have internal structure (e.g. databases) * Operations * Create * Delete * Open * Close * Read * Write ====== File Management - Inodes (I) ====== * Besides the file content, filesystems rely on data structures about the files * Also called Metadata * Inodes store information about files * Type of File (File, Directory, etc) * Ownership * Access Mode / Permissions * Timestamps (Creation, Access, Modification) * Last state change of the inode itself (status, ctime) * Size of the file * Link-Counter * One or more references to the actual data blocks ====== File Management - Inodes (II) ====== * Most filesystems use a fixed amount of inodes * Ext2 filesystem can address up to 12 blocks per inode * If more blocks are needed for a file a new inode is allocated (indirect block) * Ext2 allows up to triple nested indirect blocks which leads to a maximum size of 16 GiB to 4 TiB depending on block-size * Appending to files and writing new files is not possible when inodes run out * Depending on the amount of files/directories a filesystem can use up to 10% of its capacity for meta information * Show Inode Number with ‘ls -i’ * Show content with stat ====== File Management - Inodes (III) ====== <code> * sreinwal@rs ~/Work/vboxshared/sreinwal/pandoc/vsc-markdown/parallell-io/02_storage_technologies $ stat storage_technologies.md * File: storage_technologies.md * Size: 30837 Blocks: 64 IO Block: 4096 regular file * Device: fd04h/64772d Inode: 25373850 Links: 1 * Access: (0644/-rw-r--r--) Uid: ( 1001/sreinwal) Gid: ( 1001/sreinwal) * Access: 2017-11-28 14:25:11.191823770 +0100 * Modify: 2017-11-28 14:23:53.482827520 +0100 * Change: 2017-11-28 14:25:20.416823325 +0100 * Birth: - </code> ====== File System - Organization (I) ====== {{.:fs1.png}} (Operating Systems 7th Edition, W. Stallings, Chapter 12) ====== File System - Organization (II) ====== * Device Drivers / Hardware Layer * Dealing with I/O operations on the devices * Usually considered part of the operating system * Basic File System / Physical IO Layer * Deals with blocks of data, that are exchanged with device drivers * Manages placement of blocks and the buffering of these blocks in main memory ====== File System - Organization (III) ====== * Basic I/O supervisor / Basic I/O Layer * Responsible for file I/O and its termination * Control structures are maintained to deal with device I/O, scheduling and file status * Disk-Scheduling to optimize performance * Logical I/O Layer * Provides general purpose file I/O * Maintains basic data about files * Enables users/applications to access files ====== File System - Organization (IV) ====== {{.:fs2.png}} (Operating Systems 7th Edition, W. Stallings, Chapter 12) ====== Unix File Management ====== * 6 different types of files * Regular * Contains data. Simple stream of bytes. * Directory * Contains list of file names plus pointers to their index nodes * Special * Map physical devices to files (e.g. /dev/sda for the first SAS/SATA disc) * Named pipes * Buffers received data * Enables interprocess communication * FIFO (First-In-First-Out) * Links * Alternative filenames for existing files * Symbolic Links * Basically an ordinary file that contains the name of the file it is linked to ====== Linux Virtual File System ====== * Different File Systems under Linux * ext2, ext3, ext4, reiserfs, xfs, nfs, … * VFS provides a virtual file system * Presents a single, uniform interface to user processes * Mapping between VFS and underlying file system ====== Storage Networks - NAS vs. SAN ====== * NAS - Network Attached Storage * Centralized File Storage * Exporting File-Systems for clients to mount * Connected via Network (Ethernet, Infiniband, …) * File System Layers of the server are used * User/Access Management at the server * SAN - Storage Area Network * Centralized Device Array * Exports Block devices * Connected via Fibrechannel * ISCSI Protocol, NVMoF Protocol * Client is responsible for handling file systems on the block device * No User/Access Management at block level ====== Distributed File Systems - NFSv4 ====== * NFSv4 * Network File System Version 4 * Client Server Architecture * Provides a standardized view of its local filesystem * Allowing a heterogenous collection of processes to share a common filesystem * Allows sharing of all local filesystems as well as other network filesystems * VFS layer comes in handy here * NFS4 uses Remote File Service / Remote Access Model * File stays on the server * Locking Mechanisms (Stateful Model) * NFSv2 and NFSv3 were stateless which led to some problems (Separate Locking daemon needed) * Opposed to the Upload/Download Model * Client downloads file, does operations locally, reuploads the file ====== Client Side Caching - Asynchronous I/O ====== * Synchronous or Blocking I/O * Process issues an I/O call to the operating system and waits for the operation to complete * Leads to intervals where the CPU is completely idle waiting for the operation to finish * Asynchronous or Non-Blocking I/O * As we have seen, I/O Operations can be very slow * Client calls the operating systems for a read/write * Continues processing * Data is read/written in the background * Operating system informs the process when the file is written * Client side caching in NFS * async mount option * data is cached on the client ====== Server Side Caching - NFSv4 ====== * Server Side Caching * Server replies to requests before they have been committed to stable storage * Usually less error prone than client side caching, because servers are usually in a more controlled environment than clients * e.g. Uninterruptible Power Supply, BBU Units on RAID controllers, etc. * Problems arise * Server crashes before data is transferred from client to server * Client crashes before it transfers data to server * Network connection breaks * Power outages * etc… ====== Parallell File Systems - BeeGFS ====== * FhGFS / BeeGFS Parallell File System * Distributes data and metadata across serveral targets * Stripes data across several targets * Huge speedup compared to single server appliances * Scalability and Flexibility were key aspects for the design ====== BeeGFS - Management Server ====== * Management Server * Exactly 1 MS per filesystem * Keeps track of metadata and storage targets * Keeps track of connected clients * Tags targets with labels * Normal * Low * Critical * Not involved in filesystem operations ====== BeeGFS - Metadata Server ====== * Holds the Metadata information of the file system * Should be stored on fast storage devices (e.g. Flash Memories) * Metadata accesses are mostly tiny random accesses * Ext4 seems to be the fastest filesystem to store beegfs metadata * Additional servers can be added on demand * Each server holds exactly one metadata target * TargetChooser * Chooses targets according to the configuration when a user writes a file * Holds metadata information * File / Directory information * Creation date * Striping pattern * Affected Storage Targets * Permissions * ACLs * Extended attributes * … ====== BeeGFS - Metadata Server ====== * Entry point of the file system ‘/’ is located on MDS #1 * Defined entry point in the directory tree * Additional directories are assigned random to other MDS * Leads to effective load balancing on the metadataservers * As long as there are significantly more directories than metadata targets * Client walks down the directory tree to find the responsible MDS for a directory * Handling Meta-Data requests is creating some load on the cpu cores * For a small number of requests higher clock speeds lead to less latency * For a high number of requests multiple CPU Cores are needed * Metadata handling will create processes which are waiting for data * CPU overcommitting could increase performance ====== BeeGFS - Object Storage Server ====== * Holds the file contents on Object Storage Targets (OSTs) * Underlying devices / filesystem can be chosen freely * As long as the file system is POSIX compliant * File contents can be striped over multiple OSTs * One server can handle multiple OSTs * A typical OST consists of 6 - 12 drives running in RAID6 * Number of threads influence the number of requests that can be put on disk * Higher number of threads will lead to randomization of sequential data * We have to live with that to a certain extent in multi user environments * Chunksize sets the amount of data stored on a OST per stripe * OSTs can be added on demand. But a rebalancing of the file system might be needed to gain full performance ====== BeeGFS - Clients ====== * Clients can be added on demand * Thousands of clients is no problem as long as there are enough MDTs and OSTs available * tuneFileCacheType - Client caching is supported * Buffered Mode * Uses a pool of static buffers for write-back and read-ahead * Caches up to a few hundred kilobytes of a file * Default Mode * Native Mode * Uses the linux kernel page cache * May cache up to multiple gigabytes of data (depending on the available RAM) * Experimental / Work in progress ====== Theres much more ====== * Different Storage Devices * Optical * Laserdisc / CD-Rom / DVD / Blu-Ray * Magnetic * Floppy / ZIP Drives * Magneto-Optical * Minidisc (Sony) * Different Parallell File Systems * GPFS * GlusterFS * Lustre * Different use-cases * Storage Area Network * Different architectures * HP - The Machine ====== Storage Technologies - Bibliography ====== * Understanding Intrinsic Characteristics and System Implications of Flash Memory Based Solid State Drives, Cheng et al., 2009 * Operating Systems - Internals and Design Principles 7th Edition, Stallings, 2012 * Distributed Systems - Principles and Paradigms 2nd Edition, Tanenbaum et al., 2007 ====== Storage Technologies - The End ====== * Remember * Random I/O is magnitudes slower than sequential I/O * Sequential I/O can even be done in parallell from multiple nodes to further improve the throughput * Highly parallellized Random calls will result in degraded storage performance for ALL users and can even lead to an unresponsive storage. * Don’t do random I/O on storage devices Thank you for your attention pandoc/parallel-io/02_storage_technologies/storage_technologies.txt Last modified: 2020/10/20 09:13by pandoc