Next revision | Previous revision |
pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer [2017/10/18 11:42] – Pandoc Auto-commit pandoc | pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer [2019/10/08 09:26] (current) – Pandoc Auto-commit pandoc |
---|
====== VSC-3 – supercomputer ====== | ====== VSC-3 – supercomputer ====== |
| |
* Article written by Claudia Blaas-Schenner (VSC Team) <html><br></html>(last update 2017-10-19 by cb). | * Article written by Claudia Blaas-Schenner (VSC Team) <html><br></html>(last update 2019-10-08 by cb). |
| |
**OUTLINE:** | **OUTLINE:** |
* VSC-3 compute nodes | * VSC-3 compute nodes |
* VSC-3 node-interconnect | * VSC-3 node-interconnect |
* VSC-3 ping-pong – intra-node vs. inter-node | * VSC-3 ping-pong – intra-node vs. inter-node |
| |
| |
* **VSC is primarily devoted to research.** | * **VSC is primarily devoted to research.** |
* **Who can use VSC?** Scientific personnel of the partner universities, see: http://vsc.ac.at/access <html><nobr></html>VSC is open to users<html></nobr></html> from other academic and research institutions. | * **Who can use VSC?** Scientific personnel of the partner universities, see: http://vsc.ac.at/access <html><nobr></html>VSC is open to users<html></nobr></html> from other academic and research institutions. |
* **Projects** (test, funded, ...): Access to VSC is granted on the basis of **peer-reviewed projects**. | * **Projects** (test, funded, …): Access to VSC is granted on the basis of **peer-reviewed projects**. |
* **Project manager** (= usually your supervisor): Project application, extensions, creates user accounts, ... | * **Project manager** (= usually your supervisor): Project application, extensions, creates user accounts, … |
* **Publications**: Please [[http://typo3.vsc.ac.at/access/acknowledgments/|acknowledge VSC]] and [[http://typo3.vsc.ac.at/access/publications-database/|add publications]] <html><font color=#cc3300></html>$~~$➠$~~$<html></font></html> visible on [[http://vsc.ac.at/publications|VSC homepage]] ! | * **Publications**: Please [[http://typo3.vsc.ac.at/access/acknowledgments/|acknowledge VSC]] and [[http://typo3.vsc.ac.at/access/publications-database/|add publications]] <html><font color=#cc3300></html>$~~$➠$~~$<html></font></html> visible on [[http://vsc.ac.at/publications|VSC homepage]] ! |
| |
^VSC links: ^Information provided: ^ | |
|<html><font color=#cc3300></html>➠$~~$<html></font></html>**http://vsc.ac.at** |VSC homepage (general info) | | |
|<html><font color=#cc3300></html>➠$~~$<html></font></html>**https://service.vsc.ac.at** |VSC service website (application) | | |
|<html><font color=#cc3300></html>➠$~~$<html></font></html>**https://wiki.vsc.ac.at** |VSC user documentation | | |
|<html><font color=#cc3300></html>➠$~~$<html></font></html>{{pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:contact_vsc-red_margin.png?150}} |VSC user support $~$&$~$ contact | | |
| |
| ^VSC links: ^Information provided: ^ |
| |<html><font color=#cc3300></html>➠$~~$<html></font></html>**http://vsc.ac.at** |VSC homepage (general info) | |
| |<html><font color=#cc3300></html>➠$~~$<html></font></html>**https://service.vsc.ac.at** |VSC service website (application) | |
| |<html><font color=#cc3300></html>➠$~~$<html></font></html>**https://wiki.vsc.ac.at** |VSC user documentation | |
| |<html><font color=#cc3300></html>➠$~~$<html></font></html>{{:pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:contact_vsc-red_margin.png?150}} |VSC user support $~$&$~$ contact | |
| |
* **VSC Training Courses:** <html><br><font color=#cc3300></html>➠$~~$<html></font></html>**http://vsc.ac.at/training** <html><br></html>**VSC course slides:** <html><br><font color=#cc3300></html>➠$~~$➠$~~$➠$~~$<html></font></html>**[[https://wiki.vsc.ac.at/doku.php?id=pandoc:introduction-to-vsc:01_supercomputers_for_beginners:00_linux|VSC-Linux]]** <html><br><font color=#cc3300></html>➠$~~$➠$~~$➠$~~$<html></font></html>**[[https://wiki.vsc.ac.at/doku.php?id=pandoc:introduction-to-vsc:01_supercomputers_for_beginners:00_intro|VSC-Intro]]** | |
| * **VSC Training Courses:** <html><br></html><html><font color=#cc3300></html>➠$~~$<html></font></html>**http://vsc.ac.at/training** <html><br></html>**VSC course slides:** <html><br></html><html><font color=#cc3300></html>➠$~~$➠$~~$➠$~~$<html></font></html>**[[https://wiki.vsc.ac.at/doku.php?id=pandoc:introduction-to-vsc:01_supercomputers_for_beginners:00_linux|VSC-Linux]]** <html><br></html><html><font color=#cc3300></html>➠$~~$➠$~~$➠$~~$<html></font></html>**[[https://wiki.vsc.ac.at/doku.php?id=pandoc:introduction-to-vsc:01_supercomputers_for_beginners:00_intro|VSC-Intro]]** |
| |
| |
| |
* **What is a supercomputer ?** | * **What is a supercomputer ?** |
* A supercomputer is a computer with a high level of computing performance compared to a general-purpose computer. Performance of a supercomputer is measured in floating-point operations per second (FLOPS)... [from Wikipedia] <html><br><br></html> | * A supercomputer is a computer with a high level of computing performance compared to a general-purpose computer. Performance of a supercomputer is measured in floating-point operations per second (FLOPS)… [from Wikipedia] <html><br></html><html><br></html> |
* **A supercomputer is listed in the [[https://www.top500.org|TOP500]]** | * **A supercomputer is listed in the [[https://www.top500.org|TOP500]]** |
| |
|VSC-2 (2011) | 135 TFlop/s| 56 (06/2011)| 71 (06/2011)| 8 PFlop/s #1 (06/2011)| | |VSC-2 (2011) | 135 TFlop/s| 56 (06/2011)| 71 (06/2011)| 8 PFlop/s #1 (06/2011)| |
|[[https://www.top500.org/system/178471|VSC-3 (2014)]]| 596 TFlop/s| 85 (11/2014)| 86 (11/2014)| 33 PFlop/s #1 (11/2014)| | |[[https://www.top500.org/system/178471|VSC-3 (2014)]]| 596 TFlop/s| 85 (11/2014)| 86 (11/2014)| 33 PFlop/s #1 (11/2014)| |
|VSC-3 (.........) | 596 TFlop/s| 329 (06/2017)| 198 (06/2017)| 93 PFlop/s #1 (06/2017)| | |VSC-3 (………) | 596 TFlop/s| 460 (11/2017)| 175 (11/2017)| 93 PFlop/s #1 (11/2017)| |
| |[[https://www.top500.org/system/179697|VSC-4 (2019)]]| 2.7 PFlop/s| 82 (06/2019)| ——–| 148 PFlop/s #1 (06/2019)| |
| |
| |
====== VSC-3 – what does it look like ? ====== | ====== VSC-3 – what does it look like ? ====== |
| |
{{pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:vsc3.png}} | {{:pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:vsc3.png}} |
| |
| |
====== VSC-3 – what does it look like ? – inside ====== | ====== VSC-3 – what does it look like ? – inside ====== |
| |
{{pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:vsc3-inside.png}} | {{:pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:vsc3-inside.png}} |
| |
| |
====== VSC-3 – components of a supercomputer ====== | ====== VSC-3 – components of a supercomputer ====== |
| |
{{pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:vsc3-schematic.png}} | {{:pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:vsc3-schematic.png}} |
| |
| |
====== Parallel hardware architectures ====== | ====== Parallel hardware architectures ====== |
| |
* **how to connect cores (processing units) ?** <html><br></html>{{pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:hw-cores_margin.png?150}} | |
| |
{{pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:hw-architectures.png}} | * **how to connect cores (processing units) ?** <html><br></html>{{:pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:hw-cores_margin.png?150}} |
| |
| {{:pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:hw-architectures.png}} |
| |
| |
| |
| |
{{pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:vsc3-node.png}} | {{:pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:vsc3-node.png}} |
| |
| |
====== VSC-3 node-interconnect ====== | ====== VSC-3 node-interconnect ====== |
| |
**IB fabric = dual rail Intel QDR-80 = 3-level fat-tree** (BF: 2:1 / 4:1) -- schematic figure / numbers only <html><br><font color=#ffffff></html>**IB fabric = dual rail Intel**<html></font></html> (blocking -- BF: down- : up-links -- might introduce an additional latency) | |
| |
{{pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:vsc3-fabric-3.png}} | |
| |
| |
| |
| |
| **IB fabric = dual rail Intel QDR-80 = 3-level fat-tree** (BF: 2:1 / 4:1) – schematic figure / numbers only <html><br></html><html><font color=#ffffff></html>**IB fabric = dual rail Intel **<html></font></html> (blocking – BF: down- : up-links – might introduce an additional latency) |
| |
| {{:pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:vsc3-fabric-3.png}} |
| |
| |
---- | ---- |
| |
====== VSC-3 ping-pong – intra-node vs. inter-node ====== | ====== VSC-3 ping-pong – intra-node vs. inter-node ====== |
| |
* **1 node** $~$ = $~$ 2 sockets with 8 cores per socket $~$ + $~$ **2 HCAs** | * **1 node** $~$ = $~$ 2 sockets with 8 cores per socket $~$ + $~$ **2 HCAs** |
* **ping-pong benchmark** $~$ = $~$ module load $~$ intel/16.0.3 $~$ intel-mpi/5.1.3 $~$ | $~$ openmpi/1.10.2 $~$ (1 HCA) | * **ping-pong benchmark** $~$ = $~$ module load $~$ intel/16.0.3 $~$ intel-mpi/5.1.3 $~$ | $~$ openmpi/1.10.2 $~$ (1 HCA) |
| |
* **MPI latency & bandwidth (plus typical values for comparison):** | |
| |
| <HTML><ul></HTML> |
| <HTML><li></HTML><HTML><p></HTML>**MPI latency & bandwidth (plus typical values for comparison):**<HTML></p></HTML> |
^VSC-3: ^ latency [μs] ^ ^ typical values for: ^ latency^ bandwidth^ | ^VSC-3: ^ latency [μs] ^ ^ typical values for: ^ latency^ bandwidth^ |
|<html><font color=#0000ff></html>intra-socket<html></font></html> | <html><font color=#0000ff></html>0.3 μs<html></font></html> | | <html><font color=#696969></html>L1 cache<html></font></html> | <html><font color=#696969></html>1–2 ns<html></font></html>| <html><font color=#696969></html>100 GB/s<html></font></html>| | |<html><font color=#0000ff></html>intra-socket<html></font></html> | <html><font color=#0000ff></html>0.3 μs<html></font></html> | | <html><font color=#696969></html>L1 cache<html></font></html> | <html><font color=#696969></html>1–2 ns<html></font></html>| <html><font color=#696969></html>100 GB/s<html></font></html>| |
|<html><font color=#ff00ff></html>IB -2- leaf<html></font></html> | <html><font color=#ff00ff></html>1.8 μs<html></font></html> | | <html><font color=#696969></html>HPC networks<html></font></html> | | |<html><font color=#ff00ff></html>IB -2- leaf<html></font></html> | <html><font color=#ff00ff></html>1.8 μs<html></font></html> | | <html><font color=#696969></html>HPC networks<html></font></html> | |
|<html><font color=#ffa500></html>IB -3- spine<html></font></html> | <html><font color=#ffa500></html>2.3 μs<html></font></html> | | <html><font color=#696969></html>(per node / 2 HCAs)<html></font></html> | <html><font color=#696969></html>1–10 μs<html></font></html>| <html><font color=#696969></html>1–8 GB/s<html></font></html>| | |<html><font color=#ffa500></html>IB -3- spine<html></font></html> | <html><font color=#ffa500></html>2.3 μs<html></font></html> | | <html><font color=#696969></html>(per node / 2 HCAs)<html></font></html> | <html><font color=#696969></html>1–10 μs<html></font></html>| <html><font color=#696969></html>1–8 GB/s<html></font></html>| |
| <HTML></li></HTML><HTML></ul></HTML> |
| |
| |
| |
| {{:pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:ping-pong-bandwidth.png}} |
| |
| |
{{pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:ping-pong-bandwidth.png}} | |
| |
| {{:pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:ping-pong-bandwidth-log.png}} |
| |
{{pandoc:introduction-to-vsc:01_supercomputers_for_beginners:vsc3_supercomputer:ping-pong-bandwidth-log.png}} | |
| |
| |
---- | ---- |
| |