Differences
This shows you the differences between two versions of the page.
Next revision | Previous revision Next revisionBoth sides next revision | ||
doku:slurm_multisite_admin [2022/12/22 18:16] – created fsattari | doku:slurm_multisite_admin [2023/06/23 13:35] – [Federated Slurm] fsattari | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | slurm_multisite_admin | + | ====== Overview of Multi-Clustering ====== |
+ | * Multi-Clustering allows running several slurm clusters from the same control node. | ||
+ | * In this case, different slurmctld daemons will be running on the same machine, and the system users can target commands to any (or all) of the clusters. | ||
+ | * From an architecture standpoint, each cluster still relies on its own Slurm controller. | ||
+ | * From an end-user standpoint, they will be able to submit jobs that will not only be able to run on their local Slurm cluster but also on extra resources provided by the additional Slurm clusters. | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | ===== Multi-Cluster Slurm Operation ===== | ||
+ | |||
+ | * Multi-cluster slurm operation is designed to offer the ability to submit a job that can be run on any of the available clusters instead of the local one and receive status from those remote clusters. | ||
+ | |||
+ | < | ||
+ | [...]# sacctmgr list cluster | ||
+ | |||
+ | | ||
+ | ------------------------- ------------ ----- --------- ------- ------------- --------- ------- ------------- --------- ----------- ------ | ||
+ | | ||
+ | | ||
+ | |||
+ | </ | ||
+ | |||
+ | {{: | ||
+ | ==== Node allocation policy ==== | ||
+ | |||
+ | * To enable the multi-cluster functionality the use of SlurmDBD and MUNGE or authentication keys is required. | ||
+ | * When sbatch, salloc or srun is invoked with a cluster list, Slurm submits the job to the cluster that offers the earliest start time considering its queue of pending and running jobs | ||
+ | * BUT Slurm will make no subsequent effort to migrate the job to a different cluster whose resources become available when running jobs finish before their scheduled end times. | ||
+ | * Originally, job IDs are not unique across multiple clusters. | ||
+ | |||
+ | {{: | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | ==== Basic Multi-Cluster Slurm commands ==== | ||
+ | |||
+ | Slurm client commands offer the "-M, --clusters=" | ||
+ | The third parameter after " | ||
+ | |||
+ | * '' | ||
+ | |||
+ | < | ||
+ | [...]# sinfo -M vsc4,vsc5 | ||
+ | CLUSTER: vsc4 | ||
+ | PARTITION | ||
+ | jupyter | ||
+ | skylake_0768 | ||
+ | . | ||
+ | . | ||
+ | . | ||
+ | |||
+ | |||
+ | CLUSTER: vsc5 | ||
+ | PARTITION | ||
+ | cascadelake_0384 | ||
+ | zen3_2048 | ||
+ | . | ||
+ | . | ||
+ | . | ||
+ | </ | ||
+ | * '' | ||
+ | |||
+ | < | ||
+ | [...]# squeue -M vsc4, | ||
+ | CLUSTER: vsc4 | ||
+ | | ||
+ | | ||
+ | | ||
+ | . | ||
+ | . | ||
+ | . | ||
+ | |||
+ | |||
+ | CLUSTER: vsc5 | ||
+ | JOBID PARTITION | ||
+ | ... | ||
+ | </ | ||
+ | * '' | ||
+ | < | ||
+ | | ||
+ | </ | ||
+ | ==== Multi-Cluster job submission ==== | ||
+ | [[doku: | ||
+ | |||
+ | Assume a submission script '' | ||
+ | < | ||
+ | [username@node ~]$ sbatch job.sh </ | ||
+ | To submit a job to a list of possible clusters and have the job submitted to the cluster that could run the job the soonest. Of course, to achieve that, the authentication between clusters must be operational. | ||
+ | |||
+ | |||
+ | < | ||
+ | [username@node ~]$ sbatch -M vsc4/vsc5 job.sh | ||
+ | </ | ||
+ | To submit a job to a specific cluster (here vsc4 or vsc5) | ||
+ | |||
+ | ===== Federated Slurm ===== | ||
+ | |||
+ | The Federation is based on the multi-cluster slurm implementation but provides one aggregated system from the independent Slurm clusters and works in a peer-to-peer way | ||
+ | In federation a job is submitted to the local cluster (on which the command is invoked) and is then replicated across the clusters in the federation. Each cluster then independently attempts to the schedule the job based off of its own scheduling policies. The clusters coordinate with the " | ||
+ | |||
+ | |||
+ | * Federation is used to unify the job ID and scheduling information among all clusters in the federation | ||
+ | * Multiple, independent clusters can be used as one global resource | ||
+ | * Slurmdbd pushes updates to all clusters in the federation | ||
+ | * A cluster can only be part of one federation at a time | ||
+ | * Embed cluster ID within the originally 32-bit job ID | ||
+ | {{: | ||
+ | |||
+ | < | ||
+ | [...]# sacctmgr list cluster withfed | ||
+ | | ||
+ | ---------- --------------- ------------ ----- -------- ------- ------------- ------- ------- ------------- -------- ----------- -------------------- | ||
+ | | ||
+ | | ||
+ | |||
+ | </ | ||
+ | {{: | ||
+ | ==== Federation Job Submission ==== | ||
+ | |||
+ | When a federated cluster receives a job submission, it will submit copies of the job (sibling jobs) to each eligible cluster. Each cluster will then independently attempt to schedule the job. Once a sibling job is started, the origin cluster revokes the remaining sibling jobs. | ||
+ | |||
+ | The job submission is managed the same way as for the multi-cluster solution and the new ** --cluster-constraint=[!]< | ||
+ | |||
+ | < | ||
+ | [...]# squeue -M vscdev, | ||
+ | CLUSTER: vsc4 | ||
+ | | ||
+ | | ||
+ | |||
+ | |||
+ | CLUSTER: vsc5 | ||
+ | | ||
+ | | ||
+ | </ | ||
+ | |||
+ | < | ||
+ | [root@node]# | ||
+ | Federation: vscdev_fed | ||
+ | Self: | ||
+ | Sibling: | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | ==== Slurm Federation Workflow ==== | ||
+ | {{: | ||
+ | |||
+ | |||
+ | ===== Multi-Cluster vs Federation implementation ===== | ||
+ | |||
+ | |||
+ | On a basic approach, multi-cluster is one unique interface to submit jobs to multiple separated Slurm clusters and the Slurm database can be unique or can be dedicated to each Slurm cluster while federation is a way to federate the job and scheduling information as one and the Slurm database must be unique. | ||
+ | |||
+ | |||
+ | |||
+ | ===== Slurm Burst-Buffer ===== | ||
+ | |||
+ | I/O components are much slower than the compute parts of a supercomputer, | ||
+ | |||
+ | The data staging derives large scale of traffic on a network connecting computing nodes for moving input and output data between the computing nodes. In this network, the traffic of inter-process communication also flows and consequently mutual interference between both types of traffic may degrade network performance. For example, burst traffic derived from the data staging increases delay in inter-process communication. Also, both types of traffic compete network bandwidth and consequently communication time is increased. | ||
+ | |||
+ | Burst-Buffer plugin adds a layer between the compute nodes and the parallel file system to improve network performance, |