====== Node access and job control ====== * Article written by Jan Zabloudil (VSC Team)
(last update 2017-10-09 by jz). ---- ---- ====== Node access ====== {{.:folie_12_1_connect.png?0x600}} ---- ====== Node access ====== {{.:folie_12_2_connect.png?0x600}} ---- ====== Node access ====== {{.:folie_14_salloc.png?0x600}} ---- ====== 1) Job scripts: sbatch ====== (DEMO) [me@l31]$ sbatch job.sh # submit batch job [me@l31]$ squeue -u me # find out on which node(s) job is running [me@l31]$ ssh # connect to node [me@n320-038]$ ... # connected to node named n320-038 more about from Claudia ---- ====== Interactive jobs: salloc ======
  1. with salloc EXAMPLE: linpack (DEMO)
~/examples/06_node_access_job_control/linpack [me@l31 linpack]$ salloc -J me_hpl -N 1 # allocate "1" node(s) [me@l31 linpack]$ squeue -u me # find out which node(s) is(are) allocated [me@l31 linpack]$ srun hostname # clean & check functionality & licence [me@l31 linpack]$ ssh n3XX-YYY # connect to node nXX-YYY [me@n3XX-YYY ~]$ module purge [me@n3XX-YYY ~]$ module load intel/17 intel-mkl/2017 intel-mpi/2017 [me@n3XX-YYY ~]$ cd my_directory_name/linpack [me@n3XX-YYY linpack]$ mpirun -np 16 ./xhpl stop the process (Ctlr+C or Strg+C) [me@n3XX-YYY linpack]$ exit [me@l31 linpack]$ exit ---- ====== Interactive jobs: salloc ====== ===== notes: ===== [me@l31]$ srun hostname **NOTE:** the slurm prolog script is **not** automatically run in this case. The prolog performs basic tasks such as * clean up from previous job, * checking basic node functionality, * adapting firewall settings to access license servers. ---- ===== Exercise 2: job script ===== 1.) Now submit the same job with a job script: [training@l31 my_directory_name]$ sbatch job.sh [training@l31 my_directory_name]$ squeue -u training JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 5262833 mem_0064 hpl training R 0:04 1 n341-001 2.) Login to the node listed under NODELIST and perform commands which will give you information about your job [training@l31 my_directory_name]$ ssh n341-001 Last login: Fri Apr 21 08:43:07 2017 from l31.cm.cluster ---- ===== Exercise 2: job script, cont. ===== [training@n341-001 ~]$ top top - 13:06:06 up 19 days, 2:57, 2 users, load average: 12.42, 4.16, 1.51 Tasks: 442 total, 17 running, 425 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.1 us, 0.0 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 65940112 total, 62756072 free, 1586292 used, 1597748 buff/cache KiB Swap: 0 total, 0 free, 0 used. 62903424 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12737 training 20 0 1915480 68580 18780 R 106.2 0.1 0:12.57 xhpl 12739 training 20 0 1915480 68372 18584 R 106.2 0.1 0:12.55 xhpl 12742 training 20 0 1856120 45412 13172 R 106.2 0.1 0:12.57 xhpl 12744 training 20 0 1856120 45412 13172 R 106.2 0.1 0:12.52 xhpl 12745 training 20 0 1856120 45464 13224 R 106.2 0.1 0:12.56 xhpl ... * 1 … show cpu core load * Shift+H … show threads ---- 3.) Cancel the job: [training@n341-001 ~]$ exit [training@l31 ~]$ scancel ---- ===== Prolog Failure ===== If a check in the SLURM prolog script fails on one of the nodes assigned to your job, you will see a message like the following in your slurm-$JOBID.out file: Error running slurm prolog: 228 The error code (228) tells you what kind of check has failed. A list of currently existing error codes is: ERROR_MEMORY=200 ERROR_INFINIBAND_HW=201 ERROR_INFINIBAND_SW=202 ERROR_IPOIB=203 ERROR_BEEGFS_SERVICE=204 ERROR_BEEGFS_USER=205 ERROR_BEEGFS_SCRATCH=206 ERROR_NFS=207 ERROR_USER_GROUP=220 ERROR_USER_HOME=221 ERROR_GPFS_START=228 ERROR_GPFS_MOUNT=229 ERROR_GPFS_UNMOUNT=230 * node will be //drained// (unavailable for subsequent jobs until fixed) * resubmit your job ---- ===== squeue options ===== VSC-4 > squeue -u $user JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 409879 mem_0096 training sh R 1:10:59 3 n407-030,n411-[007-008] VSC-4 > squeue -p mem_0096 | more JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) .......................................................................... 407141 mem_0096 interfac mjech3 PD 0:00 4 (Priority) 409880_[1-6] mem_0096 OP1_2_Fi wiesmann PD 0:00 1 (Resources,Priority) 409879 mem_0096 training sh R 1:30:16 3 n407-030,n411-[007-008] 402078 mem_0096 sim mboesenh R 4-22:46:13 1 n403-007 * optional reformatting via -o, for example, VSC-4 > squeue -u $user -o "%.18i %.10Q %.12q %.8j %.8u %8U %.2t %.10M %.6D %R" JOBID PRIORITY QOS NAME USER UID ST TIME NODES NODELIST(REASON) 409879 3000982 admin training sh 71177 R 1:34:55 3 n407-030,n411-[007-008] ---- ===== scontrol ===== VSC-4 > scontrol show job 409879 JobId=409879 JobName=training UserId=sh(71177) GroupId=sysadmin(60000) MCS_label=N/A Priority=3000982 Nice=0 Account=sysadmin QOS=admin JobState=RUNNING Reason=None Dependency=(null) Requeue=0 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0 RunTime=01:36:35 TimeLimit=10-00:00:00 TimeMin=N/A SubmitTime=2020-10-12T15:50:44 EligibleTime=2020-10-12T15:50:44 AccrueTime=2020-10-12T15:50:44 StartTime=2020-10-12T15:51:07 EndTime=2020-10-22T15:51:07 Deadline=N/A PreemptTime=None SuspendTime=None SecsPreSuspend=0 LastSchedEval=2020-10-12T15:51:07 Partition=mem_0096 AllocNode:Sid=l40:186217 ReqNodeList=(null) ExcNodeList=(null) NodeList=n407-030,n411-[007-008] BatchHost=n407-030 NumNodes=3 NumCPUs=288 NumTasks=3 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=288,mem=288888M,node=3,billing=288,gres/cpu_mem_0096=288 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=96296M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null) Command=(null) WorkDir=/home/fs60000/sh Power= TresPerNode=cpu_mem_0096:96 ---- ===== Accounting: sacct ===== default: VSC-4 > sacct -j 409878 JobID JobName Partition Account AllocCPUS State ExitCode ------------ ---------- ---------- ---------- ---------- ---------- -------- 409878 training mem_0096 sysadmin 288 COMPLETED 0:0 adjust output VSC-4 > sacct -j 409878 -o jobid,jobname,cluster,nodelist,Start,End,cputime,cputimeraw,ncpus,qos,account,ExitCode JobID JobName Cluster NodeList Start End CPUTime CPUTimeRAW NCPUS QOS Account ExitCode ------------ ---------- ---------- --------------- ------------------- ------------------- ---------- ---------- ---------- ---------- ---------- -------- 409878 training vsc4 n407-030,n411-+ 2020-10-12T15:49:42 2020-10-12T15:51:06 06:43:12 24192 288 admin sysadmin 0:0 ---- ===== Accounting: vsc3CoreHours.py ===== VSC-3 > vsc3CoreHours.py -h usage: vsc3CoreHours.py [-h] [-S STARTTIME] [-E ENDTIME] [-D DURATION] [-A ACCOUNTLIST] [-u USERNAMES] [-uni UNI] [-d DETAIL_LEVEL] [-keys KEYS] getting cpu usage - start - end time optional arguments: -h, --help show this help message and exit -S STARTTIME start time, e.g., 2015-04[-01[T10[:04[:01]]]] -E ENDTIME end time, e.g., 2015-04[-01[T10[:04[:01]]]] -D DURATION duration, display the last D days -A ACCOUNTLIST give comma separated list of projects for which the accounting data is calculated, e.g., p70yyy,p70xxx. Default: primary project. -u USERNAMES give comma separated list of usernames -uni UNI get usage statistics for one university -d DETAIL_LEVEL set detail level; default=0 -keys KEYS show data from qos or user perspective, use either qos or user; default=user ---- ===== Accounting: vsc3CoreHours.py, cont. ===== VSC-3 > vsc3CoreHours.py -S 2020-01-01 -E 2020-10-01 -u sh =================================================== Accounting data time range From: 2020-01-01 00:00:00 To: 2020-10-01 00:00:00 =================================================== Getting accounting information for the following account/user combinations: account: sysadmin users: sh getting data, excluding these qos: goodluck, gpu_vis, gpu_compute =============================== account core_h _______________________________ sysadmin 25775.15 _______________________________ total 25775.15 ----