Differences

This shows you the differences between two versions of the page.

--- doku:matlab [2020/01/16 12:46] – [Run Matlab Task from command line] ir
+++ doku:matlab [2024/01/31 11:17] (current) – katrin
@@ Line 1: / Line 1: @@
-====== Matlab Batch Jobs ======
+====== Matlab VSC ======
-==== Run Matlab Task from command line ====
+===== Quickstart: Run Matlab Task from command line =====
 Generate a matlab package called ''test.m'',
@@ Line 12: / Line 12: @@
 Run this from the command line:
 <code>
-/opt/sw/matlab-80/bin/matlab -nodisplay -r  "add(2,3.4)"
+module purge
+module load Matlab/(...Version...)
+matlab -nodisplay
+run test.m
 </code>
+==== More info on MATLAB Programs and batch jobs ====
+For more information in MATLAB programs and batch jobs<html><b><font color=#cc3300> read here &#8611;</b></font></html>[[https://colab.tuwien.ac.at/display/IAVSC/MATLAB|TUCOLAB]]
+===== Getting Started with Serial and Parallel MATLAB (GUI) =====
+This page provides the steps to configure MATLAB to submit jobs to a cluster, retrieve results, and debug errors.
+The guide can be followed either on a GUI node or via JupyterHub.
+==== Configuration ====
+After logging into the cluster, start MATLAB.  On the Home tab, click Parallel > Discover Clusters… to discover the profile.  The name of the profile is **VSC**.
+Jobs will now default to the cluster rather than submit to the local machine.
+==== Configuring Jobs ====
+Prior to submitting the job, various parameters can be assigned, such as queue, e-mail, walltime, etc.  The following is a partial list of parameters.  See AdditionalProperties for the complete list.  _[Must either specify MemPerCPU for a partial node OR NumNodes for two more nodes.]_
-===== Batch Matlab job on VSC-3 =====
-In order to be able to use matlab, you have to load the program with the 'module load xyz' command
 <code>
-[username@l32]$ module avail  # select a matlab version
+>> % Get a handle to the cluster
-[username@l32]$ module load Matlab/v9.5_R2018b # load desired version
+>> c = parcluster;
-[username@l32]$ module list   # check loaded modules
-Currently Loaded Modulefiles:
+[REQUIRED]
-) Matlab/v9.5_R2018b
+>> % Specify memory per core (default: '3500mb') for a partial node
+>> c.AdditionalProperties.MemPerCPU = '6gb';
+>> % Specify number of nodes for two or more nodes
+>> c.AdditionalProperties.NumNodes = 3;
+[OPTIONAL]
+>> % Specify the account
+>> c.AdditionalProperties.AccountName = 'account-name';
+>> % Specify a constraint
+>> c.AdditionalProperties.Constraint = 'feature-name';
+>> % Request email notification of job status
+>> c.AdditionalProperties.EmailAddress = 'user-id@tuwien.ac.at';
+>> % Specify number of GPUs (default: 0)
+>> c.AdditionalProperties.GPUsPerNode = 1;
+>> % Specify the partition
+>> c.AdditionalProperties.Partition = 'partition-name';
+>> % Specify cores per node
+>> c.AdditionalProperties.ProcsPerNode = 4;
+>> % Set node exclusivity (default: false)
+>> c.AdditionalProperties.RequireExclusiveNode = true;
+>> % Use reservation
+>> c.AdditionalProperties.Reservation = 'reservation-name';
+>> % Specify the wall time (e.g., 1 day, 5 hours, 30 minutes)
+>> c.AdditionalProperties.WallTime = '1-05:30';
 </code>
-(See also the introduction to the [[https://wiki.vsc.ac.at/doku.php?id=doku:slurm|module command.]])
-Now, Matlab can be called by
+Save changes after modifying AdditionalProperties for the above changes to persist between MATLAB sessions.
 <code>
-[username@l32]$ matlab
+>> c.saveProfile
 </code>
-==== Example: Serial Matlab Task ====
-We use the matlab m-file {{doku:myplot.m|myplot.m}} and create a file called ''jobSerial.sh'' as your job script, with the following basic contents:
+To see the values of the current configuration options, display AdditionalProperties.
 <code>
-#!/bin/bash
+>> % To view current properties
-#
+>> c.AdditionalProperties
-#SBATCH -J test                     # job name
+</code>
-#SBATCH -N 1                        # number of nodes
-#SBATCH --ntasks-per-node=1
-#SBATCH --ntasks-per-core=1
-#SBATCH --threads-per-core=1
-#SBATCH --time=10                   # run time unit=minutes
-#SBATCH -L matlab@vsc
-module purge
+Unset a value when no longer needed.
-module load Matlab/v9.5_R2018b # load desired version
+<code>
+>> % Turn off email notifications
+>> c.AdditionalProperties.EmailAddress = '';
+>> c.saveProfile
+</code>
-export OMP_NUM_THREADS=1
+==== Interactive Jobs ====
+To run an interactive pool job on the cluster, continue to use parpool as before.
+<code>
+>> % Get a handle to the cluster
+>> c = parcluster;
-time matlab < myplot.m
+>> % Open a pool of 64 workers on the cluster
+>> pool = c.parpool(64);
 </code>
+Rather than running local on the local machine, the pool can now run across multiple nodes on the cluster.
 <code>
-[username@l32]$ sbatch jobSerial.sh   # submit your job
+>> % Run a parfor over 1000 iterations
-[username@l32]$ squeue -u username    # check state of your job
+>> parfor idx = 1:1000
+        a(idx) = rand;
+   end
 </code>
-==== Example: Local Matlabpool ====
-We use the m-file ''main.m'' and create the following shellscript ''jobPool.sh''
-== jobPool.sh ==
+Delete the pool when it’s no longer needed.
 <code>
-#!/bin/bash
+>> % Delete the pool
-#
+>> pool.delete
-#SBATCH -J test                     # job name
+</code>
-#SBATCH -N 1                        # number of nodes
-#SBATCH --ntasks-per-node=16        # here we use all 16 cores of the node
-#SBATCH --ntasks-per-core=1
-#SBATCH --threads-per-core=1
-#SBATCH --time=100                   # run time unit=minutes
-#SBATCH -L matlab@vsc
-module purge
+==== Independent Batch Job ====
-module load Matlab/v9.5_R2018b # load desired version
-export OMP_NUM_THREADS=1
+Use the batch command to submit asynchronous jobs to the cluster.  The batch command will return a job object which is used to access the output of the submitted job.  See the MATLAB documentation for more help on batch.
-time matlab < main.m
+<code>
+>> % Get a handle to the cluster
+>> c = parcluster;
+>> % Submit job to query where MATLAB is running on the cluster
+>> job = c.batch(@pwd, 1, {});
+>> % Query job for state
+>> job.State
+>> % If state is finished, fetch the results
+>> job.fetchOutputs{:}
+>> % Delete the job after results are no longer needed
+>> job.delete
 </code>
-== main.m ==
+To retrieve a list of running or completed jobs, call parcluster to return the cluster object.  The cluster object stores an array of jobs that are queued to run, are running, have run, or have failed.  Retrieve and view the list of jobs as shown below.
 <code>
-matlabpool %open local 8	% local profile is default setting, at most 8 workers are allowed with the local scheduler
+>> c = parcluster;
+>> jobs = c.Jobs
+>>
+>> % Get a handle to the second job in the list
+>> job2 = c.Jobs(2);
+</code>
-matlabpool size
+Once the job has been selected, fetch the results as previously done.
-n = 500; K = 50; L = 10; T = zeros(1,L);
+fetchOutputs is used to retrieve function output arguments; if calling batch with a script, use load instead.   Data that has been written to files on the cluster needs be retrieved directly from the file system (e.g., via sftp).
+<code>
+>> % Fetch all results from the second job in the list
+>> job2.fetchOutputs{:}
+</code>
-% serial inner loop
+==== Parallel Batch Job ====
-for l = 1:L
+batch can also submit parallel workflows.  Let’s use the following example for a parallel job, which is saved as parallel_example.m.
-tic
-for k = 1:K
+<code>
-    eig(rand(n));
+function [sim_t, A] = parallel_example(iter)
+if nargin==0
+    iter = 8;
 end
-T(l) = toc;
+disp('Start sim')
+t0 = tic;
+parfor idx = 1:iter
+    A(idx) = idx;
+    pause(2)
+    idx
 end
+sim_t = toc(t0);
-disp(['serial: ',num2str(mean(T))])
+disp('Sim completed')
+save RESULTS A
-% parallel inner loop
-for l = 1:L
-tic
-parfor k = 1:K
-    eig(rand(n));
-end
-T(l) = toc;
 end
+</code>
-disp(['parallel: ',num2str(mean(T))])
+This time when using the batch command, also specify a MATLAB Pool argument.
+<code>
+>> % Get a handle to the cluster
+>> c = parcluster;
-matlabpool close
+>> % Submit a batch pool job using 4 workers for 16 simulations
+>> job = c.batch(@parallel_example, 1, {16}, 'Pool',4);
+>> % View current job status
+>> job.State
+>> % Fetch the results after a finished state is retrieved
+>> job.fetchOutputs{:}
+ans =
+.8872
 </code>
-== command line on the login node ==
+The job ran in 8.89 seconds using four workers.  Note that these jobs will always request N+1 CPU cores, since one worker is required to manage the batch job and pool of workers.   For example, a job that needs eight workers will request nine CPU cores.
+Run the same simulation but increase the Pool size.  This time, to retrieve the results later, keep track of the job ID.
+**NOTE**: For some applications, there will be a diminishing return when allocating too many workers, as the overhead may exceed computation time.
 <code>
-[username@l32]$ sbatch jobPool.sh     # submit your job
+>> % Get a handle to the cluster
-[username@l32]$ squeue -u username    # check state of your job
+>> c = parcluster;
+>> % Submit a batch pool job using 8 workers for 16 simulations
+>> job = c.batch(@parallel_example, 1, {16}, 'Pool',8);
+>> % Get the job ID
+>> id = job.ID
+id =
+
+>> % Clear job from workspace (as though MATLAB exited)
+>> clear job
 </code>
+With a handle to the cluster, the findJob method searches for the job with the specified job ID.
+<code>
+>> % Get a handle to the cluster
+>> c = parcluster;
+>> % Find the old job
+>> job = c.findJob('ID', 4);
+>> % Retrieve the state of the job
+>> job.State
+ans =
+finished
+>> % Fetch the results
+>> job.fetchOutputs{:};
+ans =
+.7270
+</code>
+The job now runs in 4.73 seconds using eight workers.  Run code with different number of workers to determine the ideal number to use.
+Alternatively, to retrieve job results via a graphical user interface, use the Job Monitor (Parallel > Monitor Jobs).
+==== Helper Functions ====
+| Function   	    | Description  	|
+| clusterFeatures	| List of cluster features/constraints	|
+| clusterGpuCards   | List of cluster GPU cards             |
+| clusterPartitionNames	| List of cluster partitions	        |
+| willRun	        | Explain why job is queued	            |
+==== Debugging ====
+If a serial job produces an error, call the getDebugLog method to view the error log file.  When submitting an independent job, specify the task.
+<code>
+>> c.getDebugLog(job.Tasks)
+</code>
+For Pool jobs, only specify the job object.
+<code>
+>> c.getDebugLog(job)
+</code>
+When troubleshooting a job, the cluster admin may request the scheduler ID of the job.  This can be derived by calling getTaskSchedulerIDs
+<code>
+>> job.getTaskSchedulerIDs()
+ans =
+</code>
+==== External Resources ====
+To learn more about the MATLAB Parallel Computing Toolbox, check out these resources:
+  * [[http://www.mathworks.com/products/parallel-computing/index.html | Parallel Computing Overview]]
+  * [[http://www.mathworks.com/help/distcomp/index.html | Parallel Computing Documentation]]
+  * [[https://www.mathworks.com/help/parallel-computing/examples.html | Parallel Computing Coding Examples]]
+  * [[http://www.mathworks.com/products/parallel-computing/tutorials.html | Parallel Computing Tutorials]]
+  * [[http://www.mathworks.com/products/parallel-computing/videos.html | Parallel Computing Videos]]
+  * [[http://www.mathworks.com/products/parallel-computing/webinars.html | Parallel Computing Webinars]]