====== Submitting batch jobs ======

===== Sun grid engine (SGE) =====
==== The job script ====
It is recommended to write the job script using a [[doku:win2vsc&#the_job_filetext_editors_on_the_cluster|text editors]] on the VSC //Linux// cluster. 
Editors in //Windows// may add additional invisible characters to the job file which render it unreadable and, thus, it is not executed. 

=== A - the header of the job script ===
<code>#$ -N <job_name>
#$ -pe mpich <slots>
#$ -V
#$ -l h_rt=hh:mm:ss
#$ -M <email address to notify of job events>
#$ -m beas  # all job events sent via email </code>
  -  "<job_name>" is a freely chosen descriptive name,
  -  "<slots>" is the number of processor cores that you want to use for the calculation. The compute nodes are always  reserved for your job, exclusively. If the value for "<slots>" is no integral multiple of 16, it is corrected to the next larger integral multiple of 16.
  -  "-V" declares  that  all  environment  variables in the qsub command's environment are to be  exported  to the batch job.
  -  "-l" specifies the job's runtime. This explicit specification is in particular advisable for jobs with short run times, i.e., several hours or even minutes. In order to **reduce the time in the queue**, see also the section on  [[doku:runtime|maximum runtime specification.]]
  - "-M <email address>; -m beas" request E-Mail notifications concerning job events (b .. beginning, e .. end, a .. abort or reschedule, s .. suspend).

=== B - run executable ===
The job can be started in several ways,
  - as single core job on one core (no MPI) task <code>./<executable></code>
  - as parallel single core job (no MPI) on parallel cores (see also [[doku:sequential-codes|Sequential code]])
  - as MPI-enabled application <code>
mpirun  -m $TMPDIR/machines -np $NSLOTS <executable></code>"<executable>" is substituted by the path of the MPI-enabled application.
Please note that the particular options to mpirun depend on the MPI version that you use. Current IntelMPI versions, for example, require the option ''-machinefile'' instead of ''-m'':<code>mpirun  -machinefile $TMPDIR/machines -np $NSLOTS <executable></code>
Please always check for the correct options with<code>mpirun -help</code>
               



==== Example ==== 
Here is an example job-script, requesting 32 processor cores, which will run for a maximum of 3 hours and sends emails at the beginning and at the end of the job: <code>
#$ -N hitchhiker
#$ -pe mpich 32
#$ -V
#$ -M <my.name@example.at>
#$ -m be
#$ -l h_rt=03:00:00

mpirun -machinefile $TMPDIR/machines -np $NSLOTS ./myjob</code>



====Submit your job:====
  - The job is submitted via the following command ("<job_file>" is the name of the file you just created): <code>qsub <job_file></code>
  - Check if and where your job has been scheduled: <code>qstat</code>
  - Inspect the job output. Assuming your job was assigned the id "42" and your job's name was "hitchhiker", you should be able to find the following files in the directory you started it from: <code>$ ls -l
hitchhiker.o42
hitchhiker.e42
hitchhiker.po42
hitchhiker.pe42</code> In this example hitchhiker.o42 contains the output of your job. hitchhiker.e42 contains possible error messages. In hitchhiker.po42 and hitchhiker.pe42 you might find additional information related to the parallel computing environment.
  - Delete Jobs: <code>$ qdel <job_id></code>
  - View all jobs in the queue: <code>$ qstat -u \*</code>

==== Advanced topics ====

   *[[doku:runtime|Faster job execution by runtime specification]]
   *[[doku:epilog|Epilog scripts: copying back data after job termination ]]
   *[[doku:job_chains|Job chains]]
   *[[doku:job_arrays|Job arrays]]
   *[[doku:ompmpi|Hybrid OpenMP/MPI jobs ]]
   *[[doku:memory_running_jobs|Find out memory usage of running jobs]]
   *[[doku:copy|Start copying process]]
   *[[doku:tight_integration|Tight integration]]
   *[[doku:foreingprojectjob|Using Resources of foreign Projects]]
   *[[doku:chge_parameters|Changing job parameters of already submitted jobs (qalter)]]
   *[[doku:miscellaneous|Miscellaneous]]


==== NOTE ==== 
when the option ''#$ -V'' is specified this error message will be generated in one of the output files of the grid engine:<code>
/bin/sh: module: line 1: syntax error: unexpected end of file
/bin/sh: error importing function definition for `module'
bash: module: line 1: syntax error: unexpected end of file
bash: error importing function definition for `module'</code> 
This message occurs due to a known bug in the grid engine which cannot handle functions defined in the user environment. This message can be safely ignored. This error message can be avoided by exporting particular environment variables only in your job-script:<code>#$ -v PATH
#$ -v LD_LIBRARY_PATH
</code>