<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:393310618;
mso-list-type:hybrid;
mso-list-template-ids:-240081286 908129604 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">I can't speak to what happens on node failure, but I can at least get you a greatly simplified pair of scripts that will run only one copy on each node allocated:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="mso-element:para-border-div;border-top:double windowtext 2.25pt;border-left:none;border-bottom:double windowtext 2.25pt;border-right:none;padding:1.0pt 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><o:p> </o:p></p>
<p class="MsoNormal" style="border:none;padding:0in">#!/bin/bash<o:p></o:p></p>
<p class="MsoNormal" style="border:none;padding:0in"># notarray.sh<o:p></o:p></p>
<p class="MsoNormal" style="border:none;padding:0in">#SBATCH --nodes=28<o:p></o:p></p>
<p class="MsoNormal" style="border:none;padding:0in">#SBATCH --ntasks-per-node=1<o:p></o:p></p>
<p class="MsoNormal" style="border:none;padding:0in">#SBATCH --no-kill<o:p></o:p></p>
<p class="MsoNormal" style="border:none;padding:0in">echo "notarray.sh is running on $(hostname)"<o:p></o:p></p>
<p class="MsoNormal" style="border:none;padding:0in">srun --no-kill somescript.sh<o:p></o:p></p>
<p class="MsoNormal" style="border:none;padding:0in"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">and<o:p></o:p></p>
<div style="mso-element:para-border-div;border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="mso-element:para-border-div;border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in">#!/bin/bash<o:p></o:p></p>
<p class="MsoNormal" style="border:none;padding:0in"># somescript.sh<o:p></o:p></p>
<p class="MsoNormal" style="border:none;padding:0in">echo "somescript.sh is running on $(hostname)"<o:p></o:p></p>
<p class="MsoNormal" style="border:none;padding:0in"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I can verify that after submitting the job with "sbatch notarray.sh":<o:p></o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">notarray.sh ran on only one allocated node, and<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">somescript.sh ran once on each of the 28 nodes allocated, including the one that notarray.sh ran on.<o:p></o:p></li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">No need to pass srun a set of parameters for how many tasks to run, since it can figure that out from the sbatch context.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Robert Peck <rp1060@york.ac.uk><br>
<b>Date: </b>Friday, April 16, 2021 at 2:40 PM<br>
<b>To: </b>slurm-users@schedmd.com <slurm-users@schedmd.com><br>
<b>Subject: </b>[slurm-users] Grid engine slaughtering parallel jobs when any one of them fails (copy)<o:p></o:p></span></p>
</div>
<p align="center" style="margin:0in;text-align:center;background:white"><b><span style="font-size:12.0pt;color:red;background:white">External Email Warning</span></b><o:p></o:p></p>
<p align="center" style="mso-margin-top-alt:0in;margin-right:12.0pt;margin-bottom:0in;margin-left:12.0pt;text-align:center;background:white">
<b><span style="font-size:12.0pt;color:red">This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.</span></b><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div>
<div>
<div>
<div>
<div>
<div style="margin-top:6.0pt;font-size:0.875rem" id="gmail-:ha">
<div id="gmail-:h9">
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">Excuse me, I am trying to run some software on a cluster which uses the SLURM grid engine. IT support at my institution have exhausted their knowledge of SLURM in trying to debug
this rather nasty bug with a specific feature of the grid engine and suggested I try here for tips.
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">I am using jobs of the form:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><i><span style="font-size:12.0pt;font-family:"Arial",sans-serif">#!/bin/bash<br>
#SBATCH --job-name=name # Job name<br>
#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)<br>
#SBATCH --mail-user=my_email@thing.thing # Where to send mail <br>
<br>
#SBATCH --mem=2gb # Job memory request, not hugely intensive<br>
#SBATCH --time=47:00:00 # Time limit hrs:min:sec, the sim software being run from within the bash script is quite slow, extra memory can't speed it up and it can't run multi-core, hence long runs on weak nodes<br>
<br>
#SBATCH --nodes=100<br>
#SBATCH --ntasks=100<br>
#SBATCH --cpus-per-task=1<br>
<br>
#SBATCH --output=file_output_%j.log # Standard output and error log<br>
#SBATCH --account=code # Project account<br>
#SBATCH --ntasks-per-core=1 #only 1 task per core, must not be more<br>
#SBATCH --ntasks-per-node=1 #only 1 task per node, must not be more<br>
#SBATCH --ntasks-per-socket=1 #guessing here but fairly sure I don't want multiple instances trying to use same socket<br>
<br>
#SBATCH --no-kill # </span></i></b><span style="font-size:12.0pt;font-family:"Arial",sans-serif">supposedly<b><i> prevents restart of other jobs on other nodes if one of the 100 gets a NODE_FAIL<br>
<br>
<br>
echo My working directory is `pwd`<br>
echo Running job on host:<br>
echo -e '\t'`hostname` at `date`<br>
echo<br>
<br>
module load toolchain/foss/2018b<br>
cd scratch<br>
cd further_folder<br>
chmod +x my_bash_script.sh<br>
<br>
srun --no-kill -N "${SLURM_JOB_NUM_NODES}" -n "${SLURM_NTASKS}" ./my_bash_script.sh<br>
<br>
wait<br>
echo<br>
echo Job completed at `date`</i></b><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">I use a bash script to launch my special software and stuff which actually handles each job, this software is a bit weird and two copies of it WILL NOT EVER play nicely if made
to share a node. Hence this job acts to launch 100 copies on 100 nodes, each of which does its own stuff and writes out to a separate results file. I later proces the results files. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">In my scenario I want 100 jobs to run, but if one or two failed and I only got 99 or 95 back then I could work fine for further processing with just 99 or 95 result files. Getting
back a few less jobs then I want is no tragedy for my type of work.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">But the problem is that when any one node has a failure, not that rare when you're calling for 100 nodes simultaneously, SLURM would by default murder the WHOLE LOT of jobs,
and even more confusingly then restart a bunch of them which ends up with a very confusing pile of results files. I thought the --no-kill flag should prevent this fault, but instead of preventing the killing of all jobs due to a single failure it only prevents
the restart, now I get a misleading message from the cluster telling me of a good exit code when such slaughter occurs, but when I log in to the cluster I discover a grid engine massacre of my jobs, all because just one of them failed. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">I understand that for interacting jobs on many nodes then killing all of them because of one failure can be necessary, but my jobs are strictly parallel, no cross-interaction
between them at all. each is an utterly separate simulation with different starting parameters. I need to ensure that if one job fails and must be killed then the rest are not affected.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">I have been advised that due to the simulation software being such as to refuse to run >1 copy properly on any given node at once I am NOT able to use "array jobs" and must stick
to this sort of job which requests 100 nodes this way.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">Please can anyone suggest how to instruct SLURM not to massacre ALL my jobs because ONE (or a few) node(s) fails?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">All my research is being put on hold by this bug which is making getting large runs out of the cluster almost impossible, a very large fraction of the jobs I submit has a failure
on 1 of the 100 nodes and hence that very large fraction of my jobs get killed on all nodes even though only one is faulty. I don't get jobs lasting to give me many useful sets of the 100(ish) result files I need.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">P.S. just to warn you, I'm not an HPC expert or a linux power user. I'm comfortable with linux and with command lines and technical details but will probably need a bit more explanation
around answers than someone specialised in high performance computing would.<span style="color:#888888"><br clear="all">
<o:p></o:p></span></span></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#888888"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#888888">--</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#888888">Thank You</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#888888"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#888888">Rob<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">P.S. unsure of whether this is how one is supposed to add forum posts to this google group, sent twice as I wasn't sure if the earlier one got through as I might not have been
correctly subscribed at that time, thanks<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:Helvetica"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>