<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body dir="ltr">
<p><br>
</p>
<div class="moz-forward-container">Take a look at this extension to
SLURM:</div>
<blockquote>
<div class="moz-forward-container"><a moz-do-not-send="true"
href="https://github.com/NVIDIA/pyxis"
class="moz-txt-link-freetext">https://github.com/NVIDIA/pyxis</a><br>
</div>
</blockquote>
<div class="moz-forward-container">You put the container path on the
<font face="monospace">srun</font> command-line and each rank runs
inside it's own copy of the image.<br>
</div>
<div class="moz-forward-container"><br>
<hr width="100%" size="2">
<table class="moz-email-headers-table" cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">Subject:
</th>
<td>[slurm-users] slurm and singularity</td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">Date: </th>
<td>Tue, 7 Feb 2023 17:31:45 +0000</td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">From: </th>
<td>Groner, Rob <a class="moz-txt-link-rfc2396E" href="mailto:rug262@psu.edu"><rug262@psu.edu></a></td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">Reply-To:
</th>
<td>Slurm User Community List
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a></td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">To: </th>
<td><a class="moz-txt-link-abbreviated" href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a>
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<table border="1" bgcolor="#FFEB9C">
<tbody>
<tr>
<td><font size="1" face="verdana" color="black"><b>External
email: Use caution opening links or attachments</b>
</font></td>
</tr>
</tbody>
</table>
<br>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
I'm trying to setup the capability where a user can execute:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
$: sbatch <parameters> script_to_run.sh</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
and the end result is that a job is created on a node, and
that job will execute "singularity exec <path to
container> script_to_run.sh"</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
Also, that they could execute:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
$: salloc <parameters></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
and would end up on a node per their parameters, and instead
of a bash prompt, they have the singularity prompt because
they're inside a running container.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
Oddly, I ran: salloc <parameters> /usr/bin/singularity
shell <path to sif> and that allocated and said the node
was ready and gave me an apptainer prompt...cool! But when I
asked it what hostname I was on, I was NOT on the node that it
had said was ready, I was still on the submit node. When I
exit out of the apptainer shell, it ends my allocation.
Sooo...it gave me the allocation and started the apptainer
shell, but somehow I was still on the submit node.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
As far as the job, I've done some experiments with using
job_submit.lua to replace the script with one that has a
singularity call in it instead, and that might hold some
promise. But I'd have to write the passed-in script to a temp
file or something, and then have singularity exec that. That
MIGHT work.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
The results for "slurm and singularity" do not describe what
I'm trying to do. The closest thing I can find is what slurm
touts on their website, a leftover from Slurm 2017 talking
about a spank plugin that, as near as I can figure, doesn't
exist. I read through the OCI docs on the slurm website, but
it shows that using singularity with that requires all
commands to have sudo. That's not going to work.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
I'm running out of ideas here.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
Rob<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
</div>
</div>
</body>
</html>