[slurm-users] slurm and singularity

[Brian Andrus]

If you are using a newer slurm, using srun for an interactive shell is 
being deprecated. Salloc now defaults to a shell if no command is specified:

DESCRIPTION
salloc is used to allocate a Slurm job allocation, which is a set of 
resources (nodes), possibly with some set of constraints (e.g. number of 
processors per node). When salloc successfully obtains the requested 
allocation, it then runs the command specified by the user. Finally, 
when the user specified command is complete, salloc relinquishes the job 
allocation.

The command may be any program the user wishes. Some typical commands 
are xterm, a shell script containing srun commands, and srun (see the 
EXAMPLES section). *If no command is specified, then salloc runs the 
user's default shell.*

Brian Andrus

On 2/8/2023 7:01 AM, Jeffrey T Frey wrote:
> You may need srun to allocate a pty for the command.  The 
> InteractiveStepOptions we use (that are handed to srun when no 
> explicit command is given to salloc) are:
>
>
>     --interactive --pty --export=TERM
>
>
>
> E.g. without those flags a bare srun gives a promptless session:
>
>
>     *[(it_nss:frey)@login00.darwin ~]$ salloc -p idle srun
>     /opt/shared/singularity/3.10.0/bin/singularity shell
>     /opt/shared/singularity/prebuilt/postgresql/13.2.simg*
>     salloc: Granted job allocation 3953722
>     salloc: Waiting for resource configuration
>     salloc: Nodes r1n00 are ready for job
>     ls -l
>     total 437343
>     -rw-r--r--  1 frey it_nss      180419 Oct 26 16:56 amd.cache
>     -rw-r--r--  1 frey it_nss          72 Oct 26 16:52 amd.conf
>     -rw-r--r--  1 frey everyone       715 Nov 12 23:39
>     anaconda-activate.sh
>     drwxr-xr-x  2 frey everyone         4 Apr 11  2022 bin
>        :
>
>
>
> With the --pty flag added:
>
>
>     *[(it_nss:frey)@login00.darwin ~]$ salloc -p idle srun --pty
>     /opt/shared/singularity/3.10.0/bin/singularity shell
>     /opt/shared/singularity/prebuilt/postgresql/13.2.simg*
>     salloc: Granted job allocation 3953723
>     salloc: Waiting for resource configuration
>     salloc: Nodes r1n00 are ready for job
>     Singularity>
>
>
>
>
>> On Feb 8, 2023, at 09:47 , Groner, Rob <rug262 at psu.edu> wrote:
>>
>> I tried that, and it says the nodes have been allocated, but it never 
>> comes to an apptainer prompt.
>>
>> I then tried doing them in separate steps.  Doing salloc works, I get 
>> a prompt on the node that was allocated.  I can then run "singularity 
>> shell <sif>" and get the apptainer prompt.  If I prefix that command 
>> with "srun", then it just hangs and I never get the prompt.  So that 
>> seems to be the sticking point.  I'll have to do some experiments 
>> running singularity with srun.
>>
>> From: slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf 
>> of Jeffrey T Frey <frey at udel.edu>
>> Sent: Tuesday, February 7, 2023 6:16 PM
>> To: Slurm User Community List <slurm-users at lists.schedmd.com>
>> Subject: Re: [slurm-users] slurm and singularity
>>
>> You don't often get email from frey at udel.edu. Learn why this is important
>>> The remaining issue then is how to put them into an allocation that 
>>> is actually running a singularity container.  I don't get how what 
>>> I'm doing now is resulting in an allocation where I'm in a 
>>> container on the submit node still!
>>
>> Try prefixing the singularity command with "srun" e.g.
>>
>>
>> salloc <salloc-parameters> srun <srun-parameters> 
>> /usr/bin/singularity shell <path to sif>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20230208/c55ec510/attachment.htm>