Thanks Chris,

I can verify that unsetting all these environment variables does allow you to `srun --mem 5G` within an `srun --mem 3G` (etc). I will see if this solves my problem.

Interestingly just by running ` unset SLURM_CPU_BIND SLURM_JOB_ID` I can get it working. SLURM_JOB_ID seems to be the variable that controls whether srun is inside the same job or not. Unsetting SLURM_CPU_BIND is needed to avoid "CPU binding outside of job step allocation".

Cheers

On Sat, Apr 5, 2025 at 3:39 PM Chris Samuel via slurm-users <slurm-users@lists.schedmd.com> wrote:
On 4/4/25 5:23 am, Michael Milton via slurm-users wrote:

> Plain srun re-uses the existing Slurm allocation, and specifying
> resources like --mem will just request then from the current job rather
> than submitting a new one

srun does that as it sees all the various SLURM_* environment variables
in the environment of the running job. My bet would be that if you
eliminated them from the environment of the srun then you would get a
new allocation.

I've done similar things in the past to do an sbatch for a job that
wants to run on very different hardware with:

env $(env | awk -F= '/^(SLURM|SBATCH)/ {print "-u",$1}' | paste -s -d\ )
sbatch [...]

So it could be worth substituting srun for sbatch there and see if that
helps.

Best of luck!
Chris
--
Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA

--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-leave@lists.schedmd.com