[slurm-users] Splitting mpi rank output

[Chris Samuel]

On Thursday, 10 May 2018 2:25:49 AM AEST Christopher Benjamin Coffey wrote:

> I have a user trying to use %t to split the mpi rank outputs into different
> files and it's not working. I verified this too. Any idea why this might
> be? This is the first that I've heard of a user trying to do this.

I think they want to use that as an argument to srun, not sbatch.

I don't know why it doesn't work for sbatch, I'm guessing it doesn't get 
passed on in the environment?  From the look of the srun manual page it 
probably should set SLURM_STDOUTMODE.  But then you'd get both the batch 
output and rank 0 going to the first one.  Seems like a bug to me.

However, I can confirm that it works if you pass it to srun instead.

[csamuel at farnarkle1 tmp]$ cat test-rank.sh
#!/bin/bash
#SBATCH --ntasks=10
#SBATCH --ntasks-per-node=1

srun -o foo-%t.out hostname

[csamuel at farnarkle1 tmp]$ ls -ltr
total 264
-rw-rw-r-- 1 csamuel hpcadmin 89 May 10 17:34 test-rank.sh
-rw-rw-r-- 1 csamuel hpcadmin  0 May 10 17:34 slurm-127420.out
-rw-rw-r-- 1 csamuel hpcadmin  7 May 10 17:34 foo-9.out
-rw-rw-r-- 1 csamuel hpcadmin  7 May 10 17:34 foo-8.out
-rw-rw-r-- 1 csamuel hpcadmin  7 May 10 17:34 foo-7.out
-rw-rw-r-- 1 csamuel hpcadmin  7 May 10 17:34 foo-6.out
-rw-rw-r-- 1 csamuel hpcadmin  7 May 10 17:34 foo-5.out
-rw-rw-r-- 1 csamuel hpcadmin  7 May 10 17:34 foo-4.out
-rw-rw-r-- 1 csamuel hpcadmin  7 May 10 17:34 foo-3.out
-rw-rw-r-- 1 csamuel hpcadmin  7 May 10 17:34 foo-2.out
-rw-rw-r-- 1 csamuel hpcadmin  7 May 10 17:34 foo-1.out
-rw-rw-r-- 1 csamuel hpcadmin  7 May 10 17:34 foo-0.out


[csamuel at farnarkle1 tmp]$ more foo-*
::::::::::::::
foo-0.out
::::::::::::::
john37
::::::::::::::
foo-1.out
::::::::::::::
john38
::::::::::::::
foo-2.out
::::::::::::::
john39
::::::::::::::
foo-3.out
::::::::::::::
john40
::::::::::::::
foo-4.out
::::::::::::::
john41
::::::::::::::
foo-5.out
::::::::::::::
john42
::::::::::::::
foo-6.out
::::::::::::::
john43
::::::::::::::
foo-7.out
::::::::::::::
john44
::::::::::::::
foo-8.out
::::::::::::::
john45
::::::::::::::
foo-9.out
::::::::::::::
john46

Hope that helps,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC