[slurm-users] Slurm Job Error Output is Missing

[Maria Semple]

Hello again!

I had a realisation last night that I was probably truncating the previous
stderr output by not supplying the -a argument to tee. After some testing
this morning, I can happily say that the following script works as expected:

test.sh:
#!/bin/bash

echo "out"
echo "err" >&2
echo "err 2" 1>&2
>&2 echo "err 3"
echo "err 4" >>/dev/stderr
echo "err 5" 1>>/dev/stderr
echo "err 6" | tee -a /dev/stderr

In the end it was on my side, but I thought I would still share in case
someone else makes the same mistake.

On Mon, Sep 13, 2021 at 2:26 PM Maria Semple <maria at rstudio.com> wrote:

> Hi all,
>
> I have some jobs which write error messages to stderr, and I've noticed
> that the stderr output is not being written to file. Here is a simple
> reproduction case:
>
> test.sh:
> #!/bin/bash
>
> echo "out"
> echo "err" >&2
> echo "err 2" 1>&2
> >&2 echo "err 3"
> echo "err 4" >/dev/stderr
> echo "err 5" 1>/dev/stderr
> echo "err 6" | tee /dev/stderr
>
> When I run that script through srun, the output includes all the echoes:
> out
> err
> err 2
> err 3
> err 4
> err 5
> err 6
> err 6
>
> If I run through sbatch without separating output and error into their own
> files, I get only the last echo (err 6). If I run with separate output
> files (e.g. sbatch --output slurm-%j.out --error slurm-%j.err), I get
> this:
>
> slurm-<id>.out:
> out
> err 6
>
> slurm-<id>.err:
> err 6
>
> I would expect to see all of the err lines in the slurm-<id>.err file.
> I've tested this with Slurm 20.11.05 and Slurm 21.08.0 and the behaviour is
> the same. Any assistance would be greatly appreciated!
>
> --
> Best Regards,
> Maria
>


-- 
Best Regards,
Maria
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210914/4c427319/attachment.htm>