[slurm-users] sbatch output logs get truncated

[Timo Rothenpieler]

This has started happening after upgrading slurm from 20.02 to latest 20.11.
It seems like something exits too early, before slurm, or whatever else 
is writing that file, has a chance to flush the final output buffer to disk.

For example, take this very simple batch script, which gets submitted 
via sbatch:

> #!/bin/bash
> #SBATCH --job-name=test
> #SBATCH --ntasks=1
> #SBATCH --exclusive
> set -e
> 
> echo A
> echo B
> sleep 5
> echo C

The resulting slurm-$jobid.out file is only

 > A
 > B

The final echo never gets written to the output file.

A lot of users print a final result status at the end, which then never 
hits the logs. So this is a major for them.

The scripts run to completion just fine, it's only the log being missing 
the end.
For example touching some file after the "echo C" will touch that file 
as expected.

The behaviour is also not at all consistent. Sometimes the output log is 
written as expected, with no recognizable pattern. Though this seems to 
be the exception, majority of the time it's truncated.

This was never an issue before the recent slurm update.