[slurm-users] Slurm version 20.11.3 is now available; reverts to older step launch semantics

[Tim Wickberg]

We are pleased to announce the availability of Slurm version 20.11.3.

This does include a major functional change to how job step launch is 
handled compared to the previous 20.11 releases. This affects srun as 
well as MPI stacks - such as Open MPI - which may use srun internally as 
part of the process launch.

One of the changes made in the Slurm 20.11 release was to the semantics 
for job steps launched through the 'srun' command. This also 
inadvertently impacts many MPI releases that use srun underneath their 
own mpiexec/mpirun command.

For 20.11.{0,1,2} releases, the default behavior for srun was changed 
such that each step was allocated exactly what was requested by the 
options given to srun, and did not have access to all resources assigned 
to the job on the node by default. This change was equivalent to Slurm 
setting the --exclusive option by default on all job steps. Job steps 
desiring all resources on the node needed to explicitly request them 
through the new '--whole' option.

In the 20.11.3 release, we have reverted to the 20.02 and older behavior 
of assigning all resources on a node to the job step by default.

This reversion is a major behavioral change which we would not generally 
do on a maintenance release, but is being done in the interest of 
restoring compatibility with the large number of existing Open MPI (and 
other MPI flavors) and job scripts that exist in production, and to 
remove what has proven to be a significant hurdle in moving to the new 
release.

Please note that one change to step launch remains - by default, in 
20.11 steps are no longer permitted to overlap on the resources they 
have been assigned. If that behavior is desired, all steps must 
explicitly opt-in through the newly added '--overlap' option.

Further details and a full explanation of the issue can be found at:
https://bugs.schedmd.com/show_bug.cgi?id=10383#c63

Slurm can be downloaded from https://www.schedmd.com/downloads.php .

- Tim

-- 
Tim Wickberg
Chief Technology Officer, SchedMD LLC
Commercial Slurm Development and Support

> * Changes in Slurm 20.11.3
> ==========================
>  -- Fix segfault when parsing bad "#SBATCH hetjob" directive.
>  -- Allow countless gpu:<type> node GRES specifications in slurm.conf.
>  -- PMIx - Don't set UCX_MEM_MMAP_RELOC for older version of UCX (pre 1.5).
>  -- Don't green-light any GPU validation when core conversion fails.
>  -- Allow updates to a reservation in the database that starts in the future.
>  -- Better check/handling of primary key collision in reservation table.
>  -- Improve reported error and logging in _build_node_list().
>  -- Fix uninitialized variable in _rpc_file_bcast() which could lead to an
>     incorrect error return from sbcast / srun --bcast.
>  -- mpi/cray_shasta - fix use-after-free on error in _multi_prog_parse().
>  -- Cray - Handle setting correct prefix for cpuset cgroup with respects to
>     expected_usage_in_bytes.  This fixes Cray's OOM killer.
>  -- mpi/pmix: Fix PMIx_Abort support.
>  -- Don't reject jobs allocating more cores than tasks with MaxMemPerCPU.
>  -- Fix false error message complaining about oversubscribe in cons_tres.
>  -- scrontab - fix parsing of empty lines.
>  -- Fix regression causing spank_process_option errors to be ignored.
>  -- Avoid making multiple interactive steps.
>  -- Fix corner case issues where step creation should fail.
>  -- Fix job rejection when --gres is less than --gpus.
>  -- Fix regression causing spank prolog/epilog not to be called unless the
>     spank plugin was loaded in slurmd context.
>  -- Fix regression preventing SLURM_HINT=nomultithread from being used
>     to set defaults for salloc->srun, sbatch->srun sequence.
>  -- Reject job credential if non-superuser sets the LAUNCH_NO_ALLOC flag.
>  -- Make it so srun --no-allocate works again.
>  -- jobacct_gather/linux - Don't count memory on tasks that have already
>     finished.
>  -- Fix 19.05/20.02 batch steps talking with a 20.11 slurmctld.
>  -- jobacct_gather/common - Do not process jobacct's with same taskid when
>     calling prec_extra.
>  -- Cleanup all tracked jobacct tasks when extern step child process finishes.
>  -- slurmrestd/dbv0.0.36 - Correct structure of dbv0.0.36_tres_list.
>  -- Fix regression causing task/affinity and task/cgroup to be out of sync when
>     configured ThreadsPerCore is different than the physical threads per core.
>  -- Fix situation when --gpus is given but not max nodes (-N1-1) in a job
>     allocation.
>  -- Interactive step - ignore cpu bind and mem bind options, and do not set
>     the associated environment variables which lead to unexpected behavior
>     from srun commands launched within the interactive step.
>  -- Handle exit code from pipe when using UCX with PMIx.