[slurm-users] Constraint multiple counts not working

[Loris Bennett]

Hi Jeffrey,

Jeffrey T Frey <frey at udel.edu> writes:

> On a cluster running Slurm 17.11.8 (cons_res) I can submit a job that requests e.g. 2 nodes with unique features on each:
>
>  $ sbatch --nodes=2 --ntasks-per-node=1 --constraint="[256GB*1&192GB*1]" …
>
> The job is submitted and runs as expected: on 1 node with feature
> "256GB" and 1 node with feature "192GB."  A similar job on a cluster
> running 20.11.1 (cons_res OR cons_tres, tested with both) fails to
> submit:
>
>  sbatch: error: Batch job submission failed: Requested node configuration is not available
>
> I enabled debug5 output with NodeFeatures:
>
>  [2020-12-16T08:53:19.024] debug:  JobId=118 feature list: [512GB*1&768GB*1]
>  [2020-12-16T08:53:19.025] NODE_FEATURES: _log_feature_nodes: FEAT:512GB COUNT:1 PAREN:0 OP:XAND ACTIVE:r1n[00-47] AVAIL:r1n[00-47]
>  [2020-12-16T08:53:19.025] NODE_FEATURES: _log_feature_nodes: FEAT:768GB COUNT:1 PAREN:0 OP:END ACTIVE:r2l[00-31] AVAIL:r2l[00-31]
>  [2020-12-16T08:53:19.025] NODE_FEATURES: valid_feature_counts: feature:512GB feature_bitmap:r1n[00-47],r2l[00-31],r2x[00-10] work_bitmap:r1n[00-47],r2l[00-31],r2x[00-10] tmp_bitmap:r1n[00-47] count:1
>  [2020-12-16T08:53:19.025] NODE_FEATURES: valid_feature_counts: feature:768GB feature_bitmap:r1n[00-47],r2l[00-31],r2x[00-10] work_bitmap:r1n[00-47],r2l[00-31],r2x[00-10] tmp_bitmap:r2l[00-31] count:1
>  [2020-12-16T08:53:19.025] NODE_FEATURES: valid_feature_counts: NODES:r1n[00-47],r2l[00-31],r2x[00-10] HAS_XOR:T status:No error
>  [2020-12-16T08:53:19.025] select/cons_tres: _job_test: SELECT_TYPE: test 0 pass: test_only
>  [2020-12-16T08:53:19.026] debug2: job_allocate: setting JobId=118_* to "BadConstraints" due to a flaw in the job request (Requested node configuration is not available)
>  [2020-12-16T08:53:19.026] _slurm_rpc_submit_batch_job: Requested node configuration is not available
>
> My syntax agrees with the 20.11.1 documentation (online and man pages) so it seems correct — and it works fine in 17.11.8.  Any ideas?

We don't use features, so I am only guessing that you have defined the
features incorrectly.  Do you have  something like

  Feature=512GB,768GB

?

However, you might want to consider memory as a consumable resource and
allow Slurm to find available nodes, rather than defining the maximum
memory as a feature.

Regards

Loris

-- 
Dr. Loris Bennett (Hr./Mr.)
ZEDAT, Freie Universität Berlin         Email loris.bennett at fu-berlin.de