[slurm-users] Constraint multiple counts not working

[Jeffrey T Frey]

On a cluster running Slurm 17.11.8 (cons_res) I can submit a job that requests e.g. 2 nodes with unique features on each:


$ sbatch --nodes=2 --ntasks-per-node=1 --constraint="[256GB*1&192GB*1]" …


The job is submitted and runs as expected:  on 1 node with feature "256GB" and 1 node with feature "192GB."  A similar job on a cluster running 20.11.1 (cons_res OR cons_tres, tested with both) fails to submit:


sbatch: error: Batch job submission failed: Requested node configuration is not available


I enabled debug5 output with NodeFeatures:


[2020-12-16T08:53:19.024] debug:  JobId=118 feature list: [512GB*1&768GB*1]
[2020-12-16T08:53:19.025] NODE_FEATURES: _log_feature_nodes: FEAT:512GB COUNT:1 PAREN:0 OP:XAND ACTIVE:r1n[00-47] AVAIL:r1n[00-47]
[2020-12-16T08:53:19.025] NODE_FEATURES: _log_feature_nodes: FEAT:768GB COUNT:1 PAREN:0 OP:END ACTIVE:r2l[00-31] AVAIL:r2l[00-31]
[2020-12-16T08:53:19.025] NODE_FEATURES: valid_feature_counts: feature:512GB feature_bitmap:r1n[00-47],r2l[00-31],r2x[00-10] work_bitmap:r1n[00-47],r2l[00-31],r2x[00-10] tmp_bitmap:r1n[00-47] count:1
[2020-12-16T08:53:19.025] NODE_FEATURES: valid_feature_counts: feature:768GB feature_bitmap:r1n[00-47],r2l[00-31],r2x[00-10] work_bitmap:r1n[00-47],r2l[00-31],r2x[00-10] tmp_bitmap:r2l[00-31] count:1
[2020-12-16T08:53:19.025] NODE_FEATURES: valid_feature_counts: NODES:r1n[00-47],r2l[00-31],r2x[00-10] HAS_XOR:T status:No error
[2020-12-16T08:53:19.025] select/cons_tres: _job_test: SELECT_TYPE: test 0 pass: test_only
[2020-12-16T08:53:19.026] debug2: job_allocate: setting JobId=118_* to "BadConstraints" due to a flaw in the job request (Requested node configuration is not available)
[2020-12-16T08:53:19.026] _slurm_rpc_submit_batch_job: Requested node configuration is not available


My syntax agrees with the 20.11.1 documentation (online and man pages) so it seems correct — and it works fine in 17.11.8.  Any ideas?



::::::::::::::::::::::::::::::::::::::::::::::::::::::
 Jeffrey T. Frey, Ph.D.
 Systems Programmer V / Cluster Management
 Network & Systems Services / College of Engineering
 University of Delaware, Newark DE  19716
 Office: (302) 831-6034  Mobile: (302) 419-4976
::::::::::::::::::::::::::::::::::::::::::::::::::::::

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20201216/fa13e2b5/attachment.htm>