June 2024 - slurm-users - lists.schedmd.com

Can't schedule on cloud node: State=IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING
by Xaver Stiensmeier 20 Sep '24

20 Sep '24

Dear slurm-user list, I have a cloud node that is powered up and down on demand. Rarely it can happen that slurm's resumeTimeout is reached and the node is therefore powered down. We have set ReturnToService=2 in order to avoid the node being marked down, because the instance behind that node is created on demand and therefore after a failure nothing stops the system to start the node again as it is a different instance. I thought this would be enough, but apparently the node is still marked with "NOT_RESPONDING" which leads to slurm not trying to schedule on it. After a while NOT_RESPONDING is removed, but I would like to move it directly from within my fail script if possible so that the node can return to service immediately and not be blocked by "NOT_RESPONDING". Best regards, Xaver

2 3

_refresh_assoc_mgr_qos_list: no new list given back keeping cached one
by joao.damas＠syngenta.com 05 Aug '24

05 Aug '24

Hi all, We are doing a simple setup for a Slurm cluster (version 23.11.6). We follow the documentation and we are trying a setup still without accounting or slurmdbd. The slurm.conf is really simple: ``` ClusterName=Develop SlurmctldHost=head # Slurm configuration AuthType=auth/munge CryptoType=crypto/munge SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdLogFile=/var/log/slurm/slurmd.log SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm StateSaveLocation=/var/spool/slurmctld # Nodes NodeName=worker1 CoresPerSocket=2 Sockets=1 ThreadsPerCore=1 NodeName=worker2 CoresPerSocket=2 Sockets=1 ThreadsPerCore=1 # Partitions PartitionName=develop Default=YES MaxTime=UNLIMITED Nodes="worker1,worker2" ``` When running a simple `srun sleep 10`, all works well and the log file shows: [2024-05-15T12:34:12.741] sched: _slurm_rpc_allocate_resources JobId=1 NodeList=worker1 usec=549 [2024-05-15T12:34:22.775] _job_complete: JobId=1 WEXITSTATUS 0 [2024-05-15T12:34:22.775] _job_complete: JobId=1 done But when creating a scrip with the same sleep command, and submiting using `sbatch test.sh`, the log shows: [2024-05-15T12:35:39.916] _slurm_rpc_submit_batch_job: JobId=2 InitPrio=1 usec=368 [2024-05-15T12:35:40.000] error: _refresh_assoc_mgr_qos_list: no new list given back keeping cached one. [2024-05-15T12:35:40.000] sched: JobId=2 has invalid account [2024-05-15T12:35:40.145] sched/backfill: _start_job: Started JobId=2 in develop on worker1 [2024-05-15T12:35:50.172] _job_complete: JobId=2 WEXITSTATUS 0 [2024-05-15T12:35:50.172] _job_complete: JobId=2 done We have the same account with the UID and GID, as said in the documentation. Looking at the function that seems to spit out that error (https://github.com/SchedMD/slurm/blob/e9f28ede27795f525e62f998cb2d40931d884…), it appears like there should be some accounting setup? We do not have slurmdbd setup and the documentation states we should test basic functionality before implementing that daemon. Any tips? Thanks in advance. João

4 3

Convergence of Kube and Slurm?
by Dan Healy 29 Jul '24

29 Jul '24

Bright Cluster Manager has some verbiage on their marketing site that they can manage a cluster running both Kubernetes and Slurm. Maybe I misunderstood it. But nevertheless, I am encountering groups more frequently that want to run a stack of containers that need private container networking. What’s the current state of using the same HPC cluster for both Slurm and Kube? Note: I’m aware that I can run Kube on a single node, but we need more resources. So ultimately we need a way to have Slurm and Kube exist in the same cluster, both sharing the full amount of resources and both being fully aware of resource usage. Thanks, Daniel Healy

5 4

vers 23: slurmctld pb (memory leak and response time)
by LEROY Christine 208562 09 Jul '24

09 Jul '24

Hello Everyone, We tried several version of slurm 23-x (till the last one: 23.11.8), and we observe a memory leak in slurmctld in addition to a bad response time (sometimes). It seems to be due to a slurm.conf configuration with Partition containing Allowgroups option. Is it a known problem? Is it recommended to use the version 22, or 24 ? Thanks in advance, Christine Leroy

1 1

AllowAccounts partition setting
by Marko Markoc 02 Jul '24

02 Jul '24

Hi All, At one point, when we were running slurm 22, we set up a few new partitions with `AllowAccounts` setting enabled. We limit access to these partitions only to a few accounts. We manually create these accounts and users using the `sacctmgr` command. At that time, everything was working. Only users in these accounts were able to run jobs in that partition. Few months ago we upgraded slurm to 23.02.7 but just a couple of days ago we noticed that anyone could submit jobs to these partitions. In addition to that, you can provide any value to the `--account` directive and slurm will accept and record it in the job accounting database. In our initial testing, you could only provide `--account` value that has been previously created. I'm still looking into this but I wanted to check if there was any change in version 23 that could have impacted this? I don't remember seeing anything in release notes. Thanks, Marko

5 7

Why AllowAccounts not work in slurm-23.11.6
by daijiangkuicgo＠gmail.com 29 Jun '24

29 Jun '24

I have set AllowAccounts=sunlabc5hpc,root, but it doesn’t seem to work. User c010637 is not part of the sunlabc5hpc account but is still able to use the sunlabc5hpc partition. I have tried setting EnforcePartLimits to ALL, ANY, and NO, but none of these options resolved the issue. [c010637@sl-login ~]$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST cpu* up infinite 3 mix sl-c[0035,0042-0043] cpu* up infinite 1 idle sl-c0036 gpu up infinite 3 idle sl-c[0045-0047] sunlabc5hpc up infinite 1 idle sl-c0048 [c010637@sl-login ~]$ scontrol show partition sunlabc5hpc PartitionName=sunlabc5hpc AllowGroups=ALL AllowAccounts=sunlabc5hpc,root AllowQos=ALL AllocNodes=ALL Default=NO QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED Nodes=sl-c0048 PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=256 TotalNodes=1 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=256,mem=515000M,node=1,billing=256,gres/gpu=8 [c010637@sl-login ~]$ sacctmgr list assoc format=cluster,user,account%20,qos user=$USER Cluster User Account QOS ---------- ---------- -------------------- -------------------- snowhpc c010637 c010637_bank normal [c010637@sl-login ~]$ sacctmgr list account sunlabc5hpc Account Descr Org ---------- -------------------- -------------------- sunlabc5h+ sunlabc5hpc sunlabc5hpc [c010637@sl-login ~]$ sacctmgr show assoc where Account=sunlabc5hpc format=User,Account User Account ---------- ---------- sunlabc5h+ c010751 sunlabc5h+ snowdai sunlabc5h+

1 1

Slurm version 24.05.1 is now available
by Tim Wickberg 28 Jun '24

28 Jun '24

We are pleased to announce the availability of Slurm version 24.05.1. This release addresses a number of minor-to-moderate issues since the 24.05 release was first announced a month ago. Slurm can be downloaded from https://www.schedmd.com/downloads.php . - Tim > * Changes in Slurm 24.05.1 > ========================== > -- Fix slurmctld and slurmdbd potentially stopping instead of performing a > logrotate when recieving SIGUSR2 when using auth/slurm. > -- switch/hpe_slingshot - Fix slurmctld crash when upgrading from 23.02. > -- Fix "Could not find group" errors from validate_group() when using > AllowGroups with large /etc/group files. > -- Prevent an assertion in debugging builds when triggering log rotation > in a backup slurmctld. > -- Add AccountingStoreFlags=no_stdio which allows to not record the stdio > paths of the job when set. > -- slurmrestd - Prevent a slurmrestd segfault when parsing the crontab field, > which was never usable. Now it explicitly ignores the value and emits a > warning if it is used for the following endpoints: > 'POST /slurm/v0.0.39/job/{job_id}' > 'POST /slurm/v0.0.39/job/submit' > 'POST /slurm/v0.0.40/job/{job_id}' > 'POST /slurm/v0.0.40/job/submit' > 'POST /slurm/v0.0.41/job/{job_id}' > 'POST /slurm/v0.0.41/job/submit' > 'POST /slurm/v0.0.41/job/allocate' > -- mpi/pmi2 - Fix communication issue leading to task launch failure with > "invalid kvs seq from node". > -- Fix getting user environment when using sbatch with "--get-user-env" or > "--export=" when there is a user profile script that reads /proc. > -- Prevent slurmd from crashing if acct_gather_energy/gpu is configured but > GresTypes is not configured. > -- Do not log the following errors when AcctGatherEnergyType plugins are used > but a node does not have or cannot find sensors: > "error: _get_joules_task: can't get info from slurmd" > "error: slurm_get_node_energy: Zero Bytes were transmitted or received" > However, the following error will continue to be logged: > "error: Can't get energy data. No power sensors are available. Try later" > -- sbatch, srun - Set SLURM_NETWORK environment variable if --network is set. > -- Fix cloud nodes not being able to forward to nodes that restarted with new > IP addresses. > -- Fix cwd not being set correctly when running a SPANK plugin with a > spank_user_init() hook and the new "contain_spank" option set. > -- slurmctld - Avoid deadlock during shutdown when auth/slurm is active. > -- Fix segfault in slurmctld with topology/block. > -- sacct - Fix printing of job group for job steps. > -- scrun - Log when an invalid environment variable causes the job submission > to be rejected. > -- accounting_storage/mysql - Fix problem where listing or modifying an > association when specifying a qos list could hang or take a very long time. > -- gpu/nvml - Fix gpuutil/gpumem only tracking last GPU in step. Now, > gpuutil/gpumem will record sums of all GPUS in the step. > -- Fix error in scrontab jobs when using slurm.conf:PropagatePrioProcess=1. > -- Fix slurmctld crash on a batch job submission with "--nodes 0,...". > -- Fix dynamic IP address fanout forwarding when using auth/slurm. > -- Restrict listening sockets in the mpi/pmix plugin and sattach to the > SrunPortRange. > -- slurmrestd - Limit mime types returned from query to 'GET /openapi/v3' to > only return one mime type per serializer plugin to fix issues with OpenAPI > client generators that are unable to handle multiple mime type aliases. > -- Fix many commands possibly reporting an "Unexpected Message Received" when > in reality the connection timed out. > -- Prevent slurmctld from starting if there is not a json serializer present > and the extra_constraints feature is enabled. > -- Fix heterogeneous job components not being signaled with scancel --ctld and > 'DELETE slurm/v0.0.40/jobs' if the job ids are not explicitly given, > the heterogeneous job components match the given filters, and the > heterogeneous job leader does not match the given filters. > -- Fix regression from 23.02 impeding job licenses from being cleared. > -- Move error to log_flag which made _get_joules_task error to be logged to the > user when too many rpcs were queued in slurmd for gathering energy. > -- For scancel --ctld and the associated rest api endpoints: > 'DELETE /slurm/v0.0.40/jobs' > 'DELETE /slurm/v0.0.41/jobs' > Fix canceling the final array task in a job array when the task is pending > and all array tasks have been split into separate job records. Previously > this task was not canceled. > -- Fix power_save operation after recovering from a failed reconfigure. > -- slurmctld - Skip removing the pidfile when running under systemd. In that > situation it is never created in the first place. > -- Fix issue where altering the flags on a Slurm account (UsersAreCoords) > several limits on the account's association would be set to 0 in > Slurm's internal cache. > -- Fix memory leak in the controller when relaying stepmgr step accounting to > the dbd. > -- Fix segfault when submitting stepmgr jobs within an existing allocation. > -- Added "disable_slurm_hydra_bootstrap" as a possible MpiParams parameter in > slurm.conf. Using this will disable env variable injection to allocations > for the following variables: I_MPI_HYDRA_BOOTSTRAP, > I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS, HYDRA_BOOTSTRAP, > HYDRA_LAUNCHER_EXTRA_ARGS. > -- scrun - Delay shutdown until after start requested. This caused scrun > to never start or shutdown and hung forever when using --tty. > -- Fix backup slurmctld potentially not running the agent when taking over as > the primary controller. > -- Fix primary controller not running the agent when a reconfigure of the > slurmctld fails. > -- slurmd - fix premature timeout waiting for REQUEST_LAUNCH_PROLOG with large > array jobs causing node to drain. > -- jobcomp/{elasticsearch,kafka} - Avoid sending fields with invalid date/time. > -- jobcomp/elasticsearch - Fix slurmctld memory leak from curl usage > -- acct_gather_profile/influxdb - Fix slurmstepd memory leak from curl usage > -- Fix 24.05.0 regression not deleting job hash dirs after MinJobAge. > -- Fix filtering arguments being ignored when using squeue --json. > -- switch/nvidia_imex - Move setup call after spank_init() to allow namespace > manipulation within the SPANK plugin. > -- switch/nvidia_imex - Skip plugin operation if nvidia-caps-imex-channels > device is not present rather than preventing slurmd from starting. > -- switch/nvidia_imex - Skip plugin operation if job_container/tmpfs > is configured due to incompatibility. > -- switch/nvidia_imex - Remove any pre-existing channels when slurmd starts. > -- rpc_queue - Add support for an optional rpc_queue.yaml configuration file.

1 0

SLURM QoS Preemption Not Functioning as Expected
by 木子 27 Jun '24

27 Jun '24

Hi all; I'm testing slurm preempt with PreemptType=preempt/qos and PreemptMode=suspend,gang in slurm.conf. But job with high-qos not preempt the job with low-qos. My configuration followed: slurm version: 23.11.4 test node : 8 CPUS and 8GB RAM QoS Configuration:  Partition Configuration:  Job Submission: I submitted test jobs, one with --qos=low-qos and--partition=low , resources 4 CPUs, 4GB memory, and later, a  job  with --qos=high-qos and  --partition=g1 requesting more resources( 5 CPUs, 5GB memory). Did anyone else encounter this issue or any additional steps I might need to take to correctly enable QoS-based preemption in our SLURM setup. Best regards, Liyar

1 0

scontrol update job=... comment=... fails with "Job is no longer pending"
by Thomas Zeiser 27 Jun '24

27 Jun '24

Dear Slurm community, we are running 23.02.7. For a small fraction of jobs we get "Job is no longer pending execution for job <JOBID>" when running "scontrol update job=JOBID comment=SOME-COMMENT" on an already running job. For the majority (I guess 98%) of running jobs, this works perfectly fine. I also see no reason why updating the comment of a running job should not be allowed. Did anyone else observe this issue or are there explanations why this might be expected? (We abuse the comment field to track which GPUs within a node are assigned to a job as we did not find any option within Slurm to get the GPU IDs of assigned GPUs but we need that information for our monitoring framework.) Best regards, Thomas Zeiser, NHR@FAU

1 0

Slurmctld Problems
by stth 25 Jun '24

25 Jun '24

Dear slurm users, It is my first time setting slurm up and I am looking for a solution to this errors. Has anyone here already ecountered this problem. I would really appreciate the help. mariadb, slurmdbd and slurmd are active. *×* slurmctld.service - Slurm controller daemon Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; *enabled*; preset: *enabled*) Active: *failed* (Result: exit-code) since Tue 2024-06-25 10:06:39 UTC; 2min 42s ago Duration: 584ms Docs: man:slurmctld(8) Process: 63738 ExecStart=/usr/sbin/slurmctld --systemd $SLURMCTLD_OPTIONS *(code=exited, status=1/FAILURE)* Main PID: 63738 (code=exited, status=1/FAILURE) CPU: 25ms Jun 25 10:06:39 server systemd[1]: Starting slurmctld.service - Slurm controller daemon... Jun 25 10:06:39 server (lurmctld)[63738]: *slurmctld.service: Referenced but unset environment variable evaluates to an empty string: SLURMCTLD_OPTIONS* Jun 25 10:06:39 server slurmctld[63738]: slurmctld: slurmctld version 23.11.4 started on servercluster Jun 25 10:06:39 server systemd[1]: Started slurmctld.service - Slurm controller daemon. Jun 25 10:06:39 server slurmctld[63738]: slurmctld: accounting_storage/slurmdbd: clusteracct_storage_p_register_ctld: Registering slurmctld at port 6817 with slurmdbd Jun 25 10:06:39 server slurmctld[63738]: slurmctld: priority/multifactor: _read_last_decay_ran: No last decay (/var/spool/slurm/state/priority_last_decay_ran) to recover Jun 25 10:06:39 server slurmctld[63738]: slurmctld: No memory enforcing mechanism configured. Jun 25 10:06:39 server slurmctld[63738]: slurmctld: fatal: Can not recover last_conf_lite, incompatible version, (9472 not between 9728 and 10240), start with '-i' to ignore this. Warning: using -i will lose the data that can't be recovered. Jun 25 10:06:39 server systemd[1]: *slurmctld.service: Main process exited, code=exited, status=1/FAILURE* Jun 25 10:06:39 server systemd[1]: *slurmctld.service: Failed with result 'exit-code'.*

4 7

2024

slurm-users June 2024 ----- 2024 ----- October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users June 2024