- slurm-users - lists.schedmd.com

SLURM QoS Preemption Not Functioning as Expected
by 木子 27 Jun '24

27 Jun '24

Hi all; I'm testing slurm preempt with PreemptType=preempt/qos and PreemptMode=suspend,gang in slurm.conf. But job with high-qos not preempt the job with low-qos. My configuration followed: slurm version: 23.11.4 test node : 8 CPUS and 8GB RAM QoS Configuration:  Partition Configuration:  Job Submission: I submitted test jobs, one with --qos=low-qos and--partition=low , resources 4 CPUs, 4GB memory, and later, a  job  with --qos=high-qos and  --partition=g1 requesting more resources( 5 CPUs, 5GB memory). Did anyone else encounter this issue or any additional steps I might need to take to correctly enable QoS-based preemption in our SLURM setup. Best regards, Liyar

1 0

scontrol update job=... comment=... fails with "Job is no longer pending"
by Thomas Zeiser 27 Jun '24

27 Jun '24

Dear Slurm community, we are running 23.02.7. For a small fraction of jobs we get "Job is no longer pending execution for job <JOBID>" when running "scontrol update job=JOBID comment=SOME-COMMENT" on an already running job. For the majority (I guess 98%) of running jobs, this works perfectly fine. I also see no reason why updating the comment of a running job should not be allowed. Did anyone else observe this issue or are there explanations why this might be expected? (We abuse the comment field to track which GPUs within a node are assigned to a job as we did not find any option within Slurm to get the GPU IDs of assigned GPUs but we need that information for our monitoring framework.) Best regards, Thomas Zeiser, NHR@FAU

1 0

Slurmctld Problems
by stth 25 Jun '24

25 Jun '24

Dear slurm users, It is my first time setting slurm up and I am looking for a solution to this errors. Has anyone here already ecountered this problem. I would really appreciate the help. mariadb, slurmdbd and slurmd are active. *×* slurmctld.service - Slurm controller daemon Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; *enabled*; preset: *enabled*) Active: *failed* (Result: exit-code) since Tue 2024-06-25 10:06:39 UTC; 2min 42s ago Duration: 584ms Docs: man:slurmctld(8) Process: 63738 ExecStart=/usr/sbin/slurmctld --systemd $SLURMCTLD_OPTIONS *(code=exited, status=1/FAILURE)* Main PID: 63738 (code=exited, status=1/FAILURE) CPU: 25ms Jun 25 10:06:39 server systemd[1]: Starting slurmctld.service - Slurm controller daemon... Jun 25 10:06:39 server (lurmctld)[63738]: *slurmctld.service: Referenced but unset environment variable evaluates to an empty string: SLURMCTLD_OPTIONS* Jun 25 10:06:39 server slurmctld[63738]: slurmctld: slurmctld version 23.11.4 started on servercluster Jun 25 10:06:39 server systemd[1]: Started slurmctld.service - Slurm controller daemon. Jun 25 10:06:39 server slurmctld[63738]: slurmctld: accounting_storage/slurmdbd: clusteracct_storage_p_register_ctld: Registering slurmctld at port 6817 with slurmdbd Jun 25 10:06:39 server slurmctld[63738]: slurmctld: priority/multifactor: _read_last_decay_ran: No last decay (/var/spool/slurm/state/priority_last_decay_ran) to recover Jun 25 10:06:39 server slurmctld[63738]: slurmctld: No memory enforcing mechanism configured. Jun 25 10:06:39 server slurmctld[63738]: slurmctld: fatal: Can not recover last_conf_lite, incompatible version, (9472 not between 9728 and 10240), start with '-i' to ignore this. Warning: using -i will lose the data that can't be recovered. Jun 25 10:06:39 server systemd[1]: *slurmctld.service: Main process exited, code=exited, status=1/FAILURE* Jun 25 10:06:39 server systemd[1]: *slurmctld.service: Failed with result 'exit-code'.*

4 7

How to exclude master from computing? Set to DRAINED?
by Xaver Stiensmeier 24 Jun '24

24 Jun '24

Dear Slurm users, in our project we exclude the master from computing before starting Slurmctld. We used to exclude the master from computing by simply not mentioning it in the configuration i.e. just not having: PartitionName=SomePartition Nodes=master or something similar. Apparently, this is not the way to do this as it is now a fatal error fatal: Unable to determine this slurmd's NodeName therefore, my *question:* What is the best practice for excluding the master node from work? I personally primarily see the option to set the node into DOWN, DRAINED or RESERVED. Since we use ReturnToService=2, I guess DOWN is not the way to go. RESERVED fits with the second part "The node is in an advanced reservation and *not generally available*." and DRAINED "The node is unavailable for use per system administrator request." fits completely. So is *DRAINED* the correct setting in such a case? Best regards, Xaver

5 5

error: unpack_header: protocol_version 9472 not supported
by Arnuld 24 Jun '24

24 Jun '24

I have built Slurm 23.11.7 on two machines. Both are running Ubuntu 22.04. While Slurm runs fine on one machine, on the 2nd machine it does not. First machine is both a controller and a node while the 2nd machine is just a node. On both machines, I built the Slurm debian package as per the Slurm docs instructions. Slurmd logs show this: error: unpack_header: protocol_version 9472 not supported error: unpacking header error: destroy_forward: no init error: slurm_receive_msg_and_forward: [[host-4.attlocal.net]:38960] failed: Message receive failure error: service_connection: slurm_receive_msg: Message receive failure debug: _service_connection: incomplete message

1 1

Can Not Use A Single GPU for Multiple Jobs
by Arnuld 24 Jun '24

24 Jun '24

I have a machine with a quad-core CPU and an Nvidia GPU with 3500+ cores. I want to run around 10 jobs in parallel on the GPU (mostly are CUDA based jobs). PROBLEM: Each job asks for only 100 shards (runs usually for a minute or so), then I should be able to run 3500/100 = 35 jobs in parallel but slurm runs only 4 jobs in parallel keeping the rest in the queue. I have this in slurm.conf and gres.conf: # GPU GresTypes=gpu,shard # COMPUTE NODES PartitionName=pzero Nodes=ALL Default=YES MaxTime=INFINITE State=UP` PartitionName=pgpu Nodes=hostgpu MaxTime=INFINITE State=UP NodeName=hostgpu NodeAddr=x.x.x.x Gres=gpu:gtx_1080_ti:1,shard:3500 CPUs=4 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=64255 State=UNKNOWN ---------------------- Name=gpu Type=gtx_1080_ti File=/dev/nvidia0 Count=1 Name=shard Count=3500 File=/dev/nvidia0

5 6

How can I tell the OS that was used to build SLURM?
by Carl Ponder 20 Jun '24

20 Jun '24

We're seeing SLURM mis-behaving on one of your clusters, that runs Ubuntu 22.04. Ampng other problems, we see an error-message regarding a missing library version that would have been shipped on Ubuntu 20.04 not 22.04. It's not clear that the library is being called from a SLURM component or OpenMPI or something else. I'm able to read some of the SLURM configuration settings, but is there a way to tell what OS had been used to build it? I imagine that the build-log would show some system-detection steps that would expose that, but don't know if it's bundled with the rest of the SLURM install somewhere.

2 1

how to safely rename a slurm user's name
by taleintervenor＠sjtu.edu.cn 20 Jun '24

20 Jun '24

Hello, I'd like to ask if there is any safe method to rename an existing slurm user to new username with the same uid? As for linux itself, it's quite common to have 2 user share the same uid. So if we already have 2 system users, for example Bob(uid=1500) and HPCBob(uid=1500), and the HPCBob is the existing slurm user, how can we rename it to Bob without lost of user's history job records or associations? Do we have to manually edit the slurm database?

2 1

IMEX plugin in Slurm 24.05
by Taras Shapovalov 19 Jun '24

19 Jun '24

Hello, Does anyone know if there is any documentation about the NVIDIA IMEX plugin for Slurm 24.05? It is not even in man page for slurm.conf, though it is in the release notes. Best regards, Taras

1 0

Help with slurmdbd and Slurmctld
by stth 18 Jun '24

18 Jun '24

Hello, I wanted help with a problem I have. I just updated my operation system and the slurmdbd and slurmctld is not working anymore. I get this Errors: slurmdbd: fatal: Database schema is too old for this version of Slurm to upgrade. slurmctld: fatal: Can not recover last_tres state, incompatible version, got 9472 need >= 9728 <= 10240,start with '-i' to ignore this. Warning: using -i will lose the data that can't be recovered. I hope someone can help !

3 3