- slurm-users - lists.schedmd.com

Listen to job state changes
by egonle＠posteo.me 18 Nov '24

18 Nov '24

Hello, is there any way to listen to job state changes of slurm 23.x or newer? I’d like to kind of subscribe to job state changes instead of polling for job states. Adding this feature to slurm accounting DB seems to be last option right now, although I’d like to avoid it. Thanks&Best Egonle

4 3

How to power up all ~idle nodes and verify that they have started up without issue programmatically
by Xaver Stiensmeier 15 Nov '24

15 Nov '24

Dear Slurm User list, I would like to startup all ~idle (idle and powered down) nodes and check programmatically if all came up as expected. For context: this is for a program that sets up slurm clusters with on demand cloud scheduling. In the most easiest fashion this could be executing a command like *srun FORALL hostname* which would return the names of the nodes if it succeeds and an error message otherwise. However, there's no such input value like FORALL as far as I am aware. One could use -N{total node number} as all nodes are ~idle when this executes, but I don't know an easy way to get the total number of nodes. Best regards, Xaver

3 5

First setup of slurm with a GPU node
by Patrick Begou 13 Nov '24

13 Nov '24

Hi, I'm using slurm on a small 8 nodes cluster. I've recently added one GPU node with two Nvidia A100, one with 40Gb of RAM and one with 80Gb. As using this GPU resource increase I would like to manage this resource with Gres to avoid usage conflict. But at this time my setup do not works as I can reach a GPU without reserving it: srun -n 1 -p tenibre-gpu ./a.out can use a GPU even if the reservation do not specify this resource (checked with running nvidia-smi on the node). "tenibre-gpu" is a slurm partition with only this gpu node. From the documentation I've created a gres.conf file and it is propagated on all the nodes (9 compute nodes, 1 login node and the management node) and slurmd has been restarted. gres.conf is:* ## GPU setup on tenibre-gpu-0 NodeName=tenibre-gpu-0 Name=gpu Type=A100-40 File=/dev/nvidia0 Flags=nvidia_gpu_env NodeName=tenibre-gpu-0 Name=gpu Type=A100-80 File=/dev/nvidia1 Flags=nvidia_gpu_env * * In slurm.conf I have checked these flags: ## Basic scheduling SelectTypeParameters=CR_Core_Memory SchedulerType=sched/backfill SelectType=select/cons_tres ## Generic resources GresTypes=gpu ## Nodes list .... Nodename=tenibre-gpu-0 RealMemory=257270 Sockets=2 CoresPerSocket=16 ThreadsPerCore=1 State=UNKNOWN .... #partitions PartitionName=tenibre-gpu MaxTime=48:00:00 DefaultTime=12:00:00 DefMemPerCPU=4096 MaxMemPerCPU=8192 Shared=YES State=UP Nodes=tenibre-gpu-0 ... May be I've missed something ? I'm running Slurm 20.11.7-1. Thanks for your advices. Patrick

5 6

Slurm and NVIDIA NVML
by Matthias Leopold 13 Nov '24

13 Nov '24

Hi, I'm trying to compile Slurm with NVIDIA NVML support, but the result is unexpected. I get /usr/lib/x86_64-linux-gnu/slurm/gpu_nvml.so, but when I do "ldd /usr/lib/x86_64-linux-gnu/slurm/gpu_nvml.so" there is no reference to /lib/x86_64-linux-gnu/libnvidia-ml.so.1 (which I would expect). ~$ ldd /usr/lib/x86_64-linux-gnu/slurm/gpu_nvml.so linux-vdso.so.1 (0x00007ffd9a3f4000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0bc2c06000) /lib64/ld-linux-x86-64.so.2 (0x00007f0bc2e47000) /lib/x86_64-linux-gnu/libnvidia-ml.so.1 is present during compilation. Also I can see that NVML headers where found in config.status (else I wouldn't get gpu_nvml.so at all to my understanding). Our old cluster was deployed with NVIDIA deepops (which compiles Slurm on every node) and also has NVML support. There ldd brings the expected result ~$ ldd /usr/local/lib/slurm/gpu_nvml.so ... libnvidia-ml.so.1 => /lib/x86_64-linux-gnu/libnvidia-ml.so.1 (0x00007f3b10120000) ... I can't test actual functionality with my new binaries because I don't have a node with GPUs yet. Am I missing something? thank you Matthias

2 2

Slurm release candidate version 24.11.0rc1 available for testing
by Marshall Garey 12 Nov '24

12 Nov '24

We are pleased to announce the availability of Slurm release candidate 24.11.0rc1. To highlight some new features coming in 24.11: - New gpu/nvidia plugin. This does not rely on any NVIDIA libraries, and will build by default on all systems. It supports basic GPU detection and management, but cannot currently identify GPU-to-GPU links, or provide usage data as these are not exposed by the kernel driver. - Add autodetected GPUs to the output from "slurmd -C". - Added new QOS-based reports to "sreport". - Revamped network I/O with the "conmgr" thread-pool model. - Added new "hostlist function" syntax for management commands and configuration files. - switch/hpe_slingshot - Added support for hardware collectives setup through the fabric manager. (Requires SlurmctldParameters=enable_stepmgr) - Added SchedulerParameters=bf_allow_magnetic_slot configuration option to allow backfill planning for magnetic reservations. - Added new "scontrol listjobs" and "liststeps" commands to complement "listpids", and provide --json/--yaml output for all three subcommands. - Allow jobs to be submitted against multiple QOSes. - Added new experimental "oracle" backfill scheduling support, which permits jobs to be delayed if the oracle function determines the reduced fragmentation of the network topology is sufficiently advantageous. - Improved responsiveness of the controller when jobs are requeued by replacing the "db_index" identifier with a slurmctld-generated unique identifier. ("SLUID") - New options to job_container/tmpfs to permit site-specific scripts to modify the namespace before user steps are launched, and to ensure all steps are completely captured within that namespace. This is the first release candidate of the upcoming 24.11 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://bugs.schedmd.com against the 24.11.x version and we will address them before the first production 24.11.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Container Problems with various oci.conf settings
by Jason Pascucci 12 Nov '24

12 Nov '24

Hi, I'm running on Ubuntu 20.04. I've got a clean configuration of slurmctld and slurmd on one node. 1) I've configured oci.conf to the defaults defined by "OCI.CONF EXAMPLE FOR RUNC USING RUN (RECOMMENDED OVER USING CREATE/START):". I have a container that I can run by hand: runc --rootless true run -b /opt/pilot_results/results.20241108-184831/step1 test sh-4.2# exit exit and it returns. However, when I srun --container=/opt/pilot_results/results.20241108-184831/step1 ls it hangs after completing the ls, and I have to double ctrl-c out of it. 2) I tried using the configuration for RUNC with Create/Start and it hangs on start. 3) I tried using the configuration for CRUN using RUN, I can run the container by hand with crun, but srun fails with: srun --container=/opt/pilot_results/results.20241108-194914/step1 bash bind socket to `/run/user/1008//pd-builds-bench-1.jrp.34.0.0/notify`: Address already in use sync socket closed srun: error: pd-builds-bench-1: task 0: Exited with exit code 1 4) I tried using the configuration for CRUN with Create/Start and it errors repeatedly with: slurmstepd: error: _get_container_state: RunTimeQuery failed rc:256 output:error opening file `/run/user/1008//pd-builds-bench-1.jrp.51.0.0/status`: No such file or directory I went through the (open and closed) support tickets and couldn't find anything that reflects any of these errors, and I'm pretty stuck at this point. Any help would be welcome. Thanks, JRP

1 0

slurmctld keeps segfaulting, possibly during or just after backfill
by Marcus Lauer 12 Nov '24

12 Nov '24

We are running into a problem where slurmctld is segfaulting a few times a day. We had this problem with SLURM 23.11.8 and now with 23.11.10 as well, though the problem only appears on one of the several SLURM clusters we have, and all of them use one of those versions of SLURM. I was wondering if anyone has encountered a similar issue and has any thoughts on how to prevent this. Obviously we use "SchedulerType=sched/backfill" but strangely when I switched to sched/builtin for a while there were still slurmctld segfaults. We also set "SchedulerParameters=enable_user_top,bf_max_job_test=2000". I have tried turning those off but it did not help. I have also tried tweaking several other settings to no avail. Most of the cluster runs Rocky Linux 8.10 (including the slurmctld system) though we still have some Scientific Linux 7.9 compute nodes (we compile SLURM separately for those). Here is the crash-time error from journalctl: Oct 02 06:31:20 our.host.name kernel: sched_agent[2048355]: segfault at 8 ip 00007fec755d7ea8 sp 00007fec6bffe7e8 error 4 in libslurmfull.so[7fec7555a000+1f4000] Oct 02 06:31:20 our.host.name kernel: Code: 48 39 c1 7e 19 48 c1 f8 06 ba 01 00 00 00 48 d3 e2 48 f7 da 48 0b 54 c6 10 48 21 54 c7 10 c3 b8 00 00 00 00 eb da 48 8b 4f 08 <48> 39 4e 08 48 0f 4e 4e 08 49 89 c9 48 83 f9 3f 76 4e ba 40 00 00 Oct 02 06:31:20 our.host.name systemd[1]: Started Process Core Dump (PID 2169426/UID 0). Oct 02 06:31:20 our.host.name systemd-coredump[2169427]: Process 2048344 ( slurmctld) of user 991 dumped core. This is followed by a list of each of the dozen or so related threads. The one which is dumping core is first and looks like this: Stack trace of thread 2048355: #0 0x00007fec755d7ea8 bit_and_not (libslurmfull.so) #1 0x000000000044531f _job_alloc (slurmctld) #2 0x000000000044576b _job_alloc_whole_node_internal (slurmctld) #3 0x0000000000446e6d gres_ctld_job_alloc_whole_node (slurmctld) #4 0x00007fec722e29b8 job_res_add_job (select_cons_tres.so) #5 0x00007fec722f7c32 select_p_select_nodeinfo_set (select_cons_tres.so) #6 0x00007fec756e7dc7 select_g_select_nodeinfo_set (libslurmfull.so) #7 0x0000000000496eb3 select_nodes (slurmctld) #8 0x0000000000480826 _schedule (slurmctld) #9 0x00007fec753421ca start_thread (libpthread.so.0) #10 0x00007fec745f78d3 __clone (libc.so.6) I have run slurmctld with "debug5" level logging and it appears that the error occurs right after backfill considers a large number of jobs. Slurmctld could be failing at the end of backfill or when doing something which happens just after backfill runs. Usually this is the last message before the crash: [2024-09-25T18:39:42.076] slurmscriptd: debug: _slurmscriptd_mainloop: finished If anyone has any thoughts or advice on this that would be appreciated. Thank you. -- Marcus Lauer Systems Administrator CETS Group, Research Support

2 2

The hostname resolution case sensitive
by Bill 11 Nov '24

11 Nov '24

Hi, I want to confirm that the hostname resolution is case sensitive in SLURM ? Many thanks, Bill

3 5

Discrepancy in GPU Hour Reporting Between sacct and sreport in SLURM
by Manisha Yadav 11 Nov '24

11 Nov '24

I am trying to find the GPU hour utilization for a user during a specific time period using the sacct and sreport commands. However, I am noticing a significant difference between the outputs of these two commands. Could you explain the reasons for this discrepancy? Are there specific factors or configurations in SLURM that could lead to variations in the reported GPU hours? There is a significant discrepancy in the results produced for GPU hour utilization by the sacct and sreport commands in SLURM Thanks, Manisha ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

1 0

Access denied by pam_slurm_adopt
by John Hearns 10 Nov '24

10 Nov '24

I have cluster which uses Slurm 23.11.6 When I submit a multi-node job and run something like clush -b -w $SLURM_JOB_NODELIST "date" very often the ssh command fails with: Access denied by pam_slurm_adopt: you have no active jobs on this node This will happen maybe on 50% of the nodes There is the same behaviour of I salloc a number of nodes then try to ssh to a node. I have traced this to slurmstepd spawning a long sleep, which I believe allows proctrackd to 'see' if a job is active. On nodes that I can ssh into: root 3211 1 0 Nov08 ? 00:00:00 /usr/sbin/slurmd --systemd root 3227 1 0 Nov08 ? 00:00:00 /usr/sbin/slurmstepd infinity root 24322 1 0 15:40 ? 00:00:00 slurmstepd: [15709.extern] root 24326 24322 0 15:40 ? 00:00:00 \_ sleep 100000000 On nodes where I cannot ssh: root 3226 1 0 Nov08 ? 00:00:00 /usr/sbin/slurmd --systemd root 3258 1 0 Nov08 ? 00:00:00 /usr/sbin/slurmstepd infinity Maybe I am not understanding something here? ps. I ahve tried to run the pam_slurm_adopt module with options to debug, and have not found anything useful John H

2 2

Change primary alloc node
by Bhaskar Chakraborty 04 Nov '24

04 Nov '24

Hi, Is there a way to change/control the primary node (i.e. where the initial task starts) as part of a job's allocation. For eg, if a job requires 6 CPUs & its allocation is distributed over 3 hosts h1, h2 & h3 I find that it always starts the task in 1 particularnode (say h1) irrespective of how many slots were available in the hosts. Can we somehow let slurm have the primary node as h2? Is there any C-API inside select plugin which can do this trick if we were to control it through the configured select plugin? Thanks.-Bhaskar.

2 4

Slurm Job Sched Priority
by Bhaskar Chakraborty 03 Nov '24

03 Nov '24

Hello, Is there any DS in slurmctld which portrays the dynamic relative priority of pending jobs? We are trying to use slurm for developing a scheduling solution and 1 of the problems we face at the outset is how to determinethe order of scheduling for pending jobs. One option is to find scheduling iteration window begin & close pointers & cache the job ids as seen in order & then make them the priority order at that point of time. ( This means for 500 pending jobs, say, if we can find which are the slurmctld calls which mark the beginning & end of a sched iteration then we can use the scheduling orderof jobs as the relative priority order for that period of time, of course it may change depending on fairshare, user initiated priority modification etc. ) A concrete existing data structure showing the dynamic priority itself from slurmctld would be handy. Help appreciated. Thanks! Bhaskar.

3 4

Re: Change primary alloc node
by Bhaskar Chakraborty 31 Oct '24

31 Oct '24

Hello, Just to add some context here. We plan to use slurm for developing a sched solution which interacts with a backend system. Now, the backend system has pieces of h/w which require specific host in the allocation to be the primary/master host wherein the initial task would be launched, this in turn is driven by the job's placement orientation on the h/w itself. So, our primary task should launch in the asked primary host while secondary / remote tasks would subsequently get started on other hosts. Hope this brings some context to the problem as to why a specific host is necessary to be the starting host. Regards,Bhaskar. On Thursday 31 October, 2024 at 12:04:37 am IST, Laura Hild <lsh(a)jlab.org> wrote: I think if you tell the list why you care which of the Nodes is BatchHost, they may be able to provide you with a better solution. ________________________________________ Od: Bhaskar Chakraborty via slurm-users <slurm-users(a)lists.schedmd.com> Poslano: sreda, 30. oktober 2024 12:35 Za: slurm-users(a)schedmd.com Zadeva: [slurm-users] Change primary alloc node Hi, Is there a way to change/control the primary node (i.e. where the initial task starts) as part of a job's allocation. For eg, if a job requires 6 CPUs & its allocation is distributed over 3 hosts h1, h2 & h3 I find that it always starts the task in 1 particular node (say h1) irrespective of how many slots were available in the hosts. Can we somehow let slurm have the primary node as h2? Is there any C-API inside select plugin which can do this trick if we were to control it through the configured select plugin? Thanks. -Bhaskar.

3 2

Why AllowAccounts not work in slurm-23.11.6
by daijiangkuicgo＠gmail.com 30 Oct '24

30 Oct '24

I have set AllowAccounts=sunlabc5hpc,root, but it doesn’t seem to work. User c010637 is not part of the sunlabc5hpc account but is still able to use the sunlabc5hpc partition. I have tried setting EnforcePartLimits to ALL, ANY, and NO, but none of these options resolved the issue. [c010637@sl-login ~]$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST cpu* up infinite 3 mix sl-c[0035,0042-0043] cpu* up infinite 1 idle sl-c0036 gpu up infinite 3 idle sl-c[0045-0047] sunlabc5hpc up infinite 1 idle sl-c0048 [c010637@sl-login ~]$ scontrol show partition sunlabc5hpc PartitionName=sunlabc5hpc AllowGroups=ALL AllowAccounts=sunlabc5hpc,root AllowQos=ALL AllocNodes=ALL Default=NO QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED Nodes=sl-c0048 PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=256 TotalNodes=1 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=256,mem=515000M,node=1,billing=256,gres/gpu=8 [c010637@sl-login ~]$ sacctmgr list assoc format=cluster,user,account%20,qos user=$USER Cluster User Account QOS ---------- ---------- -------------------- -------------------- snowhpc c010637 c010637_bank normal [c010637@sl-login ~]$ sacctmgr list account sunlabc5hpc Account Descr Org ---------- -------------------- -------------------- sunlabc5h+ sunlabc5hpc sunlabc5hpc [c010637@sl-login ~]$ sacctmgr show assoc where Account=sunlabc5hpc format=User,Account User Account ---------- ---------- sunlabc5h+ c010751 sunlabc5h+ snowdai sunlabc5h+

5 9

Re: 转发: What is the safe upgrade path when upgrade from slurm21.08 and mariadb5.5?
by taleintervenor＠sjtu.edu.cn 29 Oct '24

29 Oct '24

Thanks for all your help. So it seems we can skip the trouble of compiling SLURM over different mariadb versions. Tianyang Zhang SJTU Network Information Center 发件人: Sid Young <sid.young(a)gmail.com> 发送时间: 2024年10月30日 7:19 收件人: Andrej Sec <andrej.sec(a)savba.sk> 抄送: taleintervenor(a)sjtu.edu.cn; slurm-users(a)lists.schedmd.com 主题: Re: [slurm-users] Re: 转发: What is the safe upgrade path when upgrade from slurm21.08 and mariadb5.5? I recently upgraded from 20.11 to 24.05.2, before moving the cluster from CentOS 7.9 to Oracle Linux 8.10 The DB upgrade should be pretty simple, do a mysqldump first, then uninstall the old DB, change the repo's and install the new DB version. It should recognise the DB files on disk and access them. Do another DB backup on the new DB version. then roll through the Slurm upgrades. I picked the first and last version of each release, and systematically went through each node till it was done. First the slurm controller node, then the compute nodes. To avoid Job loss, drain the nodes or you end up with a situation where the slurmd can't talk to the running slurmstepd and the job(s) gets lost. (Shows as a "Protocol Error"). Ole sent me a link to this guide which mostly worked. https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/#upgrade-slurm… Sid Young W: https://off-grid-engineering.com On Tue, Oct 29, 2024 at 6:33 PM Andrej Sec via slurm-users <slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> > wrote: Hi, we are facing a similar task. We have a Slurm 22.05 / MariaDB 5.5.68 environment and want to upgrade to a newer version. According to the documentation, it’s recommended to upgrade from 22.05 to a maximum of 23.11 in one step. With the MariaDB upgrade, there’s a challenge between 10.1 and 10.2+ due to incompatible changes (https://mariadb.com/kb/en/changes-improvements-in-mariadb-10-2). This upgrade, as I understand from the documentation, requires at least slurm 22.05, where it is automatically handled by the slurmdbd service. In the test lab, we performed the following tests: a. Incremental upgrade - according to MariaDB recommendations: 1. Upgrade MariaDB 5.5.68 -> 10.1.48 -> 10.2.44 2. Start the Slurm suite 22.05, checking content after each MariaDB upgrade step. During the 10.1 -> 10.2 upgrade, the slurmdbd service automatically converted the database to the required format. We had enabled general.log in MariaDB, allowing detailed inspection of database changes during conversion. 3. Upgrade slurmdbd to version 23.11 4. Upgrade slurmctld to version 23.11 5. Upgrade slurmd to version 23.11 6. Check the database content and compare tests before and after the upgrade (we used various reports with scontrol, sreport, sacct, sacctmgr for verification). b. Direct MariaDB upgrade from 5.5.68 to 10.2.44 using the same approach. According to the tests, this resulted in the same state as the incremental approach. PS: If you proceed with the upgrade, I would appreciate it if you could let us know about any potential challenges you encountered. Andrej Sec nscc, Bratislava, Slovakia _____ Od: "hermes via slurm-users" <slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> > Komu: slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> Odoslané: pondelok, 28. október 2024 8:48:19 Predmet: [slurm-users] =?eucgb2312_cn?q?=D7=AA=B7=A2=3A_What_is_the_safe_upgrade_path_when_upgrade_from_slurm21=2E08_and_mariadb5=2E5=3F?= Hi everyone: We are currently running business on SLURM21.08 and mariadb5.5. When talking about the upgrade, we need to keep all the users and jobs history data. And we see the official document wrote: “When upgrading an existing accounting database to MariaDB 10.2.1 or later from an older version of MariaDB or any version of MySQL, ensure you are running slurmdbd 22.05.7 or later. These versions will gracefully handle changes to MariaDB default values that can cause problems for slurmdbd.” So is this mean we have to firstly build SLURM>22.05 over mariadb5.5, and do the SLURM upgrade. Then upgrade the mariadb to newer version, and rebuild the same version of SLURM over new mariadb-devel? And is it safe to jump directly from mariadb5.5 to latest version? How can we check whether the slurm have correctly inherited the historical data? Thanks, Tianyang Zhang SJTU Network Information Center -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com> -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com>

1 0

转发: What is the safe upgrade path when upgrade from slurm21.08 and mariadb5.5?
by taleintervenor＠sjtu.edu.cn 29 Oct '24

29 Oct '24

Hi everyone: We are currently running business on SLURM21.08 and mariadb5.5. When talking about the upgrade, we need to keep all the users and jobs history data. And we see the official document wrote: “When upgrading an existing accounting database to MariaDB 10.2.1 or later from an older version of MariaDB or any version of MySQL, ensure you are running slurmdbd 22.05.7 or later. These versions will gracefully handle changes to MariaDB default values that can cause problems for slurmdbd.” So is this mean we have to firstly build SLURM>22.05 over mariadb5.5, and do the SLURM upgrade. Then upgrade the mariadb to newer version, and rebuild the same version of SLURM over new mariadb-devel? And is it safe to jump directly from mariadb5.5 to latest version? How can we check whether the slurm have correctly inherited the historical data? Thanks, Tianyang Zhang SJTU Network Information Center

3 2

Job pre / post submit scripts
by Bhaskar Chakraborty 29 Oct '24

29 Oct '24

Hi, Is there an option in slurm to launch a custom script at the time of job submission through sbatchor salloc? The script should run with submit user permission in submit area. The idea is that we need to enquire something which characterises our job’s requirement like CPUslots, memory etc from a central server and we do need read access to user area prior to that. In our use case the user doesn’t necessarily know beforehand what kind of resource his job needs.(Hence, the need for such a script which will contact the server with user area info.) Based on it we can modify the job a little later. A post submit script, if available, would inform us the slurm job id as well, it would get called just after the job has entered the system and prior to its scheduling. Thanks,Bhaskar. Sent from Yahoo Mail for iPad

4 4

Scheduling oddity with multiple GPU types in same partition
by Kevin M. Hildebrand 29 Oct '24

29 Oct '24

We have a 'gpu' partition with 30 or so nodes, some with A100s, some with H100s, and a few others. It appears that when (for example) all of the A100 GPUs are in use, if there are additional jobs requesting A100 GPUs pending, and those jobs have the highest priority in the partition, then jobs submitted for H100s won't run even if there are idle H100s. This is a small subset of our present pending queue- the four bottom jobs should be running, but aren't. The top pending job shows reason 'Resources' while the rest all show 'Priority'. Any thoughts on why this might be happening? JOBID PRIORITY TRES_ALLOC 8317749 501490 cpu=4,mem=80000M,node=1,billing=48,gres/gpu=1,gres/gpu:a100=1 8317750 501490 cpu=4,mem=80000M,node=1,billing=48,gres/gpu=1,gres/gpu:a100=1 8317745 501490 cpu=4,mem=80000M,node=1,billing=48,gres/gpu=1,gres/gpu:a100=1 8317746 501490 cpu=4,mem=80000M,node=1,billing=48,gres/gpu=1,gres/gpu:a100=1 8338679 500060 cpu=4,mem=64G,node=1,billing=144,gres/gpu=1,gres/gpu:h100=1 8338678 500060 cpu=4,mem=64G,node=1,billing=144,gres/gpu=1,gres/gpu:h100=1 8338677 500060 cpu=4,mem=64G,node=1,billing=144,gres/gpu=1,gres/gpu:h100=1 8338676 500060 cpu=4,mem=64G,node=1,billing=144,gres/gpu=1,gres/gpu:h100=1 Thanks, Kevin -- Kevin Hildebrand University of Maryland Division of IT

2 1

I need to limit the number of jobs per user per partition
by Stephen Connolly 28 Oct '24

28 Oct '24

I am unable to limit the number of jobs per user per partition. I have searched the internet, the forums and the slurm documentation. I created a partition with a QOS having MaxJobsPU=1 and MaxJobsPA=1 Created a user stephen with account=stephen and MaxJobs=1 However if I sbatch a test job (sleep 180) multiple times they all run concurrently. I am at a loss of what else to do. Help would be very appreciated . Thank you -- Stephen Connolly JSI Data Systems Ltd 613-727-9353 stephen(a)jsidata.ca

2 1

jobs dropping
by Mihai Ciubancan 24 Oct '24

24 Oct '24

Hello, We are trying to run some PiconGPU codes on a machine with 8x100H, susing slurm. But the jobs don't run, and are not in the queue. In slurmd logs I have: [2024-10-24T09:50:40.934] CPU_BIND: _set_batch_job_limits: Memory extracted from credential for StepId=1079.batch job_mem_limit= 648000 [2024-10-24T09:50:40.934] Launching batch job 1079 for UID 1009 [2024-10-24T09:50:40.938] debug: acct_gather_energy/none: init: AcctGatherEnergy NONE plugin loaded [2024-10-24T09:50:40.938] debug: acct_gather_profile/none: init: AcctGatherProfile NONE plugin loaded [2024-10-24T09:50:40.938] debug: acct_gather_interconnect/none: init: AcctGatherInterconnect NONE plugin loaded [2024-10-24T09:50:40.938] debug: acct_gather_filesystem/none: init: AcctGatherFilesystem NONE plugin loaded [2024-10-24T09:50:40.939] debug: gres/gpu: init: loaded [2024-10-24T09:50:41.022] [1079.batch] debug: cgroup/v2: init: Cgroup v2 plugin loaded [2024-10-24T09:50:41.026] [1079.batch] debug: CPUs:192 Boards:1 Sockets:2 CoresPerSocket:48 ThreadsPerCore:2 [2024-10-24T09:50:41.026] [1079.batch] debug: jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded [2024-10-24T09:50:41.026] [1079.batch] CPU_BIND: Memory extracted from credential for StepId=1079.batch job_mem_limit=648000 step_mem_limit=648000 [2024-10-24T09:50:41.027] [1079.batch] debug: laying out the 8 tasks on 1 hosts mihaigpu2 dist 2 [2024-10-24T09:50:41.027] [1079.batch] gres_job_state gres:gpu(7696487) type:(null)(0) job:1079 flags: [2024-10-24T09:50:41.027] [1079.batch] total_gres:8 [2024-10-24T09:50:41.027] [1079.batch] node_cnt:1 [2024-10-24T09:50:41.027] [1079.batch] gres_cnt_node_alloc[0]:8 [2024-10-24T09:50:41.027] [1079.batch] gres_bit_alloc[0]:0-7 of 8 [2024-10-24T09:50:41.027] [1079.batch] debug: Message thread started pid = 459054 [2024-10-24T09:50:41.027] [1079.batch] debug: Setting slurmstepd(459054) oom_score_adj to -1000 [2024-10-24T09:50:41.027] [1079.batch] debug: switch/none: init: switch NONE plugin loaded [2024-10-24T09:50:41.027] [1079.batch] debug: task/cgroup: init: core enforcement enabled [2024-10-24T09:50:41.027] [1079.batch] debug: task/cgroup: task_cgroup_memory_init: task/cgroup/memory: total:2063720M allowed:100%(enforced), swap:0%(permissive), max:100%(2063720M) max+swap:100%(4127440M) min:30M kmem:100%(2063720M permissive) min:30M [2024-10-24T09:50:41.027] [1079.batch] debug: task/cgroup: init: memory enforcement enabled [2024-10-24T09:50:41.027] [1079.batch] debug: task/cgroup: init: Tasks containment cgroup plugin loaded [2024-10-24T09:50:41.027] [1079.batch] cred/munge: init: Munge credential signature plugin loaded [2024-10-24T09:50:41.027] [1079.batch] debug: job_container/none: init: job_container none plugin loaded [2024-10-24T09:50:41.030] [1079.batch] debug: spank: opening plugin stack /etc/slurm/plugstack.conf [2024-10-24T09:50:41.030] [1079.batch] debug: task/cgroup: task_cgroup_cpuset_create: job abstract cores are '0-63' [2024-10-24T09:50:41.030] [1079.batch] debug: task/cgroup: task_cgroup_cpuset_create: step abstract cores are '0-63' [2024-10-24T09:50:41.030] [1079.batch] debug: task/cgroup: task_cgroup_cpuset_create: job physical CPUs are '0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,96,98,100,102,104,106,108,110,112,114,116,118,120,122,124,126,128,130,132,134,136,138,140,142,144,146,148,150,152,154,156,158' [2024-10-24T09:50:41.030] [1079.batch] debug: task/cgroup: task_cgroup_cpuset_create: step physical CPUs are '0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,96,98,100,102,104,106,108,110,112,114,116,118,120,122,124,126,128,130,132,134,136,138,140,142,144,146,148,150,152,154,156,158' [2024-10-24T09:50:41.031] [1079.batch] task/cgroup: _memcg_initialize: job: alloc=648000MB mem.limit=648000MB memsw.limit=unlimited [2024-10-24T09:50:41.031] [1079.batch] task/cgroup: _memcg_initialize: step: alloc=648000MB mem.limit=648000MB memsw.limit=unlimited [2024-10-24T09:50:41.064] [1079.batch] debug levels are stderr='error', logfile='debug', syslog='quiet' [2024-10-24T09:50:41.064] [1079.batch] starting 1 tasks [2024-10-24T09:50:41.064] [1079.batch] task 0 (459058) started 2024-10-24T09:50:41 [2024-10-24T09:50:41.069] [1079.batch] _set_limit: RLIMIT_NOFILE : reducing req:1048576 to max:131072 [2024-10-24T09:51:23.066] debug: _rpc_terminate_job: uid = 64030 JobId=1079 [2024-10-24T09:51:23.067] debug: credential for job 1079 revoked [2024-10-24T09:51:23.067] [1079.batch] debug: Handling REQUEST_SIGNAL_CONTAINER [2024-10-24T09:51:23.067] [1079.batch] debug: _handle_signal_container for StepId=1079.batch uid=64030 signal=18 [2024-10-24T09:51:23.068] [1079.batch] Sent signal 18 to StepId=1079.batch [2024-10-24T09:51:23.068] [1079.batch] debug: Handling REQUEST_SIGNAL_CONTAINER [2024-10-24T09:51:23.068] [1079.batch] debug: _handle_signal_container for StepId=1079.batch uid=64030 signal=15 [2024-10-24T09:51:23.068] [1079.batch] error: *** JOB 1079 ON mihaigpu2 CANCELLED AT 2024-10-24T09:51:23 *** [2024-10-24T09:51:23.069] [1079.batch] Sent signal 15 to StepId=1079.batch [2024-10-24T09:51:23.069] [1079.batch] debug: Handling REQUEST_STATE [2024-10-24T09:51:23.071] [1079.batch] task 0 (459058) exited. Killed by signal 15. [2024-10-24T09:51:23.090] [1079.batch] debug: Handling REQUEST_STATE [2024-10-24T09:51:23.141] [1079.batch] debug: Handling REQUEST_STATE [2024-10-24T09:51:23.241] [1079.batch] debug: Handling REQUEST_STATE [2024-10-24T09:51:23.741] [1079.batch] debug: Handling REQUEST_STATE [2024-10-24T09:51:24.073] [1079.batch] debug: signaling condition [2024-10-24T09:51:24.073] [1079.batch] debug: jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded [2024-10-24T09:51:24.073] [1079.batch] debug: task/cgroup: fini: Tasks containment cgroup plugin unloaded [2024-10-24T09:51:24.073] [1079.batch] debug: get_exit_code task 0 killed by cmd [2024-10-24T09:51:24.073] [1079.batch] job 1079 completed with slurm_rc = 0, job_rc = 15 [2024-10-24T09:51:24.075] [1079.batch] debug: Message thread exited [2024-10-24T09:51:24.154] [1079.batch] done with job Anyone have any idea what could be the problem? Thank you, Mihai

2 1

Randomly draining nodes
by Nacereddine Laddaoui 24 Oct '24

24 Oct '24

Hello everyone, I’ve recently encountered an issue where some nodes in our cluster enter a drain state randomly, typically after completing long-running jobs. Below is the output from the |sinfo| command showing the reason *“Prolog error”* : |root@controller-node:~# sinfo -R REASON USER TIMESTAMP NODELIST Prolog error slurm 2024-09-24T21:18:05 node[24,31] | When checking the |slurmd.log| files on the nodes, I noticed the following errors: |[2024-09-24T17:18:22.386] [217703.extern] error: _handle_add_extern_pid_internal: Job 217703 can't add pid 3311892 to jobacct_gather plugin in the extern_step. **(repeated 90 times)** [2024-09-24T17:18:22.917] [217703.extern] error: _handle_add_extern_pid_internal: Job 217703 can't add pid 3313158 to jobacct_gather plugin in the extern_step. ... [2024-09-24T21:17:45.162] launch task StepId=217703.0 request from UID:54059 GID:1600 HOST:<SLURMCTLD_IP> PORT:53514 [2024-09-24T21:18:05.166] error: Waiting for JobId=217703 REQUEST_LAUNCH_PROLOG notification failed, giving up after 20 sec [2024-09-24T21:18:05.166] error: slurm_send_node_msg: [(null)] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC_MSG) failed: Unexpected missing socket error [2024-09-24T21:18:05.166] error: _rpc_launch_tasks: unable to send return code to address:port=<SLURMCTLD_IP>:53514 msg_type=6001: No such file or directory | If you know how to solve these errors, please let me know. I would greatly appreciate any guidance or suggestions for further troubleshooting. Thank you in advance for your assistance. Best regards, -- Télécom Paris <https://www.telecom-paris.fr> *Nacereddine LADDAOUI* Ingénieur de Recherche et de Développement 19 place Marguerite Perey CS 20031 91123 Palaiseau Cedex Site web Télécom Paris <https://www.telecom-paris.fr>X Télécom Paris <https://twitter.com/TelecomParis_>Facebook Télécom Paris <https://www.facebook.com/TelecomParis>LinkedIn Télécom Paris <https://www.linkedin.com/school/telecom-paris/>Instagram Télécom Paris <https://www.instagram.com/telecom_paris/>Blog Télécom Paris <https://imtech.wp.imt.fr> Une école de l'IMT <https://www.imt.fr>

6 9

loss of "unchangeable" node features
by Laura Hild 23 Oct '24

23 Oct '24

Has anyone else noticed, somewhere between versions 22.05.11 and 23.11.9, losing fixed Features defined for a node in slurm.conf, and instead now just having those controlled by a NodeFeaturesPlugin like node_features/knl_generic?

1 0

Slurm version 24.05.4 is now available (CVE-2024-48936)
by Tim Wickberg 23 Oct '24

23 Oct '24

Slurm version 24.05.4 is now available and includes a fix for a recently discovered security issue with the new stepmgr subsystem. SchedMD customers were informed on October 9th and provided a patch on request; this process is documented in our security policy. [1] A mistake in authentication handling in stepmgr could permit an attacker to execute processes under other users' jobs. This is limited to jobs explicitly running with --stepmgr, or on systems that have globally enabled stepmgr through "SlurmctldParameters=enable_stepmgr" in their configuration. CVE-2024-48936. Downloads are available at https://www.schedmd.com/downloads.php . Release notes follow below. - Tim [1] https://www.schedmd.com/security-policy/ -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support > * Changes in Slurm 24.05.4 > ========================== > -- Fix generic int sort functions. > -- Fix user look up using possible unrealized uid in the dbd. > -- Fix FreeBSD compile issue with tls/none plugin. > -- slurmrestd - Fix regressions that allowed slurmrestd to be run as SlurmUser > when SlurmUser was not root. > -- mpi/pmix fix race conditions with het jobs at step start/end which could > make srun to hang. > -- Fix not showing some SelectTypeParameters in scontrol show config. > -- Avoid assert when dumping removed certain fields in JSON/YAML. > -- Improve how shards are scheduled with affinity in mind. > -- Fix MaxJobsAccruePU not being respected when MaxJobsAccruePA is set > in the same QOS. > -- Prevent backfill from planning jobs that use overlapping resources for the > same time slot if the job's time limit is less than bf_resolution. > -- Fix memory leak when requesting typed gres and --[cpus|mem]-per-gpu. > -- Prevent backfill from breaking out due to "system state changed" every 30 > seconds if reservations use REPLACE or REPLACE_DOWN flags. > -- slurmrestd - Make sure that scheduler_unset parameter defaults to true even > when the following flags are also set: show_duplicates, skip_steps, > disable_truncate_usage_time, run_away_jobs, whole_hetjob, > disable_whole_hetjob, disable_wait_for_result, usage_time_as_submit_time, > show_batch_script, and or show_job_environment. Additionaly, always make > sure show_duplicates and disable_truncate_usage_time default to true when > the following flags are also set: scheduler_unset, scheduled_on_submit, > scheduled_by_main, scheduled_by_backfill, and or job_started. This effects > the following endpoints: > 'GET /slurmdb/v0.0.40/jobs' > 'GET /slurmdb/v0.0.41/jobs' > -- Ignore --json and --yaml options for scontrol show config to prevent mixing > output types. > -- Fix not considering nodes in reservations with Maintenance or Overlap flags > when creating new reservations with nodecnt or when they replace down nodes. > -- Fix suspending/resuming steps running under a 23.02 slurmstepd process. > -- Fix options like sprio --me and squeue --me for users with a uid greater > than 2147483647. > -- fatal() if BlockSizes=0. This value is invalid and would otherwise cause the > slurmctld to crash. > -- sacctmgr - Fix issue where clearing out a preemption list using > preempt='' would cause the given qos to no longer be preempt-able until set > again. > -- Fix stepmgr creating job steps concurrently. > -- data_parser/v0.0.40 - Avoid dumping "Infinity" for NO_VAL tagged "number" > fields. > -- data_parser/v0.0.41 - Avoid dumping "Infinity" for NO_VAL tagged "number" > fields. > -- slurmctld - Fix a potential leak while updating a reservation. > -- slurmctld - Fix state save with reservation flags when a update fails. > -- Fix reservation update issues with parameters Accounts and Users, when > using +/- signs. > -- slurmrestd - Don't dump warning on empty wckeys in: > 'GET /slurmdb/v0.0.40/config' > 'GET /slurmdb/v0.0.41/config' > -- Fix slurmd possibly leaving zombie processes on start up in configless when > the initial attempt to fetch the config fails. > -- Fix crash when trying to drain a non-existing node (possibly deleted > before). > -- slurmctld - fix segfault when calculating limit decay for jobs with an > invalid association. > -- Fix IPMI energy gathering with multiple sensors. > -- data_parser/v0.0.39 - Remove xassert requiring errors and warnings to have a > source string. > -- slurmrestd - Prevent potential segfault when there is an error parsing an > array field which could lead to a double xfree. This applies to several > endpoints in data_parser v0.0.39, v0.0.40 and v0.0.41. > -- scancel - Fix a regression from 23.11.6 where using both the --ctld and > --sibling options would cancel the federated job on all clusters instead of > only the cluster(s) specified by --sibling. > -- accounting_storage/mysql - Fix bug when removing an association > specified with an empty partition. > -- Fix setting multiple partition state restore on a job correctly. > -- Fix difference in behavior when swapping partition order in job submission. > -- Fix security issue in stepmgr that could permit an attacker to execute > processes under other users' jobs. CVE-2024-48936.

1 0

HOST CPUs not used by jobs
by Bhaskar Chakraborty 21 Oct '24

21 Oct '24

I have a SLURM configuration of 2 hosts with 6 + 4 CPUs. I am submitting jobs with sbatch -n <CPU slots> <job script>. However, I see that even when I have exhausted all 10 CPU slots for the running jobs it's still allowing subsequent jobs to run ! The CPU slots availability is also show as full for the 2 hosts. No job is found pending. What could be problem? My Slurm.conf looks like (host names are changed to generic): ClusterName=MyClusterControlMachine=host1ControlAddr=<some address>SlurmUser=slurmsa#AuthType=auth/mungeStateSaveLocation=/var/spool/slurmdSlurmdSpoolDir=/var/spool/slurmdSlurmctldLogFile=/var/log/slurm/slurmctld.logSlurmdDebug=3SlurmctldDebug=6SlurmdLogFile=/var/log/slurm/slurmd.logAccountingStorageType=accounting_storage/slurmdbdAccountingStorageHost=host1#AccountingStoragePass=medslurmpass#AccountingStoragePass=/var/run/munge/munge.socket.2AccountingStorageUser=slurmsa#TaskPlugin=task/cgroupNodeName=host1 CPUs=6 SocketsPerBoard=3 CoresPerSocket=2 ThreadsPerCore=1 State=UNKNOWNNodeName=host2 CPUs=4 ThreadsPerCore=1 State=UNKNOWNPartitionName=debug Nodes=host1,host2 Default=YES MaxTime=INFINITE State=UPJobAcctGatherType=jobacct_gather/linuxJobAcctGatherFrequency=30 SelectType=select/cons_tresSelectTypeParameters=CR_CPUTaskPlugin=task/affinity Thanks in advance for any help! Regards,Bhaskar.

2 1

Customization of (error) messages after the job submission
by Sebastian Sitkiewicz 21 Oct '24

21 Oct '24

Dear SLUR Users and Administrators, I am interested in a way to customize the job submission exit statuses (mainly error codes) after the job has already been queued by the SLURM controller. We aim to provide more user-friendly messages and reminders in case of any errors or obstacles (also adjusted to our QoS/account system). For example, in the case of exceeding CPU minutes of given QoS (or account) and after the (successful) job submission, we would like to notify the user that his job has been queued (as it should) but won’t start until the CPU minutes limits are increased (and that he should contact the administrators to apply for more resources). Similarly, if the user queued a job that cannot be launched immediately because of exceeding the MaxJobs limit (per user), we would like to also give him some additional message after the srun/sbatch submission. We want to provide such information immediately after the job submission, without the need to check the status using `squeue` by the user. In the Job Launch Guide (https://slurm.schedmd.com/job_launch.html) there are distinguished following steps: 1. Call job_submit plugins to modify the request as appropriate 2. Validate that the options are valid for this user (e.g. valid partition name, valid limits, etc.) 3. Determine if this job is the highest priority runnable job, if so then really try to allocate resources for it now, otherwise only validate that it could run if no other jobs existed 4. Determine which nodes could be used for the job. If the feature specification uses an exclusive OR option, then multiple iterations of the selection process below will be required with disjoint sets of nodes 5. Call the select plugin to select the best resources for the request 6. The select plugin will consider network topology and the topology within a node (e.g. sockets, cores, and threads) to select the best resources for the job 7. If the job can not be initiated using available resources and preemption support is configured, the select plugin will also determine if the job can be initiated after preempting lower priority jobs. If so then initiate preemption as needed to start the job. From my understanding, to achieve our goal one would need to have access to source code or plugin related to point 2 (and some part of point 3). Unfortunately, the job_submit (lua) plugin from point 1 (and the cli_filter plugin as well) cannot be used because it only has access to the information on the parameters of the submitted job and the SLURM partitions (but not the QoS/account usage and their limits). Is there any way to extend the customization of job submission to include such features? Best regards, Sebastian -- dr inż. Sebastian Sitkiewicz Politechnika Wrocławska Wrocławskie Centrum Sieciowo-Superkomputerowe Dział Usług Obliczeniowych Wyb. Wyspiańskiego 27 50-370 Wrocław www.wcss.pl

1 0

Tracking costs - one single pool of credits, variable costs per partition
by John Snowdon 18 Oct '24

18 Oct '24

We are trying to design the charging and accounting system for our new institutional HPC facility and I'm having difficulty understanding exactly how we can use sacctmgr to achieve what we need. Until now, our previous HPC facilities have all operated as free delivery and we have not needed to track costs by user/group/project. Account codes have been purely optional. However, our new facility will be split into various resource types, with free partitions and paid/priority/reserved partitions across those resource types. All jobs will need to be submitted with an account code. For users submitting to 'free' partitions we don't need to track resource units against a balance, but the submitted account code would still be used for reporting purposes (i.e. "free resources accounted for % of all use by this project in August-September"). When submitting to a 'paid' partition, the account code needs to be checked to ensure it has a positive balance (or a balance that will not go past some negative threshold). Each of the 'paid' partitions may (will) have a different resource unit cost. A simple example: - Submit to a generic CPU paid partition -- 1 resource unit/token/credit/£/$ per allocated cpu, per hour of compute - Submit to a high-speed, non-blocking CPU paid partition -- 2 resource unit/token/credit/£/$ per allocated cpu, per hour of compute - Submit to a GPU paid partition -- 4 resource unit/token/credit/£/$ per allocated GPU card, per hour of compute We need to have *one* pool of resource units/tokens/credits per account - let's say 1000 credits, and a group of users may well decide to spend all of their credits on the generic CPU partition, all on the GPU partition, or some mixture of the two. So in the above examples, assuming one user (or group of users sharing the same account code) submit a 2 hour job to all three partitions, their one, single account code should be charged: - 2 units for the generic CPU partition - 4 units for the job on the low latency partition - 8 units for the gpu partition. - A total of 14 credits removed from their single account code Is this feasible to achieve without having to allocate credits to each of the partitions for an account, or creating a QOS variant for each and every combination of account and partition? John Snowdon Senior Research Infrastructure Engineer (HPC) Research Software Engineering Catalyst Building, Room 2.01 Newcastle University 3 Science Square Newcastle Helix Newcastle upon Tyne NE4 5TG https://rse.ncldata.dev/

2 1

How do you guys track which GPU is used by which job ?
by Sylvain MARET 18 Oct '24

18 Oct '24

Hey guys ! I'm looking to improve GPU monitoring on our cluster. I want to install this https://github.com/NVIDIA/dcgm-exporter and saw in the README that it can support tracking of job id : https://github.com/NVIDIA/dcgm-exporter?tab=readme-ov-file#enabling-hpc-job… However I haven't been able to see any examples on how to do it nor does slurm seem to expose this information by default. Does anyone do this here ? And if so do you have any examples I could try to follow ? If you have advise on best practices to monitor GPU I'd be happy to hear it out ! Regards, Sylvain Maret

5 6

Job information is not being added to accounting database on new setup
by Adrian Brady 17 Oct '24

17 Oct '24

Hi Everyone, I'm a new to slurm administration and looking for a bit of help! Just added Accounting to an existing cluster but job information is not being added to the Accounting Mariadb. When I submit a test job it gets scheduled fine and its visible with squeue, I get nothing returned from sacct! I have turned up the logging to debug5 on both slurmctld and slurmdbd logs and can't see any errors. I believe all the comms are ok between slurmctld and slurmdbd as when I enter the sacct command I can see the database is being queried but returning nothing, because nothing has been added to the tables. The cluster tables were created fine when I ran #sacctmgr add cluster ny5ktt $ sacct JobID JobName Partition Account AllocCPUS State ExitCode ------------ ---------- ---------- ---------- ---------- ---------- -------- # tail -f slurmdbd.log [2024-10-17T12:34:45.232] debug: REQUEST_PERSIST_INIT: CLUSTER:ny5ktt VERSION:9216 UID:10001 IP:10.202.233.117 CONN:10 [2024-10-17T12:34:45.232] debug2: accounting_storage/as_mysql: acct_storage_p_get_connection: acct_storage_p_get_connection: request new connection 1 [2024-10-17T12:34:45.233] debug2: Attempting to connect to localhost:3306 [2024-10-17T12:34:45.274] debug2: DBD_GET_JOBS_COND: called [2024-10-17T12:34:45.317] debug2: DBD_FINI: CLOSE:1 COMMIT:0 [2024-10-17T12:34:45.317] debug4: accounting_storage/as_mysql: acct_storage_p_commit: got 0 commits The Mariadb is running on it own node with slurmdbd and munged for authentication. I haven't setup any accounts, users, asssociations or enforcements yet. On my lab cluster, jobs were visible in the database without these being setup. I guess I must be missing something simple in the config that is stopping jobs being reported to slurmdbd. Master Node packages # rpm -qa |grep slurm slurm-slurmdbd-20.11.9-1.el8.x86_64 slurm-libs-20.11.9-1.el8.x86_64 slurm-20.11.9-1.el8.x86_64 slurm-slurmd-20.11.9-1.el8.x86_64 slurm-perlapi-20.11.9-1.el8.x86_64 slurm-doc-20.11.9-1.el8.x86_64 slurm-contribs-20.11.9-1.el8.x86_64 slurm-slurmctld-20.11.9-1.el8.x86_64 Database Node packages # rpm -qa |grep slurm slurm-slurmdbd-20.11.9-1.el8.x86_64 slurm-20.11.9-1.el8.x86_64 slurm-libs-20.11.9-1.el8.x86_64 slurm-devel-20.11.9-1.el8.x86_64 slurm.conf # # See the slurm.conf man page for more information. # ClusterName=ny5ktt ControlMachine=ny5-pr-kttslurm-01 ControlAddr=10.202.233.71 #BackupController= #BackupAddr= # AuthType=auth/munge #CheckpointType=checkpoint/none CryptoType=crypto/munge #DisableRootJobs=NO #EnforcePartLimits=NO #Epilog= #EpilogSlurmctld= #FirstJobId=1 #MaxJobId=999999 #GresTypes= #GroupUpdateForce=0 #GroupUpdateTime=600 #JobCheckpointDir=/var/slurm/checkpoint #JobCredentialPrivateKey= #JobCredentialPublicCertificate= #JobFileAppend=0 #JobRequeue=1 #JobSubmitPlugins= #KillOnBadExit=0 #LaunchType=launch/slurm #Licenses=foo*4,bar MailProg=/bin/true MaxJobCount=200000 #MaxStepCount=40000 #MaxTasksPerNode=128 MpiDefault=none #MpiParams=ports=#-# #PluginDir= #PlugStackConfig= #PrivateData=jobs ProctrackType=proctrack/cgroup #Prolog= #PrologFlags= #PrologSlurmctld= #PropagatePrioProcess=0 #PropagateResourceLimits= #PropagateResourceLimitsExcept= #RebootProgram= ReturnToService=1 #SallocDefaultCommand= SlurmctldPidFile=/var/run/slurm/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurm/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurm/d SlurmUser=slurm #SlurmdUser=root #SrunEpilog= #SrunProlog= StateSaveLocation=/var/spool/slurm/ctld SwitchType=switch/none #TaskEpilog= TaskPlugin=task/none #TaskPluginParam= #TaskProlog= #TopologyPlugin=topology/tree #TmpFS=/tmp #TrackWCKey=no #TreeWidth= #UnkillableStepProgram= #UsePAM=0 # # # TIMERS #BatchStartTimeout=10 #CompleteWait=0 #EpilogMsgTime=2000 #GetEnvTimeout=2 #HealthCheckInterval=0 #HealthCheckProgram= InactiveLimit=0 KillWait=30 #MessageTimeout=10 #ResvOverRun=0 #MinJobAge=300 #MinJobAge=43200 # CHG0057915 MinJobAge=14400 # CHG0057915 #MaxJobCount=50000 #MaxJobCount=100000 #OverTimeLimit=0 SlurmctldTimeout=120 SlurmdTimeout=300 #UnkillableStepTimeout=60 #VSizeFactor=0 Waittime=0 # # # SCHEDULING DefMemPerCPU=3000 #FastSchedule=1 #MaxMemPerCPU=0 #SchedulerTimeSlice=30 SchedulerType=sched/backfill SelectType=select/cons_tres #SelectTypeParameters=CR_Core #SelectTypeParameters=CR_CPU SelectTypeParameters=CR_CPU_Memory # ECR CHG0056915 10/14/2023 MaxArraySize=5001 # # # JOB PRIORITY #PriorityFlags= #PriorityType=priority/basic #PriorityDecayHalfLife= #PriorityCalcPeriod= #PriorityFavorSmall= #PriorityMaxAge= #PriorityUsageResetPeriod= #PriorityWeightAge= #PriorityWeightFairshare= #PriorityWeightJobSize= #PriorityWeightPartition= #PriorityWeightQOS= # # # LOGGING AND ACCOUNTING #AccountingStorageEnforce=0 #AccountingStorageEnforce=limits AccountingStorageHost=ny5-pr-kttslurmdb-01.ktt.schonfeld.com #AccountingStorageLoc= #AccountingStoragePass= #AccountingStoragePort= #AccountingStorageType=accounting_storage/none AccountingStorageType=accounting_storage/slurmdbd #AccountingStorageUser= AccountingStoreJobComment=YES #DebugFlags= #JobCompHost= #JobCompLoc= #JobCompPass= #JobCompPort= JobCompType=jobcomp/none #JobCompUser= #JobContainerType=job_container/none JobAcctGatherFrequency=60 JobAcctGatherType=jobacct_gather/none SlurmctldDebug=info SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurm/slurmd.log #SlurmdLogFile= #SlurmSchedLogFile= #SlurmSchedLogLevel= # # # POWER SAVE SUPPORT FOR IDLE NODES (optional) #SuspendProgram= #ResumeProgram= #SuspendTimeout= #ResumeTimeout= #ResumeRate= #SuspendExcNodes= #SuspendExcParts= #SuspendRate= #SuspendTime= # # # COMPUTE NODES ##using fqdn since the ctld domain is different. Can't use regex since it's not at the end ##save 17 and 18 as headnodes #NodeName=ny5-dv-kttres-17 Sockets=1 CoresPerSocket=18 ThreadsPerCore=2 Feature=HyperThread RealMemory=102400 #NodeName=ny5-dv-kttres-18 Sockets=1 CoresPerSocket=14 ThreadsPerCore=2 Feature=HyperThread RealMemory=102400 NodeName=ny5-dv-kttres-19 Sockets=1 CoresPerSocket=12 ThreadsPerCore=2 Feature=HyperThread RealMemory=102400 NodeName=ny5-dv-kttres-[20-21] Sockets=1 CoresPerSocket=18 ThreadsPerCore=2 Feature=HyperThread RealMemory=102400 NodeName=ny5-dv-kttres-[01-16] Sockets=2 CoresPerSocket=16 ThreadsPerCore=2 Feature=HyperThread RealMemory=233472 NodeName=ny5-dv-kttres-[22-35] Sockets=2 CoresPerSocket=32 ThreadsPerCore=2 Feature=HyperThread RealMemory=346884 PartitionName=ktt_slurm_light_1 Nodes=ny5-dv-kttres-[19-21] Default=NO MaxTime=INFINITE State=UP OverSubscribe=FORCE:2 PartitionName=ktt_slurm_medium_1 Nodes=ny5-dv-kttres-[01-08] Default=NO MaxTime=INFINITE State=UP OverSubscribe=FORCE:2 PartitionName=ktt_slurm_medium_2 Nodes=ny5-dv-kttres-[09-16] Default=NO MaxTime=INFINITE State=UP OverSubscribe=FORCE:2 PartitionName=ktt_slurm_medium_3 Nodes=ny5-dv-kttres-[22-28] Default=NO MaxTime=INFINITE State=UP OverSubscribe=FORCE:2 PartitionName=ktt_slurm_medium_4 Nodes=ny5-dv-kttres-[29-35] Default=NO MaxTime=INFINITE State=UP OverSubscribe=FORCE:2 PartitionName=ktt_slurm_large_1 Nodes=ny5-dv-kttres-[01-16] Default=YES MaxTime=INFINITE State=UP OverSubscribe=FORCE:2 PartitionName=ktt_slurm_large_2 Nodes=ny5-dv-kttres-[22-35] Default=NO MaxTime=INFINITE State=UP OverSubscribe=FORCE:2 Slurmdbd.conf AuthType=auth/munge DbdAddr=10.202.233.72 DbdHost=ny5-pr-kttslurmdb-01 DebugLevel=debug5 LogFile=/var/log/slurm/slurmdbd.log PidFile=/tmp/slurmdbd.pid StorageType=accounting_storage/mysql StorageHost=localhost #StorageHost=10.234.132.57 StorageUser=slurm SlurmUser=slurm StoragePass=xxxxxxx #StorageUser=slurm #StorageLoc=slurm_acct_db Database tables MariaDB [slurm_acct_db]> show tables; +--------------------------------+ | Tables_in_slurm_acct_db | +--------------------------------+ | acct_coord_table | | acct_table | | clus_res_table | | cluster_table | | convert_version_table | | federation_table | | ny5ktt_assoc_table | | ny5ktt_assoc_usage_day_table | | ny5ktt_assoc_usage_hour_table | | ny5ktt_assoc_usage_month_table | | ny5ktt_event_table | | ny5ktt_job_table | | ny5ktt_last_ran_table | | ny5ktt_resv_table | | ny5ktt_step_table | | ny5ktt_suspend_table | | ny5ktt_usage_day_table | | ny5ktt_usage_hour_table | | ny5ktt_usage_month_table | | ny5ktt_wckey_table | | ny5ktt_wckey_usage_day_table | | ny5ktt_wckey_usage_hour_table | | ny5ktt_wckey_usage_month_table | | qos_table | | res_table | | table_defs_table | | tres_table | | txn_table | | user_table | +--------------------------------+ Many Thanks Adrian Disclaimer Schonfeld Strategic Advisors (UK) LLP (“SSA UK”) is authorised and regulated by The Financial Conduct Authority. SSA UK is a limited liability partnership in England and Wales (No: OC420598) and its registered office is at 54 Jermyn Street, London, SW1Y 6LX. The contents of this message, including any attachments, are meant solely for the intended recipient and may be confidential, privileged, or otherwise protected from disclosure. If you receive this message in error, immediately alert the sender by reply e-mail, delete it and any attachments or copies from your systems, and do not read, disclose, distribute, or otherwise use the information contained herein. We do not waive any confidentiality or privilege if this message was misdirected. This e-mail does not constitute an offer to sell or a solicitation to buy any securities or an offer of any investment advisory services. If you reply to this email please note that we invest in securities and do not want to receive material, non-public information and you are instructed not to communicate any such information to us. We do not agree to keep confidential any information you provide nor restrict our trading activity, except as agreed pursuant to a written confidentiality agreement duly executed by us. We reserve the right to monitor and review the content of all messages sent to or from this e-mail address.

1 0

Issue with interactive jobs
by Nerjes, Onno 17 Oct '24

17 Oct '24

Dear all, we've set up SLURM 24.05.3 on our cluster and are experiencing an issue with interactive jobs. Before, we used 21.08 and pretty much the same settings, but without these issues. We've started with a fresh DB etc. The behavior of interactive jobs is very erratic. Sometimes they start absolutely fine, at other times they die silently in the background, while the user has to wait indefinitely. We have been unable to isolate certain users or nodes affected by this. On a given node, one user might be able to start an interactive job, while another user at the same time isn't able to. The day after, the situation might be the other way around. The exception are jobs that use a reservation. These start fine every time as far as we can tell. At the same time, the number of idle nodes does not seem to influence the behavior I described above. Failed allocation on the front end: [user1@login1 ~]$ salloc salloc: Pending job allocation 5052052 salloc: job 5052052 queued and waiting for resources The same job on the backend: 2024-10-14 11:41:57.680 slurmctld: _job_complete: JobId=5052052 done 2024-10-14 11:41:57.678 slurmctld: _job_complete: JobId=5052052 WEXITSTATUS 1 2024-10-14 11:41:57.678 slurmctld: Killing interactive JobId=5052052: Communication connection failure 2024-10-14 11:41:46.666 slurmctld: sched/backfill: _start_job: Started JobId=5052052 in devel on m02n01 2024-10-14 11:41:30.096 slurmctld: sched: _slurm_rpc_allocate_resources JobId=5052052 NodeList=(null) usec=6258 Raising the debug level has not brought additional information. We were hoping, that one of you might be able to provide some insight into what the next steps in troubleshooting might be. Best regards, Onno

1 0

Dependency jobs
by adam＠bramblecfd.com 17 Oct '24

17 Oct '24

Hi Slurm Users, I am trying to figure out if there is a way you can check if a running job has any jobs queued up after it that depend on the current running job, I know you can show job info and find what dependency a job is waiting for, But more after checking if there are jobs waiting on the current job to complete using the job ID, Many Thanks, Adam

3 2

AllowAccounts partition setting
by Marko Markoc 16 Oct '24

16 Oct '24

Hi All, At one point, when we were running slurm 22, we set up a few new partitions with `AllowAccounts` setting enabled. We limit access to these partitions only to a few accounts. We manually create these accounts and users using the `sacctmgr` command. At that time, everything was working. Only users in these accounts were able to run jobs in that partition. Few months ago we upgraded slurm to 23.02.7 but just a couple of days ago we noticed that anyone could submit jobs to these partitions. In addition to that, you can provide any value to the `--account` directive and slurm will accept and record it in the job accounting database. In our initial testing, you could only provide `--account` value that has been previously created. I'm still looking into this but I wanted to check if there was any change in version 23 that could have impacted this? I don't remember seeing anything in release notes. Thanks, Marko

6 8

Problem with nodes with 1 gpu
by Jörg Striewski 16 Oct '24

16 Oct '24

i cannot send jobs to nodes with one gpu, i don't find the bug in my configuration. can someone help me ? in slurm.conf GresTypes=gpu is set this are some nodes in slurm.conf NodeName=gpu-[001-003] CPUs=8 SocketsPerBoard=1 CoresPerSocket=4 RealMemory=31000 Gres=gpu:1080:1 NodeName=gpu-[010-019] CPUs=16 SocketsPerBoard=1 CoresPerSocket=8 RealMemory=64000 Gres=gpu:1080:2 the partition for this gpu nodes is # General GPU partitions PartitionName=GPU Nodes=gpu-[001-003,010-019] AllowAccounts=staff PreemptMode=REQUEUE PriorityTier=0 DefMemPerGPU=32000 DefCpuPerGPU=8 CpuBind=none TRESBillingWeights="GRES/gpu=1000" GraceTime=300 this are the entries for some nodes in gres.conf NodeName=gpu-[001-003] Name=gpu Type=1080 File=/dev/nvidia0 NodeName=gpu-[010-019] Name=gpu Type=1080 File=/dev/nvidia[0-1] when i send a job with sbatch to gpu-001 #SBATCH --job-name=hello #SBATCH --ntasks-per-node=1 #SBATCH --output=hello_%A.out #SBATCH --time=00:10:00 #SBATCH --mail-type=ALL #SBATCH --mail-user=striewski(a)ismll.de #SBATCH --partition=GPU #SBATCH --nodelist=gpu-001 #SBATCH --gres=gpu:1 [...] i get the error sbatch: error: Batch job submission failed: Requested node configuration is not available when i send the job to a node with 2 gpu's it runs with no error, just setting --nodelist=gpu-12 has someone a hint what i made wrong ? Mit freundlichen Grüßen / kind regards -- Jörg Striewski Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim Germany post address: Universitätsplatz 1, D-31141Hildesheim, Germany visitor address: Samelsonplatz 1, D-31141 Hildesheim,Germany Tel.(+49) 05121 / 883-40392 http://www.ismll.uni-hildesheim.de

2 2

unsuscribe
by LEROY Christine 208562 15 Oct '24

15 Oct '24

unsuscribe

1 0

How does --nodes=min[-max] determine number of nodes to allocate?
by Declan Valters 08 Oct '24

08 Oct '24

Hi, I am trying to understand how Slurm determines the actual number of nodes to allocate to a job specified with -nodes=min[-max] I've been trying to follow the source code through on Github - could anyone point me in the right area(s)/source files/functions to look at to understand the program logic behind this. Many thanks in advance, Declan The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th' ann an Oilthigh Dh?n ?ideann, cl?raichte an Alba, ?ireamh cl?raidh SC005336.

2 1

Jobs not getting scheduled, no priority calculation, but still in queue?
by Cutts, Tim 07 Oct '24

07 Oct '24

Something odd is going on on our cluster. User has a lot of pending jobs in a job array (a few thousand). squeue -u kmnx005 -r -t PD | head -5 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 3045324_875 core run_scp_ kmnx005 PD 0:00 1 (JobArrayTaskLimit) 3045324_876 core run_scp_ kmnx005 PD 0:00 1 (JobArrayTaskLimit) 3045324_877 core run_scp_ kmnx005 PD 0:00 1 (JobArrayTaskLimit) 3045324_878 core run_scp_ kmnx005 PD 0:00 1 (JobArrayTaskLimit) None are getting scheduled. But when I ask SLURM what that job’s priority is, it produces no output: $ sprio -j 3045324 JOBID PARTITION PRIORITY SITE AGE FAIRSHARE JOBSIZE PARTITION QOS TRES Any clues what’s going on here? -- Tim Cutts Scientific Computing Platform Lead AstraZeneca Find out more about R&D IT Data, Analytics & AI and how we can support you by visiting our Service Catalogue<https://azcollaboration.sharepoint.com/sites/CMU993> | ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com<https://www.astrazeneca.com>

2 3

SLURM GRES reservation not working properly on 24.05.1
by Minulakshmi S 07 Oct '24

07 Oct '24

Hello, *Issue 1:* I am using slurm version 24.05.1 , my slurmd has a single node where I connect multiple gres by enabling the overscribe feature. I am able to use the advance reservation of gres only using *gres** name* (tres=gres/gpu:*SYSTEM12*). i.e while in reservation period , if other users submits job with SYSTEM12 , then slurm places this job in queue *user1@host$ srun --gres=gpu:SYSTEM12:1 hostname* *srun: job 333 queued and waiting for resources * but when other users just submit a job without any system name , slurm jobs goes through on that gres immediately even though it is reserved. *user1@host$ srun --gres=gpu:1 hostname * *mylinux.wbi.com <http://mylinux.wbi.com/> * Also I can see GresUsed in busy mode using "*scontrol show node -d*" , this means the job is running on Gres/GPU and not on cpu etc. Same way , job submission based on Feature "rev1 in my case" is also going through even though it is reserved for other users in multiple partition slurm. *snippet of slurm.conf file* NodeName=cluster01 NodeAddr=cluster Port=6002CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2 Feature="rev1" Gres=gpu:SYSTEM12:1 RealMemory=64171 State=IDLE *Issue 2:* while execution , Slurm o/p's some extra prints in the srun output user1@host$ srun --gres=gpu:1 hostname srun: error: extract_net_cred: net_cred not provided srun: error: Malformed RPC of type RESPONSE_NODE_ALIAS_ADDRS(3017) received srun: error: slurm_unpack_received_msg: [mylinux.wbi.com]:41242] Header lengths are longer than data received *mylinux.wbi.com <http://mylinux.wbi.com/>* Regards, MS

1 1

Slurmctld process error 'double free or corruption' on RHEL 9 (Rocky Linux)
by William VINCENT 07 Oct '24

07 Oct '24

Hello I am writing to report an issue with the Slurmctld process on our RHEL 9 (Rocky Linux) . Twice in the past 5 days, the Slurmctld process has encountered an error that resulted in the service stopping. The error message displayed was "double free or corruption (out)". This error has caused significant disruption to our jobs, and we are concerned about its recurrence. We have tried troubleshooting the issue, but we have not been able to identify the root cause of the problem. We would appreciate any assistance or guidance you can provide to help us resolve this issue. Please let us know if you need any additional information or if there are any specific steps we should take to diagnose the problem further. Thank you for your attention to this matter. Best regards, _________________________ Jul 09 22:12:01 admin slurmctld[711010]: double free or corruption (fasttop) Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Main process exited, code=killed, status=6/ABRT Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Failed with result 'signal'. Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Consumed 11min 26.451s CPU time. ..... Jul 14 10:15:01 admin slurmctld[1633720]: double free or corruption (out) Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Main process exited, code=killed, status=6/ABRT Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Failed with result 'signal'. Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Consumed 7min 27.596s CPU time. _________________________ slurmctld -V slurm 22.05.9 ________________________ cat /etc/slurm/slurm.conf |grep -v '#' ClusterName=xxx SlurmctldHost=admin SlurmctldParameters=enable_configless SlurmUser=slurm AuthType=auth/munge CryptoType=crypto/munge SlurmctldPort=6817 StateSaveLocation=/var/spool/slurmctld SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmctldDebug=verbose DebugFlags=NO_CONF_HASH SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmdLogFile=/var/log/slurm/slurmd.log SlurmdDebug=verbose SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core,CR_LLN DefMemPerCPU=1024 MaxMemPerCPU=4096 GresTypes=gpu ProctrackType=proctrack/cgroup JobAcctGatherType=jobacct_gather/cgroup JobAcctGatherFrequency=15 JobCompType=jobcomp/none TaskPlugin=task/cgroup LaunchParameters=use_interactive_step AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=admin AccountingStoragePort=6819 AccountingStorageEnforce=associations AccountingStorageTRES=gres/gpu MailProg=/usr/bin/mailx EnforcePartLimits=YES MaxArraySize=200000 MaxJobCount=500000 MpiDefault=none ReturnToService=2 SwitchType=switch/none TmpFS=/tmpslurm/ UsePAM=1 InactiveLimit=0 KillWait=30 MessageTimeout=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 PriorityType=priority/multifactor PriorityFlags=FAIR_TREE,MAX_TRES PriorityDecayHalfLife=1-0 PriorityWeightFairshare=10000 NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=3500 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=2 Sockets=2 RealMemory=1700 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=1700 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=3500 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=r9nc-24-[1-12] NodeHostname=r9nc-24-[1-12] Sockets=2 CoresPerSocket=12 ThreadsPerCore=1 CPUs=24 RealMemory=180000 State=UNKNOWN NodeName=r9nc-48-[1-4] NodeHostname=r9nc-48-[1-4] Sockets=2 CoresPerSocket=24 ThreadsPerCore=1 CPUs=48 RealMemory=480000 State=UNKNOWN NodeName=r9ng-1080-[1-7] NodeHostname=r9ng-1080-[1-7] Sockets=2 CoresPerSocket=10 ThreadsPerCore=1 CPUs=20 RealMemory=180000 State=UNKNOWN Gres=gpu:1080ti:4 NodeName=r9ng-1080-8 NodeHostname=r9ng-1080-8 Sockets=2 CoresPerSocket=10 ThreadsPerCore=1 CPUs=20 RealMemory=176687 State=UNKNOWN Gres=gpu:1080ti:1 PartitionName=24CPUNodes Nodes=r9nc-24-[1-12] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=7500 DefMemPerCPU=7500 TRESBillingWeights="CPU=1.0,Mem=0.125G" Default=YES PartitionName=48CPUNodes Nodes=r9nc-48-[1-4] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=10000 DefMemPerCPU=8000 TRESBillingWeights="CPU=1.0,Mem=0.125G" PartitionName=GPUNodes Nodes=r9ng-1080-[1-7] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=9000 DefMemPerCPU=9000 PartitionName=GPUNodes1080-dev Nodes=r9ng-1080-8 State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=9000 DefMemPerCPU=9000 Hidden=Yes _________________________ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST 24CPUNodes* up infinite 12 idle r9nc-24-[1-12] 48CPUNodes up infinite 2 idle r9nc-48-[1-2] GPUNodes up infinite 4 idle r9ng-1080-[4-7] GPUNodes1080-dev up infinite 1 idle r9ng-1080-8 -- William VINCENT Administrateur systèmes et réseaux

4 15

license server redundancy
by Yves Kondoszek 03 Oct '24

03 Oct '24

Hello. I'm configuring a cluster that will run jobs that use FlexLM licenses. We have what I believe is a quite standard configuration of license servers - though I haven't found anything in the documentation or in the mailing list archives - that often provide the same features across several servers; for example, the license token "simulation" will be available at let's say 1717@server1, 1717@server2 and 4000@server3 . FlexLM clients support those through a path-style environment variable, typically LM_LICENSE_FILE=1717@server1:1717@server2:4000@server3 . Now, I have set up Slurm for remote dynamic licenses and update them live through "LastConsumed" from lmstat requests, so that Slurm is aware of some users checking them out externally (out of Slurm control). But my problem is that the documentation clearly states that: "When submitting jobs to remote licenses, the name and server must be used." So I have to specifically choose a server: $ sbatch -L simulation@server2 script.sh instead of telling Slurm to just use whatever license is available on any of the servers. Does any of you have any experience with redundant FlexLM licenses through several servers? Is there a way Slurm could support this?

1 0

GPU Accounting
by Emyr James 03 Oct '24

03 Oct '24

We have a node with 8 H100 GPUs that are split into MIG instances. We are using cgroups. This seems to work fine. Users can do something like sbatch --gres="gpu:1g.10gb:1"... and the job starts on the node with the gpus and cuda visible devices and the pytorch debug shows that the cgroup only gives them the gpu they asked for. In the accounting database, jobs in the job table always have the "gres_used" column be empty. I'd expect to see "gpu:1g.10gb:1" appearing for the job above. I have this set in slurm.conf AccountingStorageTRES=gres/gpu How can I see what gres was requested with the job ? At the moment I only see something like this in AllocTres billing=1,cpu=1,gres/gpu=1,mem=8G,node=1 and can't see any way to see what the specific MIG gpu asked for was. This is related to the email from Richard Lefebvre dated 7th June 2023 entitled "Billing/accounting for MIGs is not working". As far as I can see this got no replies. We are running slurm version 23.11.6. Regards, Emyr James Head of Scientific IT CRG - Centre for Genomic Regulation

2 1

SLUG Presentations Now Online!
by Victoria Hobson 01 Oct '24

01 Oct '24

Available presentations from this year's SLUG event are now online. They can be found at https://www.schedmd.com/publications/ We thank all those who presented and attended for a great event! -- Victoria Hobson SchedMD LLC Vice President of Marketing

1 0

Job Step State
by Emyr James 01 Oct '24

01 Oct '24

Dear all, I am working on a script to take completed job accounting data from the slurm accounting database and insert the equivalent data into a clickhouse table for fast reporting I can see that all the information is included in the cluster_job_table and cluster_job_step_table which seem to be joined on job_db_inx To get the cpu usage and peak memory usage etc. I can see that I need to parse the tres columns in the job steps. I couldn't find any column called MaxRSS in the database even though the sacct command prints this. I then found some data in tres_table and assume that sacct is using this. Please correct me if I'm wrong and if sacct is getting information from somwhere other than the accounting database? for the state column I get this... select state, count(*) as num from crg_step_table group by state order by num desc limit 10; +-------+--------+ | state | num | +-------+--------+ | 3 | 590635 | | 5 | 28345 | | 4 | 4401 | | 11 | 962 | | 1 | 8 | +-------+--------+ When I use sacct I see statuses seach as COMPLETED, OUT_OF_MEMORY etc. so there must be a mapping somewhere between these state ids and that text. Can someone prvide that mapping or point me to where it's defined in the database or in the code ? Many thanks, Emyr James Head of Scientific IT CRG - Centre for Genomic Regulation

2 2

Hardcoded CGroups v2 Slice
by Khalid Al-Hawaj 01 Oct '24

01 Oct '24

Hello, I am in the process of setting up SLURM to be used in a profiling cluster. The purpose of SLURM is to allow users to submit jobs to be profiled. The latency is a very important aspect of profiling the applications correctly. I was able to leverage cgroupsv2.0 to isolate user.slice from the cores that would be used by SLURM jobs. The issue is that slurmstepd shares the resources with system.slice; I was digging through the code, and I saw that the creation of the scope is here: https://github.com/SchedMD/slurm/blob/master/src/plugins/cgroup/v2/cgroup_v… And I noticed that the slice is hardcoded in the following line: https://github.com/SchedMD/slurm/blob/master/src/plugins/cgroup/v2/cgroup_v… So, my question, now, is about why is the slice hardcoded? What was the reason behind such a decision? I would have thought that the slice chosen would be set through cgroups.conf, instead. I would like to switch the slice for slurmstepd to a slice other than system.slice; by doing so, I would be able to isolate cores better by making sure that services' processes are isolated from the cores used for SLURM jobs. I can definitely change the defined value in the code and recompile. Are there anything to consider before doing so? Thanks, Khalid

1 0

Updated one compute node to Ubuntu 24.04 LTS, now it does not receive jobs
by Cristóbal Navarro 29 Sep '24

29 Sep '24

Dear community, I am having a strange issue I have been unable to find the cause. Last week I did a full update on the cluster, which is composed of the master node, and two compute nodes (nodeGPU01 -> DGXA100 and nodeGPU02 -> custom GPU server). After the update, I got - master node ended up with Ubuntu 24.04, - nodeGPU01 with latest DGX OS (still Ubuntu 22.04) - nodeGPU02 with Ubuntu 24.04 LTS. - Launching jobs from master choosing the partitions of nodeGPU01 works perfectly. - Launching jobs from master choosing the partition of nodeGPU02 stopped working (hangs). The nodeGPU02 (Ubuntu 24) is no longer processing jobs successfully, while the other nodeGPU01 works perfectly even when the master has Ubuntu 24. Any help is welcome, I have tried many things and had no success in finding the cause of this. Please let me know if you need more information. Many thanks in advance. This is the initial `slurmd` log of the problematic node (nodeGPU02), notice the messages in yellow ➜ ~ sudo systemctl status slurmd.service ● slurmd.service - Slurm node daemon Loaded: loaded (/etc/systemd/system/slurmd.service; enabled; preset: enabled) Active: active (running) since Sat 2024-09-28 14:00:22 -03; 4s ago Main PID: 4821 (slurmd) Tasks: 1 Memory: 17.0M (peak: 29.7M) CPU: 174ms CGroup: /system.slice/slurmd.service └─4821 /usr/sbin/slurmd -D -s Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug: MPI: Loading all types Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug: mpi/pmix_v5: init: PMIx plugin loaded Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug: mpi/pmix_v5: init: PMIx plugin loaded Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug2: No mpi.conf file (/etc/slurm/mpi.conf) Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: slurmd started on Sat, 28 Sep 2024 14:00:25 -0300 Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug: _step_connect: connect() failed for /var/spool/slurmd/slurmd/nodeGPU02_57436.0: Connection refused Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug2: health_check success rc:0 output: Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: CPUs=128 Boards=1 Sockets=2 Cores=64 Threads=1 Memory=773744 TmpDisk=899181 Uptime=2829 CPUSpecList=(null) FeaturesAvail=(nu> Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug: _step_connect: connect() failed for /var/spool/slurmd/slurmd/nodeGPU02_57436.0: Connection refused Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug: _handle_node_reg_resp: slurmctld sent back 11 TRES This is the verbose output of the srun command (notice yellow messages). ➜ ~ srun -vvvp rtx hostname srun: defined options srun: -------------------- -------------------- srun: partition : rtx srun: verbose : 3 srun: -------------------- -------------------- srun: end of defined options srun: debug: propagating RLIMIT_CPU=18446744073709551615 srun: debug: propagating RLIMIT_FSIZE=18446744073709551615 srun: debug: propagating RLIMIT_DATA=18446744073709551615 srun: debug: propagating RLIMIT_STACK=8388608 srun: debug: propagating RLIMIT_CORE=0 srun: debug: propagating RLIMIT_RSS=18446744073709551615 srun: debug: propagating RLIMIT_NPROC=3090276 srun: debug: propagating RLIMIT_NOFILE=1024 srun: debug: propagating RLIMIT_MEMLOCK=18446744073709551615 srun: debug: propagating RLIMIT_AS=18446744073709551615 srun: debug: propagating SLURM_PRIO_PROCESS=0 srun: debug: propagating UMASK=0002 srun: debug: Entering slurm_allocation_msg_thr_create() srun: debug: port from net_stream_listen is 34081 srun: debug: Entering _msg_thr_internal srun: Waiting for resource configuration srun: Nodes nodeGPU02 are ready for job srun: jobid 57463: nodes(1):`nodeGPU02', cpu counts: 1(x1) srun: debug2: creating job with 1 tasks srun: debug2: cpu:1 is not a gres: srun: debug: requesting job 57463, user 99, nodes 1 including ((null)) srun: debug: cpus 1, tasks 1, name hostname, relative 65534 srun: CpuBindType=(null type) srun: debug: Entering slurm_step_launch srun: debug: mpi/pmix_v4: pmixp_abort_agent_start: (null) [0]: pmixp_agent.c:382: Abort agent port: 41393 srun: debug: mpi/pmix_v4: mpi_p_client_prelaunch: (null) [0]: mpi_pmix.c:285: setup process mapping in srun srun: debug: Entering _msg_thr_create() srun: debug: mpi/pmix_v4: _pmix_abort_thread: (null) [0]: pmixp_agent.c:353: Start abort thread srun: debug: initialized stdio listening socket, port 33223 srun: debug: Started IO server thread (140079189182144) srun: debug: Entering _launch_tasks srun: launching StepId=57463.0 on host nodeGPU02, 1 tasks: 0 srun: debug2: Called _file_readable srun: debug2: Called _file_writable srun: route/default: init: route default plugin loaded srun: debug2: Called _file_writable srun: topology/none: init: topology NONE plugin loaded srun: debug2: Tree head got back 0 looking for 1 srun: debug: slurm_recv_timeout at 0 of 4, timeout srun: error: slurm_receive_msgs: [[nodeGPU02]:6818] failed: Socket timed out on send/recv operation srun: debug2: Tree head got back 1 srun: debug: launch returned msg_rc=1001 err=5004 type=9001 srun: debug2: marking task 0 done on failed node 0 srun: error: Task launch for StepId=57463.0 failed on node nodeGPU02: Socket timed out on send/recv operation srun: error: Application launch failed: Socket timed out on send/recv operation srun: Job step aborted srun: debug2: false, shutdown srun: debug2: false, shutdown srun: debug2: Called _file_readable srun: debug2: Called _file_writable srun: debug2: Called _file_writable srun: debug2: false, shutdown srun: debug: IO thread exiting srun: debug: mpi/pmix_v4: _conn_readable: (null) [0]: pmixp_agent.c:105: false, shutdown srun: debug: mpi/pmix_v4: _pmix_abort_thread: (null) [0]: pmixp_agent.c:355: Abort thread exit srun: debug2: slurm_allocation_msg_thr_destroy: clearing up message thread srun: debug2: false, shutdown srun: debug: Leaving _msg_thr_internal srun: debug2: spank: spank_pyxis.so: exit = 0 This is the `tail -f` log of slurmctld when launching a simple `srun hostname` [2024-09-28T14:08:10.264] ==================== [2024-09-28T14:08:10.264] JobId=57463 nhosts:1 ncpus:1 node_req:1 nodes=nodeGPU02 [2024-09-28T14:08:10.264] Node[0]: [2024-09-28T14:08:10.264] Mem(MB):65536:0 Sockets:2 Cores:64 CPUs:1:0 [2024-09-28T14:08:10.264] Socket[0] Core[0] is allocated [2024-09-28T14:08:10.264] -------------------- [2024-09-28T14:08:10.264] cpu_array_value[0]:1 reps:1 [2024-09-28T14:08:10.264] ==================== [2024-09-28T14:08:10.264] gres/gpu: state for nodeGPU02 [2024-09-28T14:08:10.264] gres_cnt found:3 configured:3 avail:3 alloc:0 [2024-09-28T14:08:10.264] gres_bit_alloc: of 3 [2024-09-28T14:08:10.264] gres_used:(null) [2024-09-28T14:08:10.264] topo[0]:(null)(0) [2024-09-28T14:08:10.264] topo_core_bitmap[0]:0-63 of 128 [2024-09-28T14:08:10.264] topo_gres_bitmap[0]:0 of 3 [2024-09-28T14:08:10.264] topo_gres_cnt_alloc[0]:0 [2024-09-28T14:08:10.264] topo_gres_cnt_avail[0]:1 [2024-09-28T14:08:10.264] topo[1]:(null)(0) [2024-09-28T14:08:10.264] topo_core_bitmap[1]:0-63 of 128 [2024-09-28T14:08:10.264] topo_gres_bitmap[1]:1 of 3 [2024-09-28T14:08:10.264] topo_gres_cnt_alloc[1]:0 [2024-09-28T14:08:10.264] topo_gres_cnt_avail[1]:1 [2024-09-28T14:08:10.264] topo[2]:(null)(0) [2024-09-28T14:08:10.264] topo_core_bitmap[2]:0-63 of 128 [2024-09-28T14:08:10.264] topo_gres_bitmap[2]:2 of 3 [2024-09-28T14:08:10.264] topo_gres_cnt_alloc[2]:0 [2024-09-28T14:08:10.264] topo_gres_cnt_avail[2]:1 [2024-09-28T14:08:10.265] sched: _slurm_rpc_allocate_resources JobId=57463 NodeList=nodeGPU02 usec=1339 [2024-09-28T14:08:10.368] ==================== [2024-09-28T14:08:10.368] JobId=57463 StepId=0 [2024-09-28T14:08:10.368] JobNode[0] Socket[0] Core[0] is allocated [2024-09-28T14:08:10.368] ==================== [2024-09-28T14:08:30.409] _job_complete: JobId=57463 WTERMSIG 12 [2024-09-28T14:08:30.410] gres/gpu: state for nodeGPU02 [2024-09-28T14:08:30.410] gres_cnt found:3 configured:3 avail:3 alloc:0 [2024-09-28T14:08:30.410] gres_bit_alloc: of 3 [2024-09-28T14:08:30.410] gres_used:(null) [2024-09-28T14:08:30.410] topo[0]:(null)(0) [2024-09-28T14:08:30.410] topo_core_bitmap[0]:0-63 of 128 [2024-09-28T14:08:30.410] topo_gres_bitmap[0]:0 of 3 [2024-09-28T14:08:30.410] topo_gres_cnt_alloc[0]:0 [2024-09-28T14:08:30.410] topo_gres_cnt_avail[0]:1 [2024-09-28T14:08:30.410] topo[1]:(null)(0) [2024-09-28T14:08:30.410] topo_core_bitmap[1]:0-63 of 128 [2024-09-28T14:08:30.410] topo_gres_bitmap[1]:1 of 3 [2024-09-28T14:08:30.410] topo_gres_cnt_alloc[1]:0 [2024-09-28T14:08:30.410] topo_gres_cnt_avail[1]:1 [2024-09-28T14:08:30.410] topo[2]:(null)(0) [2024-09-28T14:08:30.410] topo_core_bitmap[2]:0-63 of 128 [2024-09-28T14:08:30.410] topo_gres_bitmap[2]:2 of 3 [2024-09-28T14:08:30.410] topo_gres_cnt_alloc[2]:0 [2024-09-28T14:08:30.410] topo_gres_cnt_avail[2]:1 [2024-09-28T14:08:30.410] _job_complete: JobId=57463 done [2024-09-28T14:08:58.687] gres/gpu: state for nodeGPU01 [2024-09-28T14:08:58.687] gres_cnt found:8 configured:8 avail:8 alloc:0 [2024-09-28T14:08:58.687] gres_bit_alloc: of 8 [2024-09-28T14:08:58.687] gres_used:(null) [2024-09-28T14:08:58.687] topo[0]:A100(808464705) [2024-09-28T14:08:58.687] topo_core_bitmap[0]:48-63 of 128 [2024-09-28T14:08:58.687] topo_gres_bitmap[0]:0 of 8 [2024-09-28T14:08:58.687] topo_gres_cnt_alloc[0]:0 [2024-09-28T14:08:58.687] topo_gres_cnt_avail[0]:1 [2024-09-28T14:08:58.687] topo[1]:A100(808464705) [2024-09-28T14:08:58.687] topo_core_bitmap[1]:48-63 of 128 [2024-09-28T14:08:58.687] topo_gres_bitmap[1]:1 of 8 [2024-09-28T14:08:58.687] topo_gres_cnt_alloc[1]:0 [2024-09-28T14:08:58.687] topo_gres_cnt_avail[1]:1 [2024-09-28T14:08:58.687] topo[2]:A100(808464705) [2024-09-28T14:08:58.687] topo_core_bitmap[2]:16-31 of 128 [2024-09-28T14:08:58.687] topo_gres_bitmap[2]:2 of 8 [2024-09-28T14:08:58.687] topo_gres_cnt_alloc[2]:0 [2024-09-28T14:08:58.687] topo_gres_cnt_avail[2]:1 [2024-09-28T14:08:58.687] topo[3]:A100(808464705) [2024-09-28T14:08:58.687] topo_core_bitmap[3]:16-31 of 128 [2024-09-28T14:08:58.688] topo_gres_bitmap[3]:3 of 8 [2024-09-28T14:08:58.688] topo_gres_cnt_alloc[3]:0 [2024-09-28T14:08:58.688] topo_gres_cnt_avail[3]:1 [2024-09-28T14:08:58.688] topo[4]:A100(808464705) [2024-09-28T14:08:58.688] topo_core_bitmap[4]:112-127 of 128 [2024-09-28T14:08:58.688] topo_gres_bitmap[4]:4 of 8 [2024-09-28T14:08:58.688] topo_gres_cnt_alloc[4]:0 [2024-09-28T14:08:58.688] topo_gres_cnt_avail[4]:1 [2024-09-28T14:08:58.688] topo[5]:A100(808464705) [2024-09-28T14:08:58.688] topo_core_bitmap[5]:112-127 of 128 [2024-09-28T14:08:58.688] topo_gres_bitmap[5]:5 of 8 [2024-09-28T14:08:58.688] topo_gres_cnt_alloc[5]:0 [2024-09-28T14:08:58.688] topo_gres_cnt_avail[5]:1 [2024-09-28T14:08:58.688] topo[6]:A100(808464705) [2024-09-28T14:08:58.688] topo_core_bitmap[6]:80-95 of 128 [2024-09-28T14:08:58.688] topo_gres_bitmap[6]:6 of 8 [2024-09-28T14:08:58.688] topo_gres_cnt_alloc[6]:0 [2024-09-28T14:08:58.688] topo_gres_cnt_avail[6]:1 [2024-09-28T14:08:58.688] topo[7]:A100(808464705) [2024-09-28T14:08:58.688] topo_core_bitmap[7]:80-95 of 128 [2024-09-28T14:08:58.688] topo_gres_bitmap[7]:7 of 8 [2024-09-28T14:08:58.688] topo_gres_cnt_alloc[7]:0 [2024-09-28T14:08:58.688] topo_gres_cnt_avail[7]:1 [2024-09-28T14:08:58.688] type[0]:A100(808464705) [2024-09-28T14:08:58.688] type_cnt_alloc[0]:0 [2024-09-28T14:08:58.688] type_cnt_avail[0]:8 [2024-09-28T14:08:58.690] gres/gpu: state for nodeGPU02 [2024-09-28T14:08:58.690] gres_cnt found:3 configured:3 avail:3 alloc:0 [2024-09-28T14:08:58.690] gres_bit_alloc: of 3 [2024-09-28T14:08:58.690] gres_used:(null) [2024-09-28T14:08:58.690] topo[0]:(null)(0) [2024-09-28T14:08:58.690] topo_core_bitmap[0]:0-63 of 128 [2024-09-28T14:08:58.690] topo_gres_bitmap[0]:0 of 3 [2024-09-28T14:08:58.690] topo_gres_cnt_alloc[0]:0 [2024-09-28T14:08:58.690] topo_gres_cnt_avail[0]:1 [2024-09-28T14:08:58.690] topo[1]:(null)(0) [2024-09-28T14:08:58.690] topo_core_bitmap[1]:0-63 of 128 [2024-09-28T14:08:58.690] topo_gres_bitmap[1]:1 of 3 [2024-09-28T14:08:58.690] topo_gres_cnt_alloc[1]:0 [2024-09-28T14:08:58.690] topo_gres_cnt_avail[1]:1 [2024-09-28T14:08:58.690] topo[2]:(null)(0) [2024-09-28T14:08:58.690] topo_core_bitmap[2]:0-63 of 128 [2024-09-28T14:08:58.690] topo_gres_bitmap[2]:2 of 3 [2024-09-28T14:08:58.690] topo_gres_cnt_alloc[2]:0 [2024-09-28T14:08:58.690] topo_gres_cnt_avail[2]:1 [2024-09-28T14:09:49.763] Resending TERMINATE_JOB request JobId=57463 Nodelist=nodeGPU02 This is the `tail -f` log of slurmd when launching the job from master, notice the messages in yellow [2024-09-28T14:08:10.270] debug2: Processing RPC: REQUEST_LAUNCH_PROLOG [2024-09-28T14:08:10.321] debug2: prep/script: _run_subpath_command: prolog success rc:0 output: [2024-09-28T14:08:10.323] debug2: Finish processing RPC: REQUEST_LAUNCH_PROLOG [2024-09-28T14:08:10.377] debug: Checking credential with 720 bytes of sig data [2024-09-28T14:08:10.377] debug2: Start processing RPC: REQUEST_LAUNCH_TASKS [2024-09-28T14:08:10.377] debug2: Processing RPC: REQUEST_LAUNCH_TASKS [2024-09-28T14:08:10.377] launch task StepId=57463.0 request from UID:10082 GID:10088 HOST:10.10.0.1 PORT:36478 [2024-09-28T14:08:10.377] CPU_BIND: JobNode[0] CPU[0] Step alloc [2024-09-28T14:08:10.377] CPU_BIND: ==================== [2024-09-28T14:08:10.377] CPU_BIND: Memory extracted from credential for StepId=57463.0 job_mem_limit=65536 step_mem_limit=65536 [2024-09-28T14:08:10.377] debug: Waiting for job 57463's prolog to complete [2024-09-28T14:08:10.377] debug: Finished wait for job 57463's prolog to complete [2024-09-28T14:08:10.378] error: _send_slurmstepd_init failed [2024-09-28T14:08:10.384] debug2: debug level read from slurmd is 'debug2'. [2024-09-28T14:08:10.385] debug2: _read_slurmd_conf_lite: slurmd sent 11 TRES. [2024-09-28T14:08:10.385] debug2: Received CPU frequency information for 128 CPUs [2024-09-28T14:08:10.385] select/cons_tres: common_init: select/cons_tres loaded [2024-09-28T14:08:10.385] debug: switch/none: init: switch NONE plugin loaded [2024-09-28T14:08:10.385] [57463.0] debug: auth/munge: init: loaded [2024-09-28T14:08:10.385] [57463.0] debug: Reading cgroup.conf file /etc/slurm/cgroup.conf [2024-09-28T14:08:10.395] [57463.0] debug: cgroup/v2: init: Cgroup v2 plugin loaded [2024-09-28T14:08:10.396] [57463.0] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded [2024-09-28T14:08:10.396] [57463.0] debug: acct_gather_energy/none: init: AcctGatherEnergy NONE plugin loaded [2024-09-28T14:08:10.396] [57463.0] debug: acct_gather_profile/none: init: AcctGatherProfile NONE plugin loaded [2024-09-28T14:08:10.396] [57463.0] debug: acct_gather_interconnect/none: init: AcctGatherInterconnect NONE plugin loaded [2024-09-28T14:08:10.396] [57463.0] debug: acct_gather_filesystem/none: init: AcctGatherFilesystem NONE plugin loaded [2024-09-28T14:08:10.396] [57463.0] debug2: Reading acct_gather.conf file /etc/slurm/acct_gather.conf [2024-09-28T14:08:10.396] [57463.0] debug2: hwloc_topology_init [2024-09-28T14:08:10.399] [57463.0] debug2: xcpuinfo_hwloc_topo_load: xml file (/var/spool/slurmd/slurmd/hwloc_topo_whole.xml) found [2024-09-28T14:08:10.400] [57463.0] debug: CPUs:128 Boards:1 Sockets:2 CoresPerSocket:64 ThreadsPerCore:1 [2024-09-28T14:08:10.401] [57463.0] debug: task/cgroup: init: core enforcement enabled [2024-09-28T14:08:10.401] [57463.0] debug: task/cgroup: task_cgroup_memory_init: task/cgroup/memory: TotCfgRealMem:773744M allowed:100%(enforced), swap:0%(enforced), max:100%(773744M) max+swap:0%(773744M) min:30M kmem:100%(773744M permissive) min:30M [2024-09-28T14:08:10.401] [57463.0] debug: task/cgroup: init: memory enforcement enabled [2024-09-28T14:08:10.401] [57463.0] debug: task/cgroup: init: device enforcement enabled [2024-09-28T14:08:10.401] [57463.0] debug: task/cgroup: init: Tasks containment cgroup plugin loaded [2024-09-28T14:08:10.401] [57463.0] debug: jobacct_gather/linux: init: Job accounting gather LINUX plugin loaded [2024-09-28T14:08:10.401] [57463.0] cred/munge: init: Munge credential signature plugin loaded [2024-09-28T14:08:10.401] [57463.0] debug: job_container/none: init: job_container none plugin loaded [2024-09-28T14:08:10.401] [57463.0] debug: gres/gpu: init: loaded [2024-09-28T14:08:10.401] [57463.0] debug: gpu/generic: init: init: GPU Generic plugin loaded [2024-09-28T14:08:30.415] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-09-28T14:08:30.415] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-09-28T14:08:30.415] debug: _rpc_terminate_job: uid = 777 JobId=57463 [2024-09-28T14:08:30.415] debug: credential for job 57463 revoked [2024-09-28T14:08:30.415] debug: sent SUCCESS, waiting for step to start [2024-09-28T14:08:30.415] debug: Blocked waiting for JobId=57463, all steps [2024-09-28T14:08:58.688] debug2: Start processing RPC: REQUEST_NODE_REGISTRATION_STATUS [2024-09-28T14:08:58.689] debug2: Processing RPC: REQUEST_NODE_REGISTRATION_STATUS [2024-09-28T14:08:58.689] debug: _step_connect: connect() failed for /var/spool/slurmd/slurmd/nodeGPU02_57436.0: Connection refused [2024-09-28T14:08:58.692] debug: _handle_node_reg_resp: slurmctld sent back 11 TRES. [2024-09-28T14:08:58.692] debug2: Finish processing RPC: REQUEST_NODE_REGISTRATION_STATUS -- Cristóbal A. Navarro

1 1

errors compiling Slurm 18 on RHEL 9: [Makefile:577: scancel] Error 1 & It's not recommended to have unversioned Obsoletes
by Robert Kudyba 27 Sep '24

27 Sep '24

We're in the process of upgrading but first we're moving to RHEL 9. My attempt to compile using rpmbuild -v -ta --define "_lto_cflags %{nil}" slurm-18.08.9.tar.bz2 (H/T to Brian for this flag <https://groups.google.com/g/slurm-users/c/W8YfGIn1rDI/m/4bsSAoqZAAAJ>). I've stumped Google and the Slurm mailing list with the scancel error so hoping someone here knows of a work around. /bin/ld: opt.o:/root/rpmbuild/BUILD/slurm-18.08.9/src/scancel/../../src/scancel/scancel.h:78: multiple definition of `opt'; scancel.o:/root/rpmbuild/BUILD/slurm-18.08.9/src/scancel/../../src/scancel/scancel.h:78: first defined here collect2: error: ld returned 1 exit status make[3]: *** [Makefile:577: scancel] Error 1 make[3]: Leaving directory '/root/rpmbuild/BUILD/slurm-18.08.9/src/scancel' make[2]: *** [Makefile:563: all-recursive] Error 1 make[2]: Leaving directory '/root/rpmbuild/BUILD/slurm-18.08.9/src' make[1]: *** [Makefile:690: all-recursive] Error 1 make[1]: Leaving directory '/root/rpmbuild/BUILD/slurm-18.08.9' make: *** [Makefile:589: all] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.jhiGyR (%build) RPM build errors: Macro expanded in comment on line 22: %_prefix path install path for commands, libraries, etc. line 70: It's not recommended to have unversioned Obsoletes: Obsoletes: slurm-lua slurm-munge slurm-plugins Macro expanded in comment on line 158: %define _unpackaged_files_terminate_build 0 line 224: It's not recommended to have unversioned Obsoletes: Obsoletes: slurm-sql line 256: It's not recommended to have unversioned Obsoletes: Obsoletes: slurm-sjobexit slurm-sjstat slurm-seff line 275: It's not recommended to have unversioned Obsoletes: Obsoletes: pam_slurm Bad exit status from /var/tmp/rpm-tmp.jhiGyR (%build) #!/bin/sh RPM_SOURCE_DIR="/root" RPM_BUILD_DIR="/root/rpmbuild/BUILD" RPM_OPT_FLAGS="-O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection" RPM_LD_FLAGS="-Wl,-z,relro -Wl,--as-needed "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 " RPM_ARCH="x86_64" RPM_OS="linux" RPM_BUILD_NCPUS="48" export RPM_SOURCE_DIR RPM_BUILD_DIR RPM_OPT_FLAGS RPM_LD_FLAGS RPM_ARCH RPM_OS RPM_BUILD_NCPUS RPM_LD_FLAGS RPM_DOC_DIR="/usr/share/doc" export RPM_DOC_DIR RPM_PACKAGE_NAME="slurm" RPM_PACKAGE_VERSION="18.08.9" RPM_PACKAGE_RELEASE="1.el9" export RPM_PACKAGE_NAME RPM_PACKAGE_VERSION RPM_PACKAGE_RELEASE LANG=C export LANG unset CDPATH DISPLAY ||: RPM_BUILD_ROOT="/root/rpmbuild/BUILDROOT/slurm-18.08.9-1.el9.x86_64" export RPM_BUILD_ROOT PKG_CONFIG_PATH="${PKG_CONFIG_PATH}:/usr/lib64/pkgconfig:/usr/share/pkgconfig" export PKG_CONFIG_PATH CONFIG_SITE=${CONFIG_SITE:-NONE} export CONFIG_SITE set -x umask 022 cd "/root/rpmbuild/BUILD" cd 'slurm-18.08.9' CFLAGS="${CFLAGS:--O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection}" ; export CFLAGS ; CXXFLAGS="${CXXFLAGS:--O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection}" ; export CXXFLAGS ; FFLAGS="${FFLAGS:--O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules}" ; export FFLAGS ; FCFLAGS="${FCFLAGS:--O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules}" ; export FCFLAGS ; LDFLAGS="${LDFLAGS:--Wl,-z,relro -Wl,--as-needed "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 }" ; export LDFLAGS ; LT_SYS_LIBRARY_PATH="${LT_SYS_LIBRARY_PATH:-/usr/lib64:}" ; export LT_SYS_LIBRARY_PATH ; CC="${CC:-gcc}" ; export CC ; CXX="${CXX:-g++}" ; export CXX; [ ""x != x ] && for file in $(find . -type f -name configure -print); do /usr/bin/sed -r --in-place=.backup 's/^char $\*f$  = /__attribute__ ((used)) char (*f) () = /g' $file; diff -u $file.backup $file && mv $file.backup $file /usr/bin/sed -r --in-place=.backup 's/^char $\*f$ ;/__attribute__ ((used)) char (*f) ();/g' $file; diff -u $file.backup $file && mv $file.backup $file /usr/bin/sed -r --in-place=.backup 's/^char \$2 ;/__attribute__ ((used)) char \$2 ();/g' $file; diff -u $file.backup $file && mv $file.backup $file /usr/bin/sed --in-place=.backup '1{$!N;$!N};$!N;s/int x = 1;\nint y = 0;\nint z;\nint nan;/volatile int x = 1; volatile int y = 0; volatile int z, nan;/;P;D' $file; diff -u $file.backup $file && mv $file.backup $file /usr/bin/sed --in-place=.backup 's#^lt_cv_sys_global_symbol_to_cdecl=.*#lt_cv_sys_global_symbol_to_cdecl="sed -n -e '"'"'s/^T .* \$.*\$$/extern int \\1();/p'"'"' -e '"'"'s/^$symcode* .* \$.*\$$/extern char \\1;/p'"'"'"#' $file; diff -u $file.backup $file && mv $file.backup $file done; [ "1" = 1 ] && for i in $(find $(dirname ./configure) -name config.guess -o -name config.sub) ; do [ -f /usr/lib/rpm/redhat/$(basename $i) ] && /usr/bin/rm -f $i && /usr/bin/cp -fv /usr/lib/rpm/redhat/$(basename $i) $i ; done ; [ "1" = 1 ] && [ x != "x"-Wl,-z,lazy"" ] && for i in $(find . -name ltmain.sh) ; do /usr/bin/sed -i.backup -e 's~compiler_flags=$~compiler_flags=""-Wl,-z,lazy""~' $i done ; ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu \ --program-prefix= \ --disable-dependency-tracking \ \ --prefix=/usr \ --exec-prefix=/usr \ --bindir=/usr/bin \ --sbindir=/usr/sbin \ --sysconfdir=/etc/slurm \ --datadir=/usr/share \ --includedir=/usr/include \ --libdir=/usr/lib64 \ --libexecdir=/usr/libexec \ --localstatedir=/var \ --sharedstatedir=/var/lib \ --mandir=/usr/share/man \ --infodir=/usr/share/info \ \ make -j48 RPM_EC=$? for pid in $(jobs -p); do kill -9 ${pid} || continue; done exit ${RPM_EC}

2 2

Re: SLUG'24 presentation slides?
by Kilian Cavalotti 27 Sep '24

27 Sep '24

Awesome, thanks Victoria! Cheers, -- Kilian On Thu, Sep 26, 2024 at 11:17 AM Victoria Hobson <victoria(a)schedmd.com> wrote: > Hi Kilian, > > We're getting these posted now and an email will go out when they are > available! > > Thanks, > > > Victoria Hobson > > *Vice President of Marketing * > > 909.609.8889 > > www.schedmd.com > > > On Mon, Sep 23, 2024 at 10:49 AM Kilian Cavalotti via slurm-users < > slurm-users(a)lists.schedmd.com> wrote: > >> Hi SchedMD, >> >> I'm sure they will eventually, but do you know when the slides of the >> SLUG'24 presentation will be available online at >> https://slurm.schedmd.com/publications.html, like previous editions'? >> >> Thanks! >> -- >> Kilian >> >> -- >> slurm-users mailing list -- slurm-users(a)lists.schedmd.com >> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com >> > -- Kilian

1 0

A note on updating Slurm from 23.02 to 24.05 & multi-cluster
by Ward Poelmans 26 Sep '24

26 Sep '24

Hi all, We hit a snag when updating our clusters from Slurm 23.02 to 24.05. After updating the slurmdbd, our multi cluster setup was broken until everything was updated to 24.05. We had not anticipated this. SchedMD says that fixing it would be a very complex operation. Hence, this warning to everybody on planning to update: make sure to quickly updating everything once you've updated the slurmdbd daemon. Reference: https://support.schedmd.com/show_bug.cgi?id=20931 Ward

3 3

Jobs pending with reason "priority" but nodes are idle
by Long, Daniel S. 25 Sep '24

25 Sep '24

Hi, On our cluster we have some jobs that are queued even though there are available nodes to run on. The listed reason is "priority" but that doesn't really make sense to me. Slurm isn't picking another job to run on those nodes; it's just not running anything at all. We do have a quite heterogeneous cluster, but as far as I can tell the queued jobs aren't requesting anything that would preclude them from running on the idle nodes. They are array jobs, if that makes a difference. Thanks for any help you all can provide.

3 6

Max TRES per user and node
by Guillaume COCHARD 25 Sep '24

25 Sep '24

Hello, We are looking for a method to limit the TRES used by each user on a per-node basis. For example, we would like to limit the total memory allocation of jobs from a user to 200G per node. There is MaxTRESperNode (https://slurm.schedmd.com/sacctmgr.html#OPT_MaxTRESPerNode), but unfortunately, this is a per-job limit, not per user. Ideally, we would like to apply this limit on partitions and/or QoS. Does anyone know if this is possible and how to achieve it? Thank you,

4 9

SLURM Telegraf Plugin
by Pablo Collado Soto 25 Sep '24

25 Sep '24

Hi all, I recently wrote an SLURM input plugin [0] for Telegraf [1]. I just wanted to let the community know so that you can use it if you'd find that useful. Maybe its existence can also be included in the documentation somewhere? Anyway, thanks a ton for your time, Pablo Collado Soto References: 0: https://github.com/influxdata/telegraf/tree/master/plugins/inputs/slurm 1: https://www.influxdata.com/time-series-platform/telegraf/ + -------------------------------------- + | Never let your sense of morals prevent | | you from doing what is right. | | -- Salvor Hardin, "Foundation" | + -------------------------------------- +

2 1

Setting up fairshare accounting
by tluchko 24 Sep '24

24 Sep '24

Hello, We have a new cluster and I'm trying to setup fairshare accounting. I'm trying to track CPU, MEM and GPU. It seems that billing for individual jobs is correct, but billing isn't being accumulated (TRESRunMin is always 0). In my slurm.conf, I think the relevant lines are AccountingStorageType=accounting_storage/slurmdbd AccountingStorageTRES=gres/gpu PriorityFlags=MAX_TRES PartitionName=gpu Nodes=node[1-7] MaxCPUsPerNode=384 MaxTime=7-0:00:00 State=UP TRESBillingWeights="CPU=1.0,MEM=0.125G,GRES/gpu=9.6" PartitionName=cpu Nodes=node[1-7] MaxCPUsPerNode=182 MaxTime=7-0:00:00 State=UP TRESBillingWeights="CPU=1.0,MEM=0.125G,GRES/gpu=9.6" I currently have one recently finished job and one running job. sacct gives $ sacct --format=JobID,JobName,ReqTRES%50,AllocTRES%50,TRESUsageInAve%50,TRESUsageInMax%50 JobID JobName ReqTRES AllocTRES TRESUsageInAve TRESUsageInMax ------------ ---------- -------------------------------------------------- -------------------------------------------------- -------------------------------------------------- -------------------------------------------------- 154 interacti+ billing=9,cpu=1,gres/gpu=1,mem=1G,node=1 billing=9,cpu=2,gres/gpu=1,mem=2G,node=1 154.interac+ interacti+ cpu=2,gres/gpu=1,mem=2G,node=1 cpu=00:00:00,energy=0,fs/disk=2480503,mem=3M,page+ cpu=00:00:00,energy=0,fs/disk=2480503,mem=3M,page+ 155 interacti+ billing=9,cpu=1,gres/gpu=1,mem=1G,node=1 billing=9,cpu=2,gres/gpu=1,mem=2G,node=1155.interac+ interacti+ cpu=2,gres/gpu=1,mem=2G,node=1 billing=9 seems correct to me, since I have 1 GPU allocated, which has the largest score of 9.6. However, sshare doesn't show anything in TRESRunMins sshare --format=Account,User,RawShares,FairShare,RawUsage,EffectvUsage,TRESRunMins%110 Account User RawShares FairShare RawUsage EffectvUsage TRESRunMins -------------------- ---------- ---------- ---------- ----------- ------------- -------------------------------------------------------------------------------------------------------------- root 21589714 1.000000 cpu=0,mem=0,energy=0,node=0,billing=0,fs/disk=0,vmem=0,pages=0,gres/gpu=0,gres/gpumem=0,gres/gpuutil=0 abrol_group 2000 0 0.000000 cpu=0,mem=0,energy=0,node=0,billing=0,fs/disk=0,vmem=0,pages=0,gres/gpu=0,gres/gpumem=0,gres/gpuutil=0 luchko_group 2000 21589714 1.000000 cpu=0,mem=0,energy=0,node=0,billing=0,fs/disk=0,vmem=0,pages=0,gres/gpu=0,gres/gpumem=0,gres/gpuutil=0 luchko_group tluchko 1 0.333333 21589714 1.000000 cpu=0,mem=0,energy=0,node=0,billing=0,fs/disk=0,vmem=0,pages=0,gres/gpu=0,gres/gpumem=0,gres/gpuutil=0 Why is TRESRunMin all 0 but RawUsage is not for tluchko? I have checked and slurmdbd is running. Thank you, Tyler Sent with [Proton Mail](https://proton.me/) secure email.

1 2

2026

2025

2024

slurm-users