- slurm-users - lists.schedmd.com

Randomly draining nodes
by Nacereddine Laddaoui 08 Oct '24

08 Oct '24

Hello everyone, I’ve recently encountered an issue where some nodes in our cluster enter a drain state randomly, typically after completing long-running jobs. Below is the output from the |sinfo| command showing the reason *“Prolog error”* : |root@controller-node:~# sinfo -R REASON USER TIMESTAMP NODELIST Prolog error slurm 2024-09-24T21:18:05 node[24,31] | When checking the |slurmd.log| files on the nodes, I noticed the following errors: |[2024-09-24T17:18:22.386] [217703.extern] error: _handle_add_extern_pid_internal: Job 217703 can't add pid 3311892 to jobacct_gather plugin in the extern_step. **(repeated 90 times)** [2024-09-24T17:18:22.917] [217703.extern] error: _handle_add_extern_pid_internal: Job 217703 can't add pid 3313158 to jobacct_gather plugin in the extern_step. ... [2024-09-24T21:17:45.162] launch task StepId=217703.0 request from UID:54059 GID:1600 HOST:<SLURMCTLD_IP> PORT:53514 [2024-09-24T21:18:05.166] error: Waiting for JobId=217703 REQUEST_LAUNCH_PROLOG notification failed, giving up after 20 sec [2024-09-24T21:18:05.166] error: slurm_send_node_msg: [(null)] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC_MSG) failed: Unexpected missing socket error [2024-09-24T21:18:05.166] error: _rpc_launch_tasks: unable to send return code to address:port=<SLURMCTLD_IP>:53514 msg_type=6001: No such file or directory | If you know how to solve these errors, please let me know. I would greatly appreciate any guidance or suggestions for further troubleshooting. Thank you in advance for your assistance. Best regards, -- Télécom Paris <https://www.telecom-paris.fr> *Nacereddine LADDAOUI* Ingénieur de Recherche et de Développement 19 place Marguerite Perey CS 20031 91123 Palaiseau Cedex Site web Télécom Paris <https://www.telecom-paris.fr>X Télécom Paris <https://twitter.com/TelecomParis_>Facebook Télécom Paris <https://www.facebook.com/TelecomParis>LinkedIn Télécom Paris <https://www.linkedin.com/school/telecom-paris/>Instagram Télécom Paris <https://www.instagram.com/telecom_paris/>Blog Télécom Paris <https://imtech.wp.imt.fr> Une école de l'IMT <https://www.imt.fr>

2 1

How does --nodes=min[-max] determine number of nodes to allocate?
by Declan Valters 08 Oct '24

08 Oct '24

Hi, I am trying to understand how Slurm determines the actual number of nodes to allocate to a job specified with -nodes=min[-max] I've been trying to follow the source code through on Github - could anyone point me in the right area(s)/source files/functions to look at to understand the program logic behind this. Many thanks in advance, Declan The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th' ann an Oilthigh Dh?n ?ideann, cl?raichte an Alba, ?ireamh cl?raidh SC005336.

2 1

Jobs not getting scheduled, no priority calculation, but still in queue?
by Cutts, Tim 07 Oct '24

07 Oct '24

Something odd is going on on our cluster. User has a lot of pending jobs in a job array (a few thousand). squeue -u kmnx005 -r -t PD | head -5 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 3045324_875 core run_scp_ kmnx005 PD 0:00 1 (JobArrayTaskLimit) 3045324_876 core run_scp_ kmnx005 PD 0:00 1 (JobArrayTaskLimit) 3045324_877 core run_scp_ kmnx005 PD 0:00 1 (JobArrayTaskLimit) 3045324_878 core run_scp_ kmnx005 PD 0:00 1 (JobArrayTaskLimit) None are getting scheduled. But when I ask SLURM what that job’s priority is, it produces no output: $ sprio -j 3045324 JOBID PARTITION PRIORITY SITE AGE FAIRSHARE JOBSIZE PARTITION QOS TRES Any clues what’s going on here? -- Tim Cutts Scientific Computing Platform Lead AstraZeneca Find out more about R&D IT Data, Analytics & AI and how we can support you by visiting our Service Catalogue<https://azcollaboration.sharepoint.com/sites/CMU993> | ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com<https://www.astrazeneca.com>

2 3

SLURM GRES reservation not working properly on 24.05.1
by Minulakshmi S 07 Oct '24

07 Oct '24

Hello, *Issue 1:* I am using slurm version 24.05.1 , my slurmd has a single node where I connect multiple gres by enabling the overscribe feature. I am able to use the advance reservation of gres only using *gres** name* (tres=gres/gpu:*SYSTEM12*). i.e while in reservation period , if other users submits job with SYSTEM12 , then slurm places this job in queue *user1@host$ srun --gres=gpu:SYSTEM12:1 hostname* *srun: job 333 queued and waiting for resources * but when other users just submit a job without any system name , slurm jobs goes through on that gres immediately even though it is reserved. *user1@host$ srun --gres=gpu:1 hostname * *mylinux.wbi.com <http://mylinux.wbi.com/> * Also I can see GresUsed in busy mode using "*scontrol show node -d*" , this means the job is running on Gres/GPU and not on cpu etc. Same way , job submission based on Feature "rev1 in my case" is also going through even though it is reserved for other users in multiple partition slurm. *snippet of slurm.conf file* NodeName=cluster01 NodeAddr=cluster Port=6002CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2 Feature="rev1" Gres=gpu:SYSTEM12:1 RealMemory=64171 State=IDLE *Issue 2:* while execution , Slurm o/p's some extra prints in the srun output user1@host$ srun --gres=gpu:1 hostname srun: error: extract_net_cred: net_cred not provided srun: error: Malformed RPC of type RESPONSE_NODE_ALIAS_ADDRS(3017) received srun: error: slurm_unpack_received_msg: [mylinux.wbi.com]:41242] Header lengths are longer than data received *mylinux.wbi.com <http://mylinux.wbi.com/>* Regards, MS

1 1

Slurmctld process error 'double free or corruption' on RHEL 9 (Rocky Linux)
by William VINCENT 07 Oct '24

07 Oct '24

Hello I am writing to report an issue with the Slurmctld process on our RHEL 9 (Rocky Linux) . Twice in the past 5 days, the Slurmctld process has encountered an error that resulted in the service stopping. The error message displayed was "double free or corruption (out)". This error has caused significant disruption to our jobs, and we are concerned about its recurrence. We have tried troubleshooting the issue, but we have not been able to identify the root cause of the problem. We would appreciate any assistance or guidance you can provide to help us resolve this issue. Please let us know if you need any additional information or if there are any specific steps we should take to diagnose the problem further. Thank you for your attention to this matter. Best regards, _________________________ Jul 09 22:12:01 admin slurmctld[711010]: double free or corruption (fasttop) Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Main process exited, code=killed, status=6/ABRT Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Failed with result 'signal'. Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Consumed 11min 26.451s CPU time. ..... Jul 14 10:15:01 admin slurmctld[1633720]: double free or corruption (out) Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Main process exited, code=killed, status=6/ABRT Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Failed with result 'signal'. Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Consumed 7min 27.596s CPU time. _________________________ slurmctld -V slurm 22.05.9 ________________________ cat /etc/slurm/slurm.conf |grep -v '#' ClusterName=xxx SlurmctldHost=admin SlurmctldParameters=enable_configless SlurmUser=slurm AuthType=auth/munge CryptoType=crypto/munge SlurmctldPort=6817 StateSaveLocation=/var/spool/slurmctld SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmctldDebug=verbose DebugFlags=NO_CONF_HASH SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmdLogFile=/var/log/slurm/slurmd.log SlurmdDebug=verbose SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core,CR_LLN DefMemPerCPU=1024 MaxMemPerCPU=4096 GresTypes=gpu ProctrackType=proctrack/cgroup JobAcctGatherType=jobacct_gather/cgroup JobAcctGatherFrequency=15 JobCompType=jobcomp/none TaskPlugin=task/cgroup LaunchParameters=use_interactive_step AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=admin AccountingStoragePort=6819 AccountingStorageEnforce=associations AccountingStorageTRES=gres/gpu MailProg=/usr/bin/mailx EnforcePartLimits=YES MaxArraySize=200000 MaxJobCount=500000 MpiDefault=none ReturnToService=2 SwitchType=switch/none TmpFS=/tmpslurm/ UsePAM=1 InactiveLimit=0 KillWait=30 MessageTimeout=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 PriorityType=priority/multifactor PriorityFlags=FAIR_TREE,MAX_TRES PriorityDecayHalfLife=1-0 PriorityWeightFairshare=10000 NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=3500 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=2 Sockets=2 RealMemory=1700 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=1700 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=3500 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=r9nc-24-[1-12] NodeHostname=r9nc-24-[1-12] Sockets=2 CoresPerSocket=12 ThreadsPerCore=1 CPUs=24 RealMemory=180000 State=UNKNOWN NodeName=r9nc-48-[1-4] NodeHostname=r9nc-48-[1-4] Sockets=2 CoresPerSocket=24 ThreadsPerCore=1 CPUs=48 RealMemory=480000 State=UNKNOWN NodeName=r9ng-1080-[1-7] NodeHostname=r9ng-1080-[1-7] Sockets=2 CoresPerSocket=10 ThreadsPerCore=1 CPUs=20 RealMemory=180000 State=UNKNOWN Gres=gpu:1080ti:4 NodeName=r9ng-1080-8 NodeHostname=r9ng-1080-8 Sockets=2 CoresPerSocket=10 ThreadsPerCore=1 CPUs=20 RealMemory=176687 State=UNKNOWN Gres=gpu:1080ti:1 PartitionName=24CPUNodes Nodes=r9nc-24-[1-12] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=7500 DefMemPerCPU=7500 TRESBillingWeights="CPU=1.0,Mem=0.125G" Default=YES PartitionName=48CPUNodes Nodes=r9nc-48-[1-4] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=10000 DefMemPerCPU=8000 TRESBillingWeights="CPU=1.0,Mem=0.125G" PartitionName=GPUNodes Nodes=r9ng-1080-[1-7] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=9000 DefMemPerCPU=9000 PartitionName=GPUNodes1080-dev Nodes=r9ng-1080-8 State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=9000 DefMemPerCPU=9000 Hidden=Yes _________________________ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST 24CPUNodes* up infinite 12 idle r9nc-24-[1-12] 48CPUNodes up infinite 2 idle r9nc-48-[1-2] GPUNodes up infinite 4 idle r9ng-1080-[4-7] GPUNodes1080-dev up infinite 1 idle r9ng-1080-8 -- William VINCENT Administrateur systèmes et réseaux

4 15

license server redundancy
by Yves Kondoszek 03 Oct '24

03 Oct '24

Hello. I'm configuring a cluster that will run jobs that use FlexLM licenses. We have what I believe is a quite standard configuration of license servers - though I haven't found anything in the documentation or in the mailing list archives - that often provide the same features across several servers; for example, the license token "simulation" will be available at let's say 1717@server1, 1717@server2 and 4000@server3 . FlexLM clients support those through a path-style environment variable, typically LM_LICENSE_FILE=1717@server1:1717@server2:4000@server3 . Now, I have set up Slurm for remote dynamic licenses and update them live through "LastConsumed" from lmstat requests, so that Slurm is aware of some users checking them out externally (out of Slurm control). But my problem is that the documentation clearly states that: "When submitting jobs to remote licenses, the name and server must be used." So I have to specifically choose a server: $ sbatch -L simulation@server2 script.sh instead of telling Slurm to just use whatever license is available on any of the servers. Does any of you have any experience with redundant FlexLM licenses through several servers? Is there a way Slurm could support this?

1 0

GPU Accounting
by Emyr James 03 Oct '24

03 Oct '24

We have a node with 8 H100 GPUs that are split into MIG instances. We are using cgroups. This seems to work fine. Users can do something like sbatch --gres="gpu:1g.10gb:1"... and the job starts on the node with the gpus and cuda visible devices and the pytorch debug shows that the cgroup only gives them the gpu they asked for. In the accounting database, jobs in the job table always have the "gres_used" column be empty. I'd expect to see "gpu:1g.10gb:1" appearing for the job above. I have this set in slurm.conf AccountingStorageTRES=gres/gpu How can I see what gres was requested with the job ? At the moment I only see something like this in AllocTres billing=1,cpu=1,gres/gpu=1,mem=8G,node=1 and can't see any way to see what the specific MIG gpu asked for was. This is related to the email from Richard Lefebvre dated 7th June 2023 entitled "Billing/accounting for MIGs is not working". As far as I can see this got no replies. We are running slurm version 23.11.6. Regards, Emyr James Head of Scientific IT CRG - Centre for Genomic Regulation

2 1

slurmctld keeps segfaulting, possibly during or just after backfill
by Marcus Lauer 02 Oct '24

02 Oct '24

We are running into a problem where slurmctld is segfaulting a few times a day. We had this problem with SLURM 23.11.8 and now with 23.11.10 as well, though the problem only appears on one of the several SLURM clusters we have, and all of them use one of those versions of SLURM. I was wondering if anyone has encountered a similar issue and has any thoughts on how to prevent this. Obviously we use "SchedulerType=sched/backfill" but strangely when I switched to sched/builtin for a while there were still slurmctld segfaults. We also set "SchedulerParameters=enable_user_top,bf_max_job_test=2000". I have tried turning those off but it did not help. I have also tried tweaking several other settings to no avail. Most of the cluster runs Rocky Linux 8.10 (including the slurmctld system) though we still have some Scientific Linux 7.9 compute nodes (we compile SLURM separately for those). Here is the crash-time error from journalctl: Oct 02 06:31:20 our.host.name kernel: sched_agent[2048355]: segfault at 8 ip 00007fec755d7ea8 sp 00007fec6bffe7e8 error 4 in libslurmfull.so[7fec7555a000+1f4000] Oct 02 06:31:20 our.host.name kernel: Code: 48 39 c1 7e 19 48 c1 f8 06 ba 01 00 00 00 48 d3 e2 48 f7 da 48 0b 54 c6 10 48 21 54 c7 10 c3 b8 00 00 00 00 eb da 48 8b 4f 08 <48> 39 4e 08 48 0f 4e 4e 08 49 89 c9 48 83 f9 3f 76 4e ba 40 00 00 Oct 02 06:31:20 our.host.name systemd[1]: Started Process Core Dump (PID 2169426/UID 0). Oct 02 06:31:20 our.host.name systemd-coredump[2169427]: Process 2048344 ( slurmctld) of user 991 dumped core. This is followed by a list of each of the dozen or so related threads. The one which is dumping core is first and looks like this: Stack trace of thread 2048355: #0 0x00007fec755d7ea8 bit_and_not (libslurmfull.so) #1 0x000000000044531f _job_alloc (slurmctld) #2 0x000000000044576b _job_alloc_whole_node_internal (slurmctld) #3 0x0000000000446e6d gres_ctld_job_alloc_whole_node (slurmctld) #4 0x00007fec722e29b8 job_res_add_job (select_cons_tres.so) #5 0x00007fec722f7c32 select_p_select_nodeinfo_set (select_cons_tres.so) #6 0x00007fec756e7dc7 select_g_select_nodeinfo_set (libslurmfull.so) #7 0x0000000000496eb3 select_nodes (slurmctld) #8 0x0000000000480826 _schedule (slurmctld) #9 0x00007fec753421ca start_thread (libpthread.so.0) #10 0x00007fec745f78d3 __clone (libc.so.6) I have run slurmctld with "debug5" level logging and it appears that the error occurs right after backfill considers a large number of jobs. Slurmctld could be failing at the end of backfill or when doing something which happens just after backfill runs. Usually this is the last message before the crash: [2024-09-25T18:39:42.076] slurmscriptd: debug: _slurmscriptd_mainloop: finished If anyone has any thoughts or advice on this that would be appreciated. Thank you. -- Marcus Lauer Systems Administrator CETS Group, Research Support

1 0

SLUG Presentations Now Online!
by Victoria Hobson 01 Oct '24

01 Oct '24

Available presentations from this year's SLUG event are now online. They can be found at https://www.schedmd.com/publications/ We thank all those who presented and attended for a great event! -- Victoria Hobson SchedMD LLC Vice President of Marketing

1 0

Job Step State
by Emyr James 01 Oct '24

01 Oct '24

Dear all, I am working on a script to take completed job accounting data from the slurm accounting database and insert the equivalent data into a clickhouse table for fast reporting I can see that all the information is included in the cluster_job_table and cluster_job_step_table which seem to be joined on job_db_inx To get the cpu usage and peak memory usage etc. I can see that I need to parse the tres columns in the job steps. I couldn't find any column called MaxRSS in the database even though the sacct command prints this. I then found some data in tres_table and assume that sacct is using this. Please correct me if I'm wrong and if sacct is getting information from somwhere other than the accounting database? for the state column I get this... select state, count(*) as num from crg_step_table group by state order by num desc limit 10; +-------+--------+ | state | num | +-------+--------+ | 3 | 590635 | | 5 | 28345 | | 4 | 4401 | | 11 | 962 | | 1 | 8 | +-------+--------+ When I use sacct I see statuses seach as COMPLETED, OUT_OF_MEMORY etc. so there must be a mapping somewhere between these state ids and that text. Can someone prvide that mapping or point me to where it's defined in the database or in the code ? Many thanks, Emyr James Head of Scientific IT CRG - Centre for Genomic Regulation

2 2

2024

slurm-users ----- 2024 ----- October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users