- slurm-users - lists.schedmd.com

Slurm PID Files
by Matthias Leopold 20 Nov '24

20 Nov '24

Hi, I compiled and installed Slurm 24.05 on Ubuntu 22.04 following this tutorial: https://www.schedmd.com/slurm/installation-tutorial/ Systemd service files are from deb packages that result from this. Do I have to worry that slurmctld and slurmd don't write PID files although SlurmctldPidFile and SlurmdPidFile are defined in slurm.conf? Paths for PID files exist and are writeable, logs don't show any error. slurmdbd does write a PID file as defined in slurmdbd.conf. thx Matthias

1 0

Does Slurm support DSP
by shaobo liu 20 Nov '24

20 Nov '24

Dear all Does slurm support DSP (Digital Signal Processing)? slurm website does not see DSP related content。

3 4

Does Slurm support DSP
by shaobo liu 19 Nov '24

19 Nov '24

Dear all Does slurm support DSP (Digital Signal Processing)? slurm website does not see DSP related content。

1 0

error: Unable to contact slurm controller (connect failure)
by Daniel Rodriguez Lopez (ext) 19 Nov '24

19 Nov '24

Dear all, We recently tried to fix our version of slurm in every node of our cluster. After the instalation (slurm 20.11.9) in one of the compute nodes, most of the commads (squeue, sinfo, scontrol show config etc) returns this error: error: Unable to contact slurm controller (connect failure) The .log files don't show any errors, we have both debugs values equal to debug5. Also, the rest of the cluster works as usual. I appreciate any insight on what could be the cause. Thank you and regards, Daniel

4 3

Slurm services status=1/FAILURE for CentOs cluster
by Mojtaba Farrokhbin 19 Nov '24

19 Nov '24

<https://superuser.com/posts/1862116/timeline> I have reinstalled slurm resource management on a HPC cluster. But it seems there is a problem on starting slurmd services. Here is the system status: "systemctl status slurmd" shows: ● slurmd.service - Slurm node daemon Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Sun 2024-11-10 11:31:24 +0130; 1 weeks 1 days ago ------------------------------------------- The slurm.conf output is: # # Example slurm.conf file. Please run configurator.html # (in doc/html) to build a configuration file customized # for your environment. # # # slurm.conf file generated by configurator.html. # # See the slurm.conf man page for more information. # ClusterName=linux ControlMachine=master.cluster.... #ControlAddr= #BackupController= #BackupAddr= # SlurmUser=slurm #SlurmdUser=root SlurmctldPort=6817 SlurmdPort=6818 AuthType=auth/munge #JobCredentialPrivateKey= #JobCredentialPublicCertificate= StateSaveLocation=/var/spool/slurm/ctld SlurmdSpoolDir=/var/spool/slurm/d SwitchType=switch/none MpiDefault=none SlurmctldPidFile=/var/run/slurmctld.pid SlurmdPidFile=/var/run/slurmd.pid #ProctrackType=proctrack/pgid ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup TaskPluginParam=cpusets #PluginDir= #FirstJobId= #MaxJobCount= #PlugStackConfig= #PropagatePrioProcess= #PropagateResourceLimits= #PropagateResourceLimitsExcept= #Prolog= #Epilog= #SrunProlog= #SrunEpilog= #TaskProlog= #TaskEpilog= #TaskPlugin= #TrackWCKey=no #TreeWidth=50 #TmpFS= #UsePAM= # # GPU definition (Added by S 2022/11) GresTypes=gpu # TIMERS SlurmctldTimeout=300 SlurmdTimeout=300 InactiveLimit=0 MinJobAge=300 KillWait=30 Waittime=0 # # SCHEDULING SchedulerType=sched/backfill #SchedulerAuth= #SelectType=select/linear SelectType=select/cons_res # select partial node #SelectTypeParameters=CR_CPU_Memory SelectTypeParameters=CR_Core_Memory #FastSchedule=1 FastSchedule=0 PriorityType=priority/multifactor PriorityDecayHalfLife=0 PriorityUsageResetPeriod=NONE PriorityWeightFairshare=100000 #PriorityWeightAge=1000 PriorityWeightPartition=10000 #PriorityType=priority/multifactor #PriorityDecayHalfLife=14-0 #PriorityUsageResetPeriod=14-0 #PriorityWeightFairshare=100000 #PriorityWeightAge=1000 #PriorityWeightPartition=10000 #PriorityWeightJobSize=1000 #PriorityMaxAge=1-0 DefMemPerCPU=2000 # LOGGING SlurmctldDebug=3 SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=3 SlurmdLogFile=/var/log/slurmd.log JobCompType=jobcomp/none #JobCompLoc= # # ACCOUNTING #JobAcctGatherType=jobacct_gather/linux #JobAcctGatherFrequency=30 # AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=master #AccountingStorageLoc= #AccountingStoragePass= #AccountingStorageUser= # # COMPUTE NODES # OpenHPC default configuration # Edited by Surin 1399/11 #AccountingStorageEnforce=limits AccountingStorageEnforce=QOS,Limits,Associations #TaskPlugin=task/affinity #PropagateResourceLimitsExcept=MEMLOCK #AccountingStorageType=accounting_storage/filetxt Epilog=/etc/slurm/slurm.epilog.clean NodeName=cn0[1-5] NodeHostName=cn0[1-5] RealMemory=128307 Sockets=2 CoresPerSocket=16 ThreadsPerCore=2 Feature=HyperThread State=UNKNOWN NodeName=gp01 NodeHostName=gp01 RealMemory=128307 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 Feature=HyperThread Gres=gpu:4 State=UNKNOWN #PartitionName=all Nodes=cn0[1-5],gp01 MaxTime=INFINITE State=UP Oversubscribe=EXCLUSIVE PartitionName=all Nodes=cn0[1-5],gp01 MaxTime=10-00:00:00 State=UP MaxNodes=1 PartitionName=normal Nodes=cn0[1-5] Default=YES MaxTime=10-00:00:00 State=UP MaxNodes=1 #PartitionName=normal Nodes=cn0[1-5] Default=YES MaxTime=INFINITE State=UP Oversubscribe=EXCLUSIVE PartitionName=gpu Nodes=gp01 MaxTime=10-00:00:00 State=UP SlurmctldParameters=enable_configless ReturnToService=1 HealthCheckProgram=/usr/sbin/nhc HealthCheckInterval=300 ------------------------------------------- scontrol show node cn01 shows: NodeName=cn01 CoresPerSocket=16 CPUAlloc=0 CPUTot=64 CPULoad=N/A AvailableFeatures=HyperThread ActiveFeatures=HyperThread Gres=(null) NodeAddr=cn01 NodeHostName=cn01 RealMemory=128557 AllocMem=0 FreeMem=N/A Sockets=2 Boards=1 State=DOWN* ThreadsPerCore=2 TmpDisk=64278 Weight=1 Owner=N/A MCS_label=N/A Partitions=all,normal BootTime=None SlurmdStartTime=None CfgTRES=cpu=64,mem=128557M,billing=64 AllocTRES= CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s Reason=undraining ------------------------------------------- "scontrol ping" also works as: Slurmctld(primary) at master.cluster... is UP ------------------------------------------- "systemctl start slurmd" shows: Job for slurmd.service failed because the control process exited with error code. See "systemctl status slurmd.service" and "journalctl -xe" for details. ------------------------------------------- "systemctl status slurmd.service" shows: ● slurmd.service - Slurm node daemon Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Mon 2024-11-18 17:02:50 +0130; 41s ago Process: 219025 ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS (code=exited, status=1/FAILURE) Nov 18 17:02:50 master.cluster.... systemd[1]: Starting Slurm node daemon... Nov 18 17:02:50 master.cluster.... slurmd[219025]: fatal: Unable to determine this slurmd's NodeName Nov 18 17:02:50 master.cluster.... systemd[1]: slurmd.service: control process exited, code=exited status=1 Nov 18 17:02:50 master.cluster.... systemd[1]: Failed to start Slurm node daemon. Nov 18 17:02:50 master.cluster.... systemd[1]: Unit slurmd.service entered failed state. Nov 18 17:02:50 master.cluster.... systemd[1]: slurmd.service failed. ------------------------------------------- "journalctl -xe" output is: Nov 18 17:04:54 master.cluster.... dhcpd[2250]: DHCPDISCOVER from 70:35:09:f8:13:40 via eno2: network eno2: no free leases Nov 18 17:05:04 master.cluster.... dhcpd[2250]: DHCPDISCOVER from 70:35:09:f8:13:40 via eno2: network eno2: no free leases Nov 18 17:05:08 master.cluster.... dhcpd[2250]: DHCPDISCOVER from 70:35:09:f8:13:40 via eno2: network eno2: no free leases Nov 18 17:05:12 master.cluster.... dhcpd[2250]: DHCPDISCOVER from 70:35:09:f8:13:40 via eno2: network eno2: no free leases Nov 18 17:05:22 master.cluster.... dhcpd[2250]: DHCPDISCOVER from 70:35:09:f8:13:40 via eno2: network eno2: no free leases Nov 18 17:05:26 master.cluster.... dhcpd[2250]: DHCPDISCOVER from 70:35:09:f8:13:40 via eno2: network eno2: no free leases Nov 18 17:06:34 master.cluster.... munged[2514]: Purged 2 credentials from replay hash ------------------------------------------- slurmd.services contains the following information: [Unit] Description=Slurm node daemon After=munge.service network.target remote-fs.target ConditionPathExists=/etc/slurm/slurm.conf [Service] Type=forking EnvironmentFile=-/etc/sysconfig/slurmd ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID PIDFile=/var/run/slurmd.pid KillMode=process LimitNOFILE=131072 LimitMEMLOCK=infinity LimitSTACK=infinity Delegate=yes [Install] WantedBy=multi-user.target [root@master system]# which slurmd /usr/sbin/slurmd [root@master system]# cat slurmd.service [Unit] Description=Slurm node daemon After=munge.service network.target remote-fs.target ConditionPathExists=/etc/slurm/slurm.conf [Service] Type=forking EnvironmentFile=-/etc/sysconfig/slurmd ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID PIDFile=/var/run/slurmd.pid KillMode=process LimitNOFILE=131072 LimitMEMLOCK=infinity LimitSTACK=infinity Delegate=yes [Install] WantedBy=multi-user.target ------------------------------------------- For "which slurmd" command I get this address: /usr/sbin/slurm ------------------------------------------- ls -l /usr/sbin/slurmd shows: lrwxrwxrwx 1 root root 69 Nov 10 10:54 /usr/sbin/slurmd -> /install/centos7.9/compute_gpu/rootimg/usr/sbin/slurmd

1 0

InvalidAccount
by hmeij＠wesleyan.edu 18 Nov '24

18 Nov '24

Manual compilation of 24.05.4. slurmctld and slurmd run on same server. All works ok but all test jobs end up pending with InvalidAccount message. I do not use slurm database and have not enabled accounting. Can not find an answer for this behavior or a misconfiguration. slurm.conf file was generated using easy config tool. Any ideas how to fix this? Thx, -Henk ## looks like all users have access to test queue [hmeij@sharptail2 slurm]$ sinfo -o "%g %.10R %.20l" GROUPS PARTITION TIMELIMIT all test infinite [hmeij@sharptail2 slurm]$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST test* up infinite 1 idle sharptail2 ## simple sleep job [hmeij@sharptail2 slurm]$ sbatch sleep Submitted batch job 8 [hmeij@sharptail2 slurm]$ squeue JOBID PARTITION NAME USER ST TIME NODES CPUS MIN_MEMORY NODELIST(REASON) 8 test sleep hmeij PD 0:00 1 1 1G (InvalidAccount) [hmeij@sharptail2 slurm]$ scontrol show job 8 JobId=8 JobName=sleep UserId=hmeij(8216) GroupId=its(623) MCS_label=N/A Priority=1 Nice=0 Account=(null) QOS=(null) JobState=PENDING Reason=InvalidAccount Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=UNLIMITED TimeMin=N/A SubmitTime=2024-11-11T13:27:14 EligibleTime=2024-11-11T13:27:14 AccrueTime=2024-11-11T13:27:14 StartTime=Unknown EndTime=Unknown Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-11-11T13:27:14 Scheduler=Main Partition=test AllocNode:Sid=sharptail2:644662 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1-1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:1:1 ReqTRES=cpu=1,mem=1G,node=1,billing=1 AllocTRES=(null) Socks/Node=1 NtasksPerN:B:S:C=1:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=1G MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=/zfshomes/hmeij/slurm/sleep WorkDir=/zfshomes/hmeij/slurm StdErr=/zfshomes/hmeij/slurm/err StdIn=/dev/null StdOut=/zfshomes/hmeij/slurm/out TresPerTask=cpu=1 ## within a minute or so that InvalidAccount changes to None ( ## but job remains pending; 1-7 stuck over the weekend) [hmeij@sharptail2 slurm]$ squeue JOBID PARTITION NAME USER ST TIME NODES CPUS MIN_MEMORY NODELIST(REASON) 8 test sleep hmeij PD 0:00 1 1 1G (None) ## in the slurmctld.log slurmctld: sched: JobId=8 has invalid account slurmctld: debug: set_job_failed_assoc_qos_ptr: Filling in assoc for JobId=8 Assoc=0 slurmctld: debug: sched: Running job scheduler for full queue. slurmctld: error: _refresh_assoc_mgr_qos_list: no new list given back keeping cached one. ##and the slurm.conf accounting section (both AccountingStorageType lines yield same behavior) #AccountingStorageType= AccountingStorageType=accounting_storage/none #JobAcctGatherFrequency=30 #JobAcctGatherType= ## using SchedulerType = sched/builtin

3 5

Listen to job state changes
by egonle＠posteo.me 18 Nov '24

18 Nov '24

Hello, is there any way to listen to job state changes of slurm 23.x or newer? I’d like to kind of subscribe to job state changes instead of polling for job states. Adding this feature to slurm accounting DB seems to be last option right now, although I’d like to avoid it. Thanks&Best Egonle

4 3

How to power up all ~idle nodes and verify that they have started up without issue programmatically
by Xaver Stiensmeier 15 Nov '24

15 Nov '24

Dear Slurm User list, I would like to startup all ~idle (idle and powered down) nodes and check programmatically if all came up as expected. For context: this is for a program that sets up slurm clusters with on demand cloud scheduling. In the most easiest fashion this could be executing a command like *srun FORALL hostname* which would return the names of the nodes if it succeeds and an error message otherwise. However, there's no such input value like FORALL as far as I am aware. One could use -N{total node number} as all nodes are ~idle when this executes, but I don't know an easy way to get the total number of nodes. Best regards, Xaver

3 5

First setup of slurm with a GPU node
by Patrick Begou 13 Nov '24

13 Nov '24

Hi, I'm using slurm on a small 8 nodes cluster. I've recently added one GPU node with two Nvidia A100, one with 40Gb of RAM and one with 80Gb. As using this GPU resource increase I would like to manage this resource with Gres to avoid usage conflict. But at this time my setup do not works as I can reach a GPU without reserving it: srun -n 1 -p tenibre-gpu ./a.out can use a GPU even if the reservation do not specify this resource (checked with running nvidia-smi on the node). "tenibre-gpu" is a slurm partition with only this gpu node. From the documentation I've created a gres.conf file and it is propagated on all the nodes (9 compute nodes, 1 login node and the management node) and slurmd has been restarted. gres.conf is:* ## GPU setup on tenibre-gpu-0 NodeName=tenibre-gpu-0 Name=gpu Type=A100-40 File=/dev/nvidia0 Flags=nvidia_gpu_env NodeName=tenibre-gpu-0 Name=gpu Type=A100-80 File=/dev/nvidia1 Flags=nvidia_gpu_env * * In slurm.conf I have checked these flags: ## Basic scheduling SelectTypeParameters=CR_Core_Memory SchedulerType=sched/backfill SelectType=select/cons_tres ## Generic resources GresTypes=gpu ## Nodes list .... Nodename=tenibre-gpu-0 RealMemory=257270 Sockets=2 CoresPerSocket=16 ThreadsPerCore=1 State=UNKNOWN .... #partitions PartitionName=tenibre-gpu MaxTime=48:00:00 DefaultTime=12:00:00 DefMemPerCPU=4096 MaxMemPerCPU=8192 Shared=YES State=UP Nodes=tenibre-gpu-0 ... May be I've missed something ? I'm running Slurm 20.11.7-1. Thanks for your advices. Patrick

5 6

Slurm and NVIDIA NVML
by Matthias Leopold 13 Nov '24

13 Nov '24

Hi, I'm trying to compile Slurm with NVIDIA NVML support, but the result is unexpected. I get /usr/lib/x86_64-linux-gnu/slurm/gpu_nvml.so, but when I do "ldd /usr/lib/x86_64-linux-gnu/slurm/gpu_nvml.so" there is no reference to /lib/x86_64-linux-gnu/libnvidia-ml.so.1 (which I would expect). ~$ ldd /usr/lib/x86_64-linux-gnu/slurm/gpu_nvml.so linux-vdso.so.1 (0x00007ffd9a3f4000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0bc2c06000) /lib64/ld-linux-x86-64.so.2 (0x00007f0bc2e47000) /lib/x86_64-linux-gnu/libnvidia-ml.so.1 is present during compilation. Also I can see that NVML headers where found in config.status (else I wouldn't get gpu_nvml.so at all to my understanding). Our old cluster was deployed with NVIDIA deepops (which compiles Slurm on every node) and also has NVML support. There ldd brings the expected result ~$ ldd /usr/local/lib/slurm/gpu_nvml.so ... libnvidia-ml.so.1 => /lib/x86_64-linux-gnu/libnvidia-ml.so.1 (0x00007f3b10120000) ... I can't test actual functionality with my new binaries because I don't have a node with GPUs yet. Am I missing something? thank you Matthias

2 2

Slurm release candidate version 24.11.0rc1 available for testing
by Marshall Garey 12 Nov '24

12 Nov '24

We are pleased to announce the availability of Slurm release candidate 24.11.0rc1. To highlight some new features coming in 24.11: - New gpu/nvidia plugin. This does not rely on any NVIDIA libraries, and will build by default on all systems. It supports basic GPU detection and management, but cannot currently identify GPU-to-GPU links, or provide usage data as these are not exposed by the kernel driver. - Add autodetected GPUs to the output from "slurmd -C". - Added new QOS-based reports to "sreport". - Revamped network I/O with the "conmgr" thread-pool model. - Added new "hostlist function" syntax for management commands and configuration files. - switch/hpe_slingshot - Added support for hardware collectives setup through the fabric manager. (Requires SlurmctldParameters=enable_stepmgr) - Added SchedulerParameters=bf_allow_magnetic_slot configuration option to allow backfill planning for magnetic reservations. - Added new "scontrol listjobs" and "liststeps" commands to complement "listpids", and provide --json/--yaml output for all three subcommands. - Allow jobs to be submitted against multiple QOSes. - Added new experimental "oracle" backfill scheduling support, which permits jobs to be delayed if the oracle function determines the reduced fragmentation of the network topology is sufficiently advantageous. - Improved responsiveness of the controller when jobs are requeued by replacing the "db_index" identifier with a slurmctld-generated unique identifier. ("SLUID") - New options to job_container/tmpfs to permit site-specific scripts to modify the namespace before user steps are launched, and to ensure all steps are completely captured within that namespace. This is the first release candidate of the upcoming 24.11 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://bugs.schedmd.com against the 24.11.x version and we will address them before the first production 24.11.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Container Problems with various oci.conf settings
by Jason Pascucci 12 Nov '24

12 Nov '24

Hi, I'm running on Ubuntu 20.04. I've got a clean configuration of slurmctld and slurmd on one node. 1) I've configured oci.conf to the defaults defined by "OCI.CONF EXAMPLE FOR RUNC USING RUN (RECOMMENDED OVER USING CREATE/START):". I have a container that I can run by hand: runc --rootless true run -b /opt/pilot_results/results.20241108-184831/step1 test sh-4.2# exit exit and it returns. However, when I srun --container=/opt/pilot_results/results.20241108-184831/step1 ls it hangs after completing the ls, and I have to double ctrl-c out of it. 2) I tried using the configuration for RUNC with Create/Start and it hangs on start. 3) I tried using the configuration for CRUN using RUN, I can run the container by hand with crun, but srun fails with: srun --container=/opt/pilot_results/results.20241108-194914/step1 bash bind socket to `/run/user/1008//pd-builds-bench-1.jrp.34.0.0/notify`: Address already in use sync socket closed srun: error: pd-builds-bench-1: task 0: Exited with exit code 1 4) I tried using the configuration for CRUN with Create/Start and it errors repeatedly with: slurmstepd: error: _get_container_state: RunTimeQuery failed rc:256 output:error opening file `/run/user/1008//pd-builds-bench-1.jrp.51.0.0/status`: No such file or directory I went through the (open and closed) support tickets and couldn't find anything that reflects any of these errors, and I'm pretty stuck at this point. Any help would be welcome. Thanks, JRP

1 0

slurmctld keeps segfaulting, possibly during or just after backfill
by Marcus Lauer 12 Nov '24

12 Nov '24

We are running into a problem where slurmctld is segfaulting a few times a day. We had this problem with SLURM 23.11.8 and now with 23.11.10 as well, though the problem only appears on one of the several SLURM clusters we have, and all of them use one of those versions of SLURM. I was wondering if anyone has encountered a similar issue and has any thoughts on how to prevent this. Obviously we use "SchedulerType=sched/backfill" but strangely when I switched to sched/builtin for a while there were still slurmctld segfaults. We also set "SchedulerParameters=enable_user_top,bf_max_job_test=2000". I have tried turning those off but it did not help. I have also tried tweaking several other settings to no avail. Most of the cluster runs Rocky Linux 8.10 (including the slurmctld system) though we still have some Scientific Linux 7.9 compute nodes (we compile SLURM separately for those). Here is the crash-time error from journalctl: Oct 02 06:31:20 our.host.name kernel: sched_agent[2048355]: segfault at 8 ip 00007fec755d7ea8 sp 00007fec6bffe7e8 error 4 in libslurmfull.so[7fec7555a000+1f4000] Oct 02 06:31:20 our.host.name kernel: Code: 48 39 c1 7e 19 48 c1 f8 06 ba 01 00 00 00 48 d3 e2 48 f7 da 48 0b 54 c6 10 48 21 54 c7 10 c3 b8 00 00 00 00 eb da 48 8b 4f 08 <48> 39 4e 08 48 0f 4e 4e 08 49 89 c9 48 83 f9 3f 76 4e ba 40 00 00 Oct 02 06:31:20 our.host.name systemd[1]: Started Process Core Dump (PID 2169426/UID 0). Oct 02 06:31:20 our.host.name systemd-coredump[2169427]: Process 2048344 ( slurmctld) of user 991 dumped core. This is followed by a list of each of the dozen or so related threads. The one which is dumping core is first and looks like this: Stack trace of thread 2048355: #0 0x00007fec755d7ea8 bit_and_not (libslurmfull.so) #1 0x000000000044531f _job_alloc (slurmctld) #2 0x000000000044576b _job_alloc_whole_node_internal (slurmctld) #3 0x0000000000446e6d gres_ctld_job_alloc_whole_node (slurmctld) #4 0x00007fec722e29b8 job_res_add_job (select_cons_tres.so) #5 0x00007fec722f7c32 select_p_select_nodeinfo_set (select_cons_tres.so) #6 0x00007fec756e7dc7 select_g_select_nodeinfo_set (libslurmfull.so) #7 0x0000000000496eb3 select_nodes (slurmctld) #8 0x0000000000480826 _schedule (slurmctld) #9 0x00007fec753421ca start_thread (libpthread.so.0) #10 0x00007fec745f78d3 __clone (libc.so.6) I have run slurmctld with "debug5" level logging and it appears that the error occurs right after backfill considers a large number of jobs. Slurmctld could be failing at the end of backfill or when doing something which happens just after backfill runs. Usually this is the last message before the crash: [2024-09-25T18:39:42.076] slurmscriptd: debug: _slurmscriptd_mainloop: finished If anyone has any thoughts or advice on this that would be appreciated. Thank you. -- Marcus Lauer Systems Administrator CETS Group, Research Support

2 2

The hostname resolution case sensitive
by Bill 11 Nov '24

11 Nov '24

Hi, I want to confirm that the hostname resolution is case sensitive in SLURM ? Many thanks, Bill

3 5

Discrepancy in GPU Hour Reporting Between sacct and sreport in SLURM
by Manisha Yadav 11 Nov '24

11 Nov '24

I am trying to find the GPU hour utilization for a user during a specific time period using the sacct and sreport commands. However, I am noticing a significant difference between the outputs of these two commands. Could you explain the reasons for this discrepancy? Are there specific factors or configurations in SLURM that could lead to variations in the reported GPU hours? There is a significant discrepancy in the results produced for GPU hour utilization by the sacct and sreport commands in SLURM Thanks, Manisha ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

1 0

Access denied by pam_slurm_adopt
by John Hearns 10 Nov '24

10 Nov '24

I have cluster which uses Slurm 23.11.6 When I submit a multi-node job and run something like clush -b -w $SLURM_JOB_NODELIST "date" very often the ssh command fails with: Access denied by pam_slurm_adopt: you have no active jobs on this node This will happen maybe on 50% of the nodes There is the same behaviour of I salloc a number of nodes then try to ssh to a node. I have traced this to slurmstepd spawning a long sleep, which I believe allows proctrackd to 'see' if a job is active. On nodes that I can ssh into: root 3211 1 0 Nov08 ? 00:00:00 /usr/sbin/slurmd --systemd root 3227 1 0 Nov08 ? 00:00:00 /usr/sbin/slurmstepd infinity root 24322 1 0 15:40 ? 00:00:00 slurmstepd: [15709.extern] root 24326 24322 0 15:40 ? 00:00:00 \_ sleep 100000000 On nodes where I cannot ssh: root 3226 1 0 Nov08 ? 00:00:00 /usr/sbin/slurmd --systemd root 3258 1 0 Nov08 ? 00:00:00 /usr/sbin/slurmstepd infinity Maybe I am not understanding something here? ps. I ahve tried to run the pam_slurm_adopt module with options to debug, and have not found anything useful John H

2 2

Change primary alloc node
by Bhaskar Chakraborty 04 Nov '24

04 Nov '24

Hi, Is there a way to change/control the primary node (i.e. where the initial task starts) as part of a job's allocation. For eg, if a job requires 6 CPUs & its allocation is distributed over 3 hosts h1, h2 & h3 I find that it always starts the task in 1 particularnode (say h1) irrespective of how many slots were available in the hosts. Can we somehow let slurm have the primary node as h2? Is there any C-API inside select plugin which can do this trick if we were to control it through the configured select plugin? Thanks.-Bhaskar.

2 4

Slurm Job Sched Priority
by Bhaskar Chakraborty 03 Nov '24

03 Nov '24

Hello, Is there any DS in slurmctld which portrays the dynamic relative priority of pending jobs? We are trying to use slurm for developing a scheduling solution and 1 of the problems we face at the outset is how to determinethe order of scheduling for pending jobs. One option is to find scheduling iteration window begin & close pointers & cache the job ids as seen in order & then make them the priority order at that point of time. ( This means for 500 pending jobs, say, if we can find which are the slurmctld calls which mark the beginning & end of a sched iteration then we can use the scheduling orderof jobs as the relative priority order for that period of time, of course it may change depending on fairshare, user initiated priority modification etc. ) A concrete existing data structure showing the dynamic priority itself from slurmctld would be handy. Help appreciated. Thanks! Bhaskar.

3 4

Re: Change primary alloc node
by Bhaskar Chakraborty 31 Oct '24

31 Oct '24

Hello, Just to add some context here. We plan to use slurm for developing a sched solution which interacts with a backend system. Now, the backend system has pieces of h/w which require specific host in the allocation to be the primary/master host wherein the initial task would be launched, this in turn is driven by the job's placement orientation on the h/w itself. So, our primary task should launch in the asked primary host while secondary / remote tasks would subsequently get started on other hosts. Hope this brings some context to the problem as to why a specific host is necessary to be the starting host. Regards,Bhaskar. On Thursday 31 October, 2024 at 12:04:37 am IST, Laura Hild <lsh(a)jlab.org> wrote: I think if you tell the list why you care which of the Nodes is BatchHost, they may be able to provide you with a better solution. ________________________________________ Od: Bhaskar Chakraborty via slurm-users <slurm-users(a)lists.schedmd.com> Poslano: sreda, 30. oktober 2024 12:35 Za: slurm-users(a)schedmd.com Zadeva: [slurm-users] Change primary alloc node Hi, Is there a way to change/control the primary node (i.e. where the initial task starts) as part of a job's allocation. For eg, if a job requires 6 CPUs & its allocation is distributed over 3 hosts h1, h2 & h3 I find that it always starts the task in 1 particular node (say h1) irrespective of how many slots were available in the hosts. Can we somehow let slurm have the primary node as h2? Is there any C-API inside select plugin which can do this trick if we were to control it through the configured select plugin? Thanks. -Bhaskar.

3 2

Why AllowAccounts not work in slurm-23.11.6
by daijiangkuicgo＠gmail.com 30 Oct '24

30 Oct '24

I have set AllowAccounts=sunlabc5hpc,root, but it doesn’t seem to work. User c010637 is not part of the sunlabc5hpc account but is still able to use the sunlabc5hpc partition. I have tried setting EnforcePartLimits to ALL, ANY, and NO, but none of these options resolved the issue. [c010637@sl-login ~]$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST cpu* up infinite 3 mix sl-c[0035,0042-0043] cpu* up infinite 1 idle sl-c0036 gpu up infinite 3 idle sl-c[0045-0047] sunlabc5hpc up infinite 1 idle sl-c0048 [c010637@sl-login ~]$ scontrol show partition sunlabc5hpc PartitionName=sunlabc5hpc AllowGroups=ALL AllowAccounts=sunlabc5hpc,root AllowQos=ALL AllocNodes=ALL Default=NO QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED Nodes=sl-c0048 PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=256 TotalNodes=1 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=256,mem=515000M,node=1,billing=256,gres/gpu=8 [c010637@sl-login ~]$ sacctmgr list assoc format=cluster,user,account%20,qos user=$USER Cluster User Account QOS ---------- ---------- -------------------- -------------------- snowhpc c010637 c010637_bank normal [c010637@sl-login ~]$ sacctmgr list account sunlabc5hpc Account Descr Org ---------- -------------------- -------------------- sunlabc5h+ sunlabc5hpc sunlabc5hpc [c010637@sl-login ~]$ sacctmgr show assoc where Account=sunlabc5hpc format=User,Account User Account ---------- ---------- sunlabc5h+ c010751 sunlabc5h+ snowdai sunlabc5h+

5 9

Re: 转发: What is the safe upgrade path when upgrade from slurm21.08 and mariadb5.5?
by taleintervenor＠sjtu.edu.cn 29 Oct '24

29 Oct '24

Thanks for all your help. So it seems we can skip the trouble of compiling SLURM over different mariadb versions. Tianyang Zhang SJTU Network Information Center 发件人: Sid Young <sid.young(a)gmail.com> 发送时间: 2024年10月30日 7:19 收件人: Andrej Sec <andrej.sec(a)savba.sk> 抄送: taleintervenor(a)sjtu.edu.cn; slurm-users(a)lists.schedmd.com 主题: Re: [slurm-users] Re: 转发: What is the safe upgrade path when upgrade from slurm21.08 and mariadb5.5? I recently upgraded from 20.11 to 24.05.2, before moving the cluster from CentOS 7.9 to Oracle Linux 8.10 The DB upgrade should be pretty simple, do a mysqldump first, then uninstall the old DB, change the repo's and install the new DB version. It should recognise the DB files on disk and access them. Do another DB backup on the new DB version. then roll through the Slurm upgrades. I picked the first and last version of each release, and systematically went through each node till it was done. First the slurm controller node, then the compute nodes. To avoid Job loss, drain the nodes or you end up with a situation where the slurmd can't talk to the running slurmstepd and the job(s) gets lost. (Shows as a "Protocol Error"). Ole sent me a link to this guide which mostly worked. https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/#upgrade-slurm… Sid Young W: https://off-grid-engineering.com On Tue, Oct 29, 2024 at 6:33 PM Andrej Sec via slurm-users <slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> > wrote: Hi, we are facing a similar task. We have a Slurm 22.05 / MariaDB 5.5.68 environment and want to upgrade to a newer version. According to the documentation, it’s recommended to upgrade from 22.05 to a maximum of 23.11 in one step. With the MariaDB upgrade, there’s a challenge between 10.1 and 10.2+ due to incompatible changes (https://mariadb.com/kb/en/changes-improvements-in-mariadb-10-2) This upgrade, as I understand from the documentation, requires at least slurm 22.05, where it is automatically handled by the slurmdbd service. In the test lab, we performed the following tests: a. Incremental upgrade - according to MariaDB recommendations: 1. Upgrade MariaDB 5.5.68 -> 10.1.48 -> 10.2.44 2. Start the Slurm suite 22.05, checking content after each MariaDB upgrade step. During the 10.1 -> 10.2 upgrade, the slurmdbd service automatically converted the database to the required format. We had enabled general.log in MariaDB, allowing detailed inspection of database changes during conversion. 3. Upgrade slurmdbd to version 23.11 4. Upgrade slurmctld to version 23.11 5. Upgrade slurmd to version 23.11 6. Check the database content and compare tests before and after the upgrade (we used various reports with scontrol, sreport, sacct, sacctmgr for verification). b. Direct MariaDB upgrade from 5.5.68 to 10.2.44 using the same approach. According to the tests, this resulted in the same state as the incremental approach. PS: If you proceed with the upgrade, I would appreciate it if you could let us know about any potential challenges you encountered. Andrej Sec nscc, Bratislava, Slovakia _____ Od: "hermes via slurm-users" <slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> > Komu: slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> Odoslané: pondelok, 28. október 2024 8:48:19 Predmet: [slurm-users] =?eucgb2312_cn?q?=D7=AA=B7=A2=3A_What_is_the_safe_upgrade_path_when_upgrade_from_slurm21=2E08_and_mariadb5=2E5=3F?= Hi everyone: We are currently running business on SLURM21.08 and mariadb5.5. When talking about the upgrade, we need to keep all the users and jobs history data. And we see the official document wrote: “When upgrading an existing accounting database to MariaDB 10.2.1 or later from an older version of MariaDB or any version of MySQL, ensure you are running slurmdbd 22.05.7 or later. These versions will gracefully handle changes to MariaDB default values that can cause problems for slurmdbd.” So is this mean we have to firstly build SLURM>22.05 over mariadb5.5, and do the SLURM upgrade. Then upgrade the mariadb to newer version, and rebuild the same version of SLURM over new mariadb-devel? And is it safe to jump directly from mariadb5.5 to latest version? How can we check whether the slurm have correctly inherited the historical data? Thanks, Tianyang Zhang SJTU Network Information Center -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com> -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com>

1 0

转发: What is the safe upgrade path when upgrade from slurm21.08 and mariadb5.5?
by taleintervenor＠sjtu.edu.cn 29 Oct '24

29 Oct '24

Hi everyone: We are currently running business on SLURM21.08 and mariadb5.5. When talking about the upgrade, we need to keep all the users and jobs history data. And we see the official document wrote: “When upgrading an existing accounting database to MariaDB 10.2.1 or later from an older version of MariaDB or any version of MySQL, ensure you are running slurmdbd 22.05.7 or later. These versions will gracefully handle changes to MariaDB default values that can cause problems for slurmdbd.” So is this mean we have to firstly build SLURM>22.05 over mariadb5.5, and do the SLURM upgrade. Then upgrade the mariadb to newer version, and rebuild the same version of SLURM over new mariadb-devel? And is it safe to jump directly from mariadb5.5 to latest version? How can we check whether the slurm have correctly inherited the historical data? Thanks, Tianyang Zhang SJTU Network Information Center

3 2

Job pre / post submit scripts
by Bhaskar Chakraborty 29 Oct '24

29 Oct '24

Hi, Is there an option in slurm to launch a custom script at the time of job submission through sbatchor salloc? The script should run with submit user permission in submit area. The idea is that we need to enquire something which characterises our job’s requirement like CPUslots, memory etc from a central server and we do need read access to user area prior to that. In our use case the user doesn’t necessarily know beforehand what kind of resource his job needs.(Hence, the need for such a script which will contact the server with user area info.) Based on it we can modify the job a little later. A post submit script, if available, would inform us the slurm job id as well, it would get called just after the job has entered the system and prior to its scheduling. Thanks,Bhaskar. Sent from Yahoo Mail for iPad

4 4

Scheduling oddity with multiple GPU types in same partition
by Kevin M. Hildebrand 29 Oct '24

29 Oct '24

We have a 'gpu' partition with 30 or so nodes, some with A100s, some with H100s, and a few others. It appears that when (for example) all of the A100 GPUs are in use, if there are additional jobs requesting A100 GPUs pending, and those jobs have the highest priority in the partition, then jobs submitted for H100s won't run even if there are idle H100s. This is a small subset of our present pending queue- the four bottom jobs should be running, but aren't. The top pending job shows reason 'Resources' while the rest all show 'Priority'. Any thoughts on why this might be happening? JOBID PRIORITY TRES_ALLOC 8317749 501490 cpu=4,mem=80000M,node=1,billing=48,gres/gpu=1,gres/gpu:a100=1 8317750 501490 cpu=4,mem=80000M,node=1,billing=48,gres/gpu=1,gres/gpu:a100=1 8317745 501490 cpu=4,mem=80000M,node=1,billing=48,gres/gpu=1,gres/gpu:a100=1 8317746 501490 cpu=4,mem=80000M,node=1,billing=48,gres/gpu=1,gres/gpu:a100=1 8338679 500060 cpu=4,mem=64G,node=1,billing=144,gres/gpu=1,gres/gpu:h100=1 8338678 500060 cpu=4,mem=64G,node=1,billing=144,gres/gpu=1,gres/gpu:h100=1 8338677 500060 cpu=4,mem=64G,node=1,billing=144,gres/gpu=1,gres/gpu:h100=1 8338676 500060 cpu=4,mem=64G,node=1,billing=144,gres/gpu=1,gres/gpu:h100=1 Thanks, Kevin -- Kevin Hildebrand University of Maryland Division of IT

2 1

I need to limit the number of jobs per user per partition
by Stephen Connolly 28 Oct '24

28 Oct '24

I am unable to limit the number of jobs per user per partition. I have searched the internet, the forums and the slurm documentation. I created a partition with a QOS having MaxJobsPU=1 and MaxJobsPA=1 Created a user stephen with account=stephen and MaxJobs=1 However if I sbatch a test job (sleep 180) multiple times they all run concurrently. I am at a loss of what else to do. Help would be very appreciated . Thank you -- Stephen Connolly JSI Data Systems Ltd 613-727-9353 stephen(a)jsidata.ca

2 1