May 2024 - slurm-users - lists.schedmd.com

Slurm version 24.05.0 is now available
by Marshall Garey 30 May '24

30 May '24

We are pleased to announce the availability of Slurm 24.05.0. To highlight some new features in 24.05: - Isolated Job Step management. Enabled on a job-by-job basis with the --stepmgr option, or globally through SlurmctldParameters=enable_stepmgr. - Federation - Allow for client command operation while SlurmDBD is unavailable. - New MaxTRESRunMinsPerAccount and MaxTRESRunMinsPerUser QOS limits. - New USER_DELETE reservation flag. - New Flags=rebootless option on Features for node_features/helpers which indicates the given feature can be enabled without rebooting the node. - Cloud power management options: New "max_powered_nodes=<limit>" option in SlurmctldParamters, and new SuspendExcNodes=<nodes>:<count> syntax allowing for <count> nodes out of a given node list to be excluded. - StdIn/StdOut/StdErr now stored in SlurmDBD accounting records for batch jobs. - New switch/nvidia_imex plugin for IMEX channel management on NVIDIA systems. - New RestrictedCoresPerGPU option at the Node level, designed to ensure GPU workloads always have access to a certain number of CPUs even when nodes are running non-GPU workloads concurrently. The Slurm documentation has also been updated to the 24.05 release. (Older versions can be found in the archive, linked from the main documentation page.) Slurm can be downloaded from https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Executing srun -n X where X is greater than total CPU in entire cluster
by Dan Healy 30 May '24

30 May '24

Hi there, SLURM community, I swear I've done this before, but now it's failing on a new cluster I'm deploying. We have 6 compute nodes with 64 cpu each (384 CPU total). When I run `srun -n 500 hostname`, the task gets queued since there's not 500 available CPU. Wasn't there an option that allows for this to be run where the first 384 tasks execute, and then the remaining execute when resources free up? Here's my conf: # Slurm Cgroup Configs used on controllers and workersslurm_cgroup_config: CgroupAutomount: yes ConstrainCores: yes ConstrainRAMSpace: yes ConstrainSwapSpace: yes ConstrainDevices: yes# Slurm conf file settingsslurm_config: AccountingStorageType: "accounting_storage/slurmdbd" AccountingStorageEnforce: "limits" AuthAltTypes: "auth/jwt" ClusterName: "cluster" AccountingStorageHost : "{{ hostvars[groups['controller'][0]].ansible_hostname }}" DefMemPerCPU: 1024 InactiveLimit: 120 JobAcctGatherType: "jobacct_gather/cgroup" JobCompType: "jobcomp/none" MailProg: "/usr/bin/mail" MaxArraySize: 40000 MaxJobCount: 100000 MinJobAge: 3600 ProctrackType: "proctrack/cgroup" ReturnToService: 2 SelectType: "select/cons_tres" SelectTypeParameters: "CR_Core_Memory" SlurmctldTimeout: 30 SlurmctldLogFile: "/var/log/slurm/slurmctld.log" SlurmdLogFile: "/var/log/slurm/slurmd.log" SlurmdSpoolDir: "/var/spool/slurm/d" SlurmUser: "{{ slurm_user.name }}" SrunPortRange: "60000-61000" StateSaveLocation: "/var/spool/slurm/ctld" TaskPlugin: "task/affinity,task/cgroup" UnkillableStepTimeout: 120 -- Thanks, Daniel Healy

2 2

Jobs showing running but not running
by Sushil Mishra 29 May '24

29 May '24

Hi All, I'm managing a cluster with Slurm, consisting of 4 nodes. One of the compute nodes appears to be experiencing issues. While the front node's 'squeue' command indicates that jobs are running, upon connecting to the problematic node, I observe no active processes and GPUs are not being utilized. [sushil@ccbrc ~]$ sinfo -Nel Wed May 29 12:00:08 2024 NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON gag 1 defq* mixed 48 2:24:1 370000 0 1 (null) none gag 1 glycore mixed 48 2:24:1 370000 0 1 (null) none glyco1 1 defq* completing* 128 2:64:1 500000 0 1 (null) none glyco1 1 glycore completing* 128 2:64:1 500000 0 1 (null) none glyco2 1 defq* mixed 128 2:64:1 500000 0 1 (null) none glyco2 1 glycore mixed 128 2:64:1 500000 0 1 (null) none mannose 1 defq* mixed 24 2:12:1 180000 0 1 (null) none mannose 1 glycore mixed 24 2:12:1 180000 0 1 (null) none On glyco1 (affected node!): squeue # gets stuck sudo systemctl restart slurmd # gets stuck I tried the following to clear the jobs stuck in CG state, but any new job appears to be stuck in a 'running' state without actually running. scontrol update nodename=glyco1 state=down reason=cg scontrol update nodename=glyco1 state=resume reason=cg There is no I/O issue in that node, and all file systems are under 30% in use. Any advice on how to resolve this without rebooting the machine? Best, Sushil

3 2

dynamical configuration || meta configuration mgmt
by Heckes, Frank 29 May '24

29 May '24

Hello all, I’m sorry if this has been asked and answered before, but I couldn’t find anything related. Does anyone know whether a framework of sorts exists that allow to change certain SLURM configuration parameters provided some conditions in the batch system’s state are detected and of course are revert if the state became the old one again? (To be more concrete: We like to raise or unset maxjobPU to run as much small jobs as possible to allocate all nodes as soon as certain threshold of free nodes are available and of course some other scenarios) Many thanks in advance. Cheers, -Frank Max-Planck-Institut für Sonnensystemforschung Justus-von-Liebig-Weg 3 D-37077 Göttingen Phone: [+49] 551 – 384 979 320 E-Mail: <mailto:heckes@mps.mpg.de> heckes(a)mps.mpg.de

2 1

Configuring sacct to report state=OUT_OF_MEMORY
by Lee 29 May '24

29 May '24

Hello, *Background :* I am working on a small cluster that is managed by Base Command Manager v10.0 using Slurm 23.02.7 with Ubuntu 22.04.2. I have a small testing script that simply consumes memory and processors. I run my test script, it consumes more memory than allocated by Slurm and as expected it gets killed by OOM killer. In /var/log/slurmd, I see entries like : [2024-05-29T08:53:04.975] Launching batch job 65 for UID 1001 [2024-05-29T08:53:05.016] [65.batch] task/cgroup: _memcg_initialize: job: alloc=10868MB mem.limit=10868MB memsw.limit=10868MB job_swappiness=1 [2024-05-29T08:53:05.016] [65.batch] task/cgroup: _memcg_initialize: step: alloc=10868MB mem.limit=10868MB memsw.limit=10868MB job_swappiness=1 [2024-05-29T08:53:19.530] [65.batch] task/cgroup: task_cgroup_memory_check_oom: StepId=65.batch hit memory+swap limit at least once during execution. This may or may not result in some failure. [2024-05-29T08:53:19.563] [65.batch] done with job Inspecting with sacct, I see : $ sacct -j 65 --format="jobid,jobname,state,exitcode" JobID JobName State ExitCode ------------ ---------- ---------- -------- 65 wrap FAILED 9:0 65.batch batch FAILED 9:0 Based on my previous experience with a RHEL based Slurm cluster, I would expect the state to be listed as OUT_OF_MEMORY and the exitcode to be 0:125. *Question : * 1. How do I configure [slurm,cgroup].conf such that when Slurm kills a job due to exceeding the allocated memory, sacct reports the state as OUT_OF_MEMORY? See below for my configuration : *Configuration:* /etc/default/grub : ~# grep -v "^#" /etc/default/grub GRUB_DEFAULT=0 GRUB_TIMEOUT_STYLE=hidden GRUB_TIMEOUT=10 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="" GRUB_CMDLINE_LINUX="biosdevname=0 cgroup_enable=memory swapaccount=1" GRUB_GFXMODE="1024x768,800x600,auto" GRUB_BACKGROUND="/boot/grub/bcm.png" slurm.conf : ~# grep -v "^#" /cm/shared/apps/slurm/var/etc/bcm10-slurm/slurm.conf SlurmUser=slurm SlurmctldPort=6817 SlurmdPort=6818 AuthType=auth/munge SlurmdSpoolDir=/cm/local/apps/slurm/var/spool SwitchType=switch/none MpiDefault=pmix SlurmctldPidFile=/var/run/slurmctld.pid SlurmdPidFile=/var/run/slurmd.pid ProctrackType=proctrack/cgroup ReturnToService=2 TaskPlugin=task/cgroup InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=300 SlurmdTimeout=300 Waittime=0 SlurmctldDebug=info SlurmctldLogFile=/var/log/slurmctld SlurmdDebug=info SlurmdLogFile=/var/log/slurmd JobAcctGatherType=jobacct_gather/cgroup JobAcctGatherFrequency=30 AccountingStorageType=accounting_storage/slurmdbd AccountingStorageUser=slurm SlurmctldHost=bcm10-h01 AccountingStorageHost=master NodeName=bcm10-n[01,02] Procs=4 CoresPerSocket=4 RealMemory=15988 SocketsPerBoard=1 ThreadsPerCore=1 Boards=1 MemSpecLimit=5120 Feature=location=local PartitionName="defq" Default=YES MinNodes=1 DefaultTime=UNLIMITED MaxTime=UNLIMITED AllowGroups=ALL PriorityJobFactor=1 PriorityTier=1 OverSubscribe=NO PreemptMode=OFF AllowAccounts=ALL AllowQos=ALL Nodes=bcm10-n[01,02] ClusterName=bcm10-slurm SchedulerType=sched/backfill StateSaveLocation=/cm/shared/apps/slurm/var/cm/statesave/bcm10-slurm PrologFlags=Alloc GresTypes=gpu Prolog=/cm/local/apps/cmd/scripts/prolog Epilog=/cm/local/apps/cmd/scripts/epilog SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory cgroup.conf ~# grep -v "^#" /cm/shared/apps/slurm/var/etc/bcm10-slurm/cgroup.conf CgroupMountpoint="/sys/fs/cgroup" CgroupAutomount=no ConstrainCores=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes ConstrainDevices=yes AllowedRamSpace=100.00 AllowedSwapSpace=0.00 MemorySwappiness=1 MaxRAMPercent=100.00 MaxSwapPercent=100.00 MinRAMSpace=30 Best regards, Lee

1 0

sbatch problem
by Mihai Ciubancan 29 May '24

29 May '24

Hello, My name is Mihai and a have an issue with a small GPU cluster manage with slurm 22.05.11. I got 2 different output when I'm trying to find out the name of the nodes(one correct and one wrong). The script is: #!/bin/bash #SBATCH --job-name=test #SBATCH --output=/data/mihai/res.txt #SBATCH --partition=eli #SBATCH --nodes=2 srun echo Running on host: $(hostname) srun hostname srun sleep 15 And the output look like this: cat res.txt Running on host: mihai-x8640 Running on host: mihai-x8640 mihaigpu2 mihai-x8640 As you can see the output of the command 'srun echo Running on host: $(hostname)' is the same, as the jobs was running twice on the same node, while command 'srun hostname' it's giving me the correct output. Do you have any idea why the outputs of the 2 commands are different? Thank you, Mihai

2 6

slurm_bufs_sendto failed
by shaobo liu 28 May '24

28 May '24

slurm 23.11.0 , the slurmd error is as follows，what is the reason for this? [2024-05-29T09:21:32.450] task/affinity: task_p_slurmd_batch_request: task_p_slurmd_batch_request: 32259 [2024-05-29T09:21:32.451] task/affinity: batch_bind: job 32259 CPU input mask for node: 0xFFFFFFFFFFFFFFFF [2024-05-29T09:21:32.451] task/affinity: batch_bind: job 32259 CPU final HW mask for node: 0xFFFFFFFFFFFFFFFF [2024-05-29T09:21:32.451] Launching batch job 32259 for UID 66003 [2024-05-29T09:21:32.868] launch task StepId=32259.0 request from UID:66003 GID:66003 HOST:192.168.87.45 PORT:31104 [2024-05-29T09:21:32.868] task/affinity: lllp_distribution: JobId=32259 implicit auto binding: cores,one_thread, dist 2 [2024-05-29T09:21:32.868] task/affinity: _task_layout_lllp_cyclic: _task_layout_lllp_cyclic [2024-05-29T09:21:32.868] task/affinity: _lllp_generate_cpu_bind: _lllp_generate_cpu_bind jobid [32259]: mask_cpu,one_thread, 0x0000000000000001,0x0000000100000000,0x0000000000000002,0x0000000200000000,0x0000000000000004,0x0000000400000000,0x0000000000000008,0x0000000800000000,0x0000000000000010,0x0000001000000000,0x0000000000000020,0x0000002000000000,0x0000000000000040,0x0000004000000000,0x0000000000000080,0x0000008000000000,0x0000000000000100,0x0000010000000000,0x0000000000000200,0x0000020000000000,0x0000000000000400,0x0000040000000000,0x0000000000000800,0x0000080000000000,0x0000000000001000,0x0000100000000000,0x0000000000002000,0x0000200000000000,0x0000000000004000,0x0000400000000000,0x0000000000008000,0x0000800000000000,0x0000000000010000,0x0001000000000000,0x0000000000020000,0x0002000000000000,0x0000000000040000,0x0004000000000000,0x0000000000080000,0x0008000000000000,0x0000000000100000,0x0010000000000000,0x0000000000200000,0x0020000000000000,0x0000000000400000,0x0040000000000000,0x0000000000800000,0x0080000000000000,0x0000000001000000,0x0100000000000000,0x0000000002000000,0x0200000000000000,0x0000000004000000,0x0400000000000000,0x0000000008000000,0x0800000000000000,0x0000000010000000,0x1000000000000000,0x0000000020000000,0x2000000000000000,0x0000000040000000,0x4000000000000000,0x0000000080000000,0x8000000000000000 [2024-05-29T09:32:04.541] [32259.batch] done with step [2024-05-29T09:33:31.000] error: slurm_send_node_msg: [socket:[7270627]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:34:02.000] error: slurm_send_node_msg: [socket:[7270629]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:34:03.016] [32259.extern] done with step [2024-05-29T09:34:41.000] error: slurm_send_node_msg: [socket:[7270635]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:35:13.000] error: slurm_send_node_msg: [socket:[7270637]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:35:52.000] error: slurm_send_node_msg: [socket:[7270639]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:36:01.000] error: slurm_send_node_msg: [socket:[7270630]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:36:24.000] error: slurm_send_node_msg: [socket:[7270641]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:37:03.000] error: slurm_send_node_msg: [socket:[7270644]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:37:31.000] [32259.0] error: *** STEP 32259.0 STEPD TERMINATED ON node045 AT 2024-05-29T09:37:30 DUE TO JOB NOT RUNNING *** [2024-05-29T09:37:33.000] [32259.0] stepd_cleanup: done with step (rc[0x0]:No error, cleanup_rc[0xfbb]:Job step not running) [2024-05-29T09:37:35.000] error: slurm_send_node_msg: [socket:[7270646]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:38:14.000] error: slurm_send_node_msg: [socket:[7270649]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:38:46.000] error: slurm_send_node_msg: [socket:[7270651]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:39:25.000] error: slurm_send_node_msg: [socket:[7270657]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error

1 0

Final Call for SLUG Presenters & Early Bird Registration!
by Victoria Hobson 28 May '24

28 May '24

Slurm User Group (SLUG) 2024 is set for September 12-13 at the University of Oslo in Oslo, Norway. Registration information and a high-level schedule can be found here:https://slug24.splashthat.com/ The last day to register at the early bird pricing is this Friday, May 31st. Friday is also the deadline to submit a presentation abstract. We do not intend to extend this deadline. If you are interested in presenting your own usage, developments, site report, tutorial, etc about Slurm, please fill out the following form:https://forms.gle/N7bFo5EzwuTuKkBN7 Notifications of final presentations accepted will go out by Friday, June 14th. -- Victoria Hobson SchedMD LLC Vice President of Marketing

1 0

slurmdbd archive format
by O'Neal, Doug (NIH/NCI) [C] 28 May '24

28 May '24

My organization needs to access historic job information records for metric reporting and resource forecasting. slurmdbd is archiving only the job information, which should be sufficient for our numbers, but is using the default archive script. In retrospect, this data should have been migrated to a secondary MariaDB instance, but that train has passed. The format of the archive files is not well documented. Does anyone have a program (python/C/whatever) that will read a job_table_archive file and decode it into a parsable structure? Douglas O'Neal, Ph.D. (contractor) Manager, HPC Systems Administration Group, ITOG Frederick National Laboratory for Cancer Research Leidos Biomedical Research, Inc. Phone: 301-228-4656 Email: Douglas.O'Neal(a)nih.gov<mailto:Doug%20O'Neal%20%3cDouglas.O'Neal(a)nih.gov%3e>

2 2

Fwd: Re: Performance Discrepancy between Slurm and Direct mpirun for VASP Jobs.
by Hongyi Zhao 28 May '24

28 May '24

---------- Forwarded message --------- From: Hermann Schwärzler <hermann.schwaerzler(a)uibk.ac.at> Date: Tue, May 28, 2024 at 4:10 PM Subject: Re: [slurm-users] Re: Performance Discrepancy between Slurm and Direct mpirun for VASP Jobs. To: Hongyi Zhao <hongyi.zhao(a)gmail.com> Hi Zhao, On 5/28/24 03:08, Hongyi Zhao wrote: [...] > > What's the complete content of cli_filter.lua and where should I put this file? [...] Below you find the complete content of our cli_filter.lua. It has to be put into the same directory as "slurm.conf". --------------------------------- 8< --------------------------------- -- see https://github.com/SchedMD/slurm/blob/master/etc/cli_filter.lua.example function slurm_cli_pre_submit(options, pack_offset) return slurm.SUCCESS end function slurm_cli_setup_defaults(options, early_pass) -- Make --hint=nomultithread the default behavior -- if users specify an other --hint=XX option then -- it will override the setting done here options['hint'] = 'nomultithread' return slurm.SUCCESS end function slurm_cli_post_submit(offset, job_id, step_id) return slurm.SUCCESS end --------------------------------- >8 --------------------------------- Hopefully this helps... Regards, Hermann -- Assoc. Prof. Hongsheng Zhao <hongyi.zhao(a)gmail.com> Theory and Simulation of Materials Hebei Vocational University of Technology and Engineering No. 473, Quannan West Street, Xindu District, Xingtai, Hebei province

1 0