- slurm-users - lists.schedmd.com

Issue with Enforcing GPU Usage Limits in Slurm
by lyz＠simplehpc.com 17 Apr '25

17 Apr '25

Hi, I am currently encountering an issue with Slurm's GPU resource limitation. I have attempted to restrict the number of GPUs a user can utilize by executing the following command: sacctmgr modify user lyz set MaxTRES=gres/gpu=2 This command is intended to limit user 'lyz' to using a maximum of 2 GPUs. However, when the user submits a job using srun, specifying CUDA 0, 1, 2, and 3 in the job script, or os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3", the job still utilizes all 4 GPUs during execution. This indicates that the GPU usage limit is not being enforced as expected. How can I resolve this situation.

4 14

pam_slurm_adopt and multiple jobs on the same worker node
by Massimo Sgaravatto 14 Apr '25

14 Apr '25

Dear all With the pam_slurm_adopt module as far as I understand you can ssh to a worker node if there is at least a job running on the node by that user. If there are multiple jobs, if I am not wrong you will be "mapped" to the last job started on the node. And, if you are using cgroups, you will be "confined" to the resources assigned to this "last" job. Is it possible in some way to specify the job to be mapped, in case there are multiple jobs for that user on the same node ? Thanks, Massimo

2 2

sllurmrestd via unix socket
by Brian Andrus 11 Apr '25

11 Apr '25

All, Maybe someone has seen this. I have slurmrestd running listening on port 8081 as well as a unix socket at /run/slurmrestd/slurmrestd.sock I am able to query the port with curl and do a ping. Everything seems fine. Other commands work as well. "pings": [ { "hostname": "head01", "pinged": "UP", "responding": true, "latency": 1605, "mode": "primary", "primary": true }, { "hostname": "head02", "pinged": "UP", "responding": true, "latency": 2353, "mode": "backup", "primary": false } When I try doing so via socket, however, my two head nodes show 'DOWN' : "pings": [ { "hostname": "head01", "pinged": "DOWN", "latency": 11784, "mode": "primary" }, { "hostname": "head02", "pinged": "DOWN", "latency": 12668, "mode": "backup" } Other commands fail with: "error_number": 1007, "error": "Protocol authentication error", I'll admit, I don't usually use sockets, so I could easily be overlooking something there. Permissions on the socket look right. I am getting json back, so it is connecting. Note: slurmrestd is running under it's own user (not root and not slurmuser). Any ideas? Thanks in advance, Brian Andrus

1 0

Minimum cpu cores per node partition level configuration
by Jeherul Islam 10 Apr '25

10 Apr '25

Dear All, I need to configure the slurm so the user must take a certain minimum number of CPU cores for a particular partition(not system-wide). Otherwise, the job must not run. Any suggestions will be highly appreciated. With Thanks and Regards -- Jeherul Islam

4 4

weird sacct behavior?
by Pierre Abele 10 Apr '25

10 Apr '25

Hi everyone, I am currently stuck with an sacct issue and would appreciate any help/hints/ideas: My users cannot retrieve job data from their currently running jobs through sacct anymore. Running sacct -a as root also reproduces this issue: It does not show running jobs, but both sacct -j <JobID> and squeue -j <JobID> do. AFAICT, this is not intended behavior (?). Also including longer time windows witch sacct -S ... -E did not help. root@slurmmaster:~# sacct -a | grep 154415 # this returns nothing root@slurmmaster:~# sacct -j 154415 JobID JobName Partition Account AllocCPUS State ExitCode ------------ ---------- ---------- ---------- ---------- ---------- -------- 154415 allocation primevo 0 PENDING 0:0 154415.batch batch primevo 2 RUNNING 0:0 154415.exte+ extern primevo 2 RUNNING 0:0 root@slurmmaster:~# squeue -j 154415 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 154415 standard genedrop username R 1:31 1 hpc020 Also, possibly related, we had a slurmdbd crash before this changed. We run Ubuntu Server 24.04 LTS with Slurm 24.05.4, using a MariaDB accounting database hosted on the same machine as the Slurm controller. Does anyone here have any ideas? Best, Pierre -- Pierre Abele, M.Sc. HPC Administrator Max-Planck-Institute for Evolutionary Anthropology Department of Primate Behavior and Evolution Deutscher Platz 6 04103 Leipzig Room: U2.80 E-Mail: pierre_abele(a)eva.mpg.de Phone: +49 (0) 341 3550 245

1 0

Re: slurmctld HA ; backup controller doesn't schedule and start any job
by Hiromasa Watanabe 10 Apr '25

10 Apr '25

Hi all, Fortunately I solved this problem by changing Slurm version from 22.05.9 to 23.11.10. With Slurm 23.11.10, after stopping the primary slurmctld and slurmdbd, when I submit a job with sbatch while backup slurmctld and slurmdbd are running, the job becomes scheduled and runs. I don't know why, but this is OK. Thanks, Hiro

1 0

errors while trying to setup slurmdbd.
by Steven Jones 10 Apr '25

10 Apr '25

root@vuwunicohpcdbp1 ~]# systemctl status slurmdbd × slurmdbd.service - Slurm DBD accounting daemon Loaded: loaded (/usr/lib/systemd/system/slurmdbd.service; enabled; preset: disabled) Active: failed (Result: exit-code) since Thu 2025-04-10 10:28:52 NZST; 2min 33s ago Duration: 85ms Process: 2413 ExecStart=/usr/sbin/slurmdbd -D -s $SLURMDBD_OPTIONS (code=exited, status=1/FAILURE) Main PID: 2413 (code=exited, status=1/FAILURE) CPU: 11ms Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz systemd[1]: Started Slurm DBD accounting daemon. Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz slurmdbd[2413]: slurmdbd: accounting_storage/as_mysql: _check_mysql_concat_is_sane: MySQL server version > Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz slurmdbd[2413]: slurmdbd: accounting_storage/as_mysql: init: Accounting storage MYSQL plugin loaded Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz slurmdbd[2413]: slurmdbd: fatal: This host not configured to run SlurmDBD ((vuwunicohpcdbp1 or vuwunicohp> Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz systemd[1]: slurmdbd.service: Main process exited, code=exited, status=1/FAILURE Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz systemd[1]: slurmdbd.service: Failed with result 'exit-code'. lines 1-14/14 (END) This is a current RHEL9.5 [root@vuwunicohpcdbp1 ~]# rpm -qi mariadb Name : mariadb Epoch : 3 Version : 10.5.27 Release : 1.el9_5 Architecture: x86_64 Install Date: Fri 04 Apr 2025 11:13:14 AM NZDT Group : Unspecified Size : 18784786 License : GPLv2 and LGPLv2 Signature : RSA/SHA256, Fri 06 Dec 2024 10:47:26 PM NZDT, Key ID 199e2f91fd431d51 Source RPM : mariadb-10.5.27-1.el9_5.src.rpm Build Date : Thu 05 Dec 2024 03:22:03 PM NZDT Build Host : x86-64-04.build.eng.rdu2.redhat.com Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> Vendor : Red Hat, Inc. URL : http://mariadb.org Summary : A very fast and robust SQL database server Description : MariaDB is a community developed fork from MySQL - a multi-user, multi-threaded SQL database server. It is a client/server implementation consisting of a server daemon (mariadbd) and many different client programs and libraries. The base package contains the standard MariaDB/MySQL client programs and utilities. [root@vuwunicohpcdbp1 ~]# ========= # Example slurmdbd.conf file. # # See the slurmdbd.conf man page for more information. # # Archive info #ArchiveJobs=yes #ArchiveDir="/tmp" #ArchiveSteps=yes #ArchiveScript= #JobPurge=12 #StepPurge=1 # # Authentication info AuthType=auth/munge #AuthInfo=/var/run/munge/munge.socket.2 # # slurmDBD info DbdAddr=127.0.0.1 #DbdAddr=localhost DbdHost=vuwunicoslurmrp1.ods.vuw.ac.nz #DbdHost=localhost DbdPort=6819 SlurmUser=slurm #MessageTimeout=300 DebugLevel=verbose #DefaultQOS=normal,standby LogFile=/var/log/slurm/slurmdbd.log PidFile=/var/run/slurmdbd/slurmdbd.pid #PluginDir=/usr/lib/slurm #PrivateData=accounts,users,usage,jobs #TrackWCKey=yes # # Database info StorageType=accounting_storage/mysql #StorageHost=localhost StorageHost=127.0.0.1 #StoragePort=1234 StoragePort=3306 #StoragePort=6819 #StoragePass=xxxxx StoragePass=xxxxxx #StoragePass=aPIkaEqxLrA StorageUser=slurm StorageLoc=slurm_acct_db #ssj # ##ssj fini ============ This ran in test just fine but the slurm version is slightly newer in this case, [root@vuwunicohpcdbp1 admjonesst1]# rpm -qa |grep slurm slurm-24.11.3-1.el9.x86_64 slurm-slurmdbd-24.11.3-1.el9.x86_64 [root@vuwunicohpcdbp1 admjonesst1]# rpm -qi slurm-slurmdbd-24.11.3-1.el9.x86_64 Name : slurm-slurmdbd Version : 24.11.3 Release : 1.el9 Architecture: x86_64 Install Date: Mon 07 Apr 2025 01:58:32 PM NZST Group : System Environment/Base Size : 2959288 License : GPLv2+ Signature : (none) Source RPM : slurm-24.11.3-1.el9.src.rpm Build Date : Mon 07 Apr 2025 12:54:21 PM NZST Build Host : vuwunicoslurmp1.ods.vuw.ac.nz URL : https://slurm.schedmd.com/ Summary : Slurm database daemon Description : Slurm database daemon. Used to accept and process database RPCs and upload database changes to slurmctld daemons on each cluster [root@vuwunicohpcdbp1 admjonesst1]# regards Steven

2 2

pam error, related to accounting?
by David Bremner 10 Apr '25

10 Apr '25

Recently I enabled accounting on my tiny (1 compute node, one head node) slurm cluster. slurmdbd.conf looks like AuthType=auth/munge DbdHost=vertex DbdPort=6819 SlurmUser=slurm StorageHost=localhost StorageType=accounting_storage/mysql StorageUser=slurm StoragePass=#elided# StoragePort=3306 StorageLoc=slurm_acct_db PurgeEventAfter=30days PurgeJobAfter=30days PurgeResvAfter=30days PurgeStepAfter=30days PurgeSuspendAfter=30days Some potentially relevant parts of slurmd.conf are: UsePAM=1 ProctrackType=proctrack/cgroup SelectTypeParameters=CR_Core_Memory AccountingStorageType=accounting_storage/slurmdbd ClusterName=simplex JobCompType=jobcomp/none JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/cgroup Since I enabled accounting the output from every job (whether started by srun or sbatch) finishes with slurmstepd-simplex: error: pam_close_session: Cannot make/remove an entry for the specified session /etc/pamd.d/slurm has the following content auth required pam_localuser.so auth required pam_shells.so account required pam_unix.so account required pam_access.so session required pam_unix.so session optional pam_systemd.so Accounting seems to be working OK. Does anyone know what PAM related code paths could be triggered by enabling accounting? d

1 0

recommended freeIPMI version
by Heckes, Frank 09 Apr '25

09 Apr '25

Hi all, I’d like to update to SLURM version 24.11.4. I was searching for a recommendation for freeIPMI (power measurement), but couldn’t find one on the schedMD web-pages (sorry, if I overlooked it). The latest available freeIPMI is 1.6.15 (https://ftp.gnu.org/gnu/freeipmi/freeipmi-1.6.15.tar.gz). Does anyone knows whether this is the best choice? Many thanks in advance for any advice. Cheers, -Frank

3 4

slurmctld HA ; backup controller doesn't schedule and start any job
by hiromasa.watanabe＠gmail.com 09 Apr '25

09 Apr '25

slurmctld HA ; backup controller doesn't schedule and start any job Hi all, I am trying a slurmctld HA configuration on two servers, using slurm version 22.05.9 of AlmaLinux 9.4. The problem is, after stopping the primary slurmctld and slurmdbd, when I submit a job with sbatch while backup slurmctld and slurmdbd are running, the job will be pending with Reason=None, and it will not be scheduled and will not start running. $ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 43 cpu_multi job hpc PD 0:00 1 (None) Why won't my job start running? What should I change to get the job to start running? The slurmctld.log and configuration files are shown below. Backup's slurmctld.log： [2025-04-09T15:31:17.000] debug3: Heartbeat at 1744180276 [2025-04-09T15:31:18.000] debug3: Heartbeat at 1744180277 [2025-04-09T15:31:19.022] debug3: Heartbeat at 1744180279 [2025-04-09T15:31:19.605] debug2: Processing RPC: REQUEST_SUBMIT_BATCH_JOB from UID=1000 [2025-04-09T15:31:19.605] debug3: _set_hostname: Using auth hostname for alloc_node: compute1 [2025-04-09T15:31:19.605] debug3: JobDesc: user_id=1000 JobId=N/A partition=cpu_multi name=job [2025-04-09T15:31:19.605] debug3: cpus=2-4294967294 pn_min_cpus=1 core_spec=-1 [2025-04-09T15:31:19.605] debug3: Nodes=1-[1] Sock/Node=65534 Core/Sock=65534 Thread/Core=65534 [2025-04-09T15:31:19.605] debug3: pn_min_memory_job=18446744073709551615 pn_min_tmp_disk=-1 [2025-04-09T15:31:19.605] debug3: immediate=0 reservation=(null) [2025-04-09T15:31:19.605] debug3: features=(null) batch_features=(null) cluster_features=(null) prefer=(null) [2025-04-09T15:31:19.605] debug3: req_nodes=(null) exc_nodes=(null) [2025-04-09T15:31:19.605] debug3: time_limit=-1--1 priority=-1 contiguous=0 shared=-1 [2025-04-09T15:31:19.605] debug3: kill_on_node_fail=-1 script=#! /bin/bash #SBATCH -p cpu_multi #SBATC... [2025-04-09T15:31:19.605] debug3: argv="./twocore.sh" [2025-04-09T15:31:19.605] debug3: environment=SHELL=/bin/bash,PYENV_SHELL=bash,HISTCONTROL=ignoredups,... [2025-04-09T15:31:19.605] debug3: stdin=/dev/null stdout=/misc/home/hpc/slurmtest/twocore_%J.out stderr=(null) [2025-04-09T15:31:19.605] debug3: work_dir=/misc/home/hpc/slurmtest alloc_node:sid=compute1:281600 [2025-04-09T15:31:19.605] debug3: power_flags= [2025-04-09T15:31:19.605] debug3: resp_host=(null) alloc_resp_port=0 other_port=0 [2025-04-09T15:31:19.605] debug3: dependency=(null) account=(null) qos=(null) comment=(null) [2025-04-09T15:31:19.605] debug3: mail_type=0 mail_user=(null) nice=0 num_tasks=2 open_mode=0 overcommit=-1 acctg_freq=(null) [2025-04-09T15:31:19.605] debug3: network=(null) begin=Unknown cpus_per_task=1 requeue=-1 licenses=(null) [2025-04-09T15:31:19.605] debug3: end_time= signal=0@0 wait_all_nodes=-1 cpu_freq= [2025-04-09T15:31:19.605] debug3: ntasks_per_node=-1 ntasks_per_socket=-1 ntasks_per_core=-1 ntasks_per_tres=-1 [2025-04-09T15:31:19.605] debug3: mem_bind=0:(null) plane_size:65534 [2025-04-09T15:31:19.605] debug3: array_inx=(null) [2025-04-09T15:31:19.605] debug3: burst_buffer=(null) [2025-04-09T15:31:19.605] debug3: mcs_label=(null) [2025-04-09T15:31:19.605] debug3: deadline=Unknown [2025-04-09T15:31:19.605] debug3: bitflags=0x1a00c000 delay_boot=4294967294 [2025-04-09T15:31:19.605] debug3: assoc_mgr_fill_in_user: found correct user: hpc(1000) [2025-04-09T15:31:19.605] debug5: assoc_mgr_fill_in_assoc: looking for assoc of user=hpc(1000), acct=hpc, cluster=cluster, partition=cpu_multi [2025-04-09T15:31:19.605] debug3: assoc_mgr_fill_in_assoc: found correct association of user=hpc(1000), acct=hpc, cluster=cluster, partition=cpu_multi to assoc=16 acct=hpc [2025-04-09T15:31:19.605] debug3: found correct qos [2025-04-09T15:31:19.607] debug2: priority/multifactor: priority_p_set: initial priority for job 44 is 33 [2025-04-09T15:31:19.607] debug2: found 1 usable nodes from config containing compute1 [2025-04-09T15:31:19.607] debug2: found 1 usable nodes from config containing compute2 [2025-04-09T15:31:19.607] debug3: _pick_best_nodes: JobId=44 idle_nodes 2 share_nodes 2 [2025-04-09T15:31:19.607] debug2: select/cons_tres: select_p_job_test: evaluating JobId=44 [2025-04-09T15:31:19.607] debug2: sched: JobId=44 allocated resources: NodeList=(null) [2025-04-09T15:31:19.607] _slurm_rpc_submit_batch_job: JobId=44 InitPrio=33 usec=2490 [2025-04-09T15:31:19.608] debug3: create_mmap_buf: loaded file `/var/spool/slurm/ctld/job_state` as buf_t [2025-04-09T15:31:19.609] debug3: Writing job id 45 to header record of job_state file [2025-04-09T15:31:21.000] debug3: Heartbeat at 1744180280 [2025-04-09T15:31:21.257] debug2: _slurm_connect: failed to connect to 192.168.56.11:6817: Connection refused [2025-04-09T15:31:21.257] debug2: Error connecting slurm stream socket at 192.168.56.11:6817: Connection refused [2025-04-09T15:31:22.000] debug3: Heartbeat at 1744180282 [2025-04-09T15:31:24.000] debug3: Heartbeat at 1744180283 [2025-04-09T15:31:25.004] debug3: Heartbeat at 1744180285 [2025-04-09T15:31:27.001] debug3: Heartbeat at 1744180287 [2025-04-09T15:31:29.000] debug3: Heartbeat at 1744180288 [2025-04-09T15:31:30.000] debug3: Heartbeat at 1744180289 [2025-04-09T15:31:31.006] debug3: Heartbeat at 1744180291 [2025-04-09T15:31:32.822] debug2: _slurm_connect: failed to connect to 192.168.56.11:6817: Connection refused [2025-04-09T15:31:32.822] debug2: Error connecting slurm stream socket at 192.168.56.11:6817: Connection refused [2025-04-09T15:31:33.007] debug3: Heartbeat at 1744180293 [2025-04-09T15:31:35.000] debug3: Heartbeat at 1744180294 [2025-04-09T15:31:36.002] debug3: Heartbeat at 1744180296 [2025-04-09T15:31:36.395] debug2: select/cons_tres: select_p_job_test: evaluating JobId=43 [2025-04-09T15:31:36.395] debug2: select/cons_tres: select_p_job_test: evaluating JobId=44 [2025-04-09T15:31:38.000] debug3: Heartbeat at 1744180297 [2025-04-09T15:31:38.497] debug2: Performing purge of old job records [2025-04-09T15:31:39.000] debug3: Heartbeat at 1744180298 [2025-04-09T15:31:40.000] debug3: Heartbeat at 1744180300 [2025-04-09T15:31:40.655] debug2: Testing job time limits and checkpoints [2025-04-09T15:31:42.000] debug3: Heartbeat at 1744180301 [2025-04-09T15:31:43.000] debug3: Heartbeat at 1744180302 slurm.conf: ClusterName=cluster SlurmctldHost=gateway1 #Primary(192.168.56.11) SlurmctldHost=gateway2 #Backup(192.168.56.12) MpiDefault=pmix ProctrackType=proctrack/cgroup PrologFlags=Contain ReturnToService=0 SlurmctldPidFile=/var/run/slurm/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurm/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurm/d SlurmUser=slurm StateSaveLocation=/var/spool/slurm/ctld SwitchType=switch/none TaskEpilog=/etc/slurm/taskepilog.sh TaskPlugin=task/cgroup,task/affinity TaskProlog=/etc/slurm/taskprolog.sh InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=10 SlurmdTimeout=300 Waittime=0 DefMemPerCPU=32 SchedulerType=sched/builtin SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory PriorityType=priority/multifactor PriorityWeightPartition=1000 AccountingStorageHost=gateway1 AccountingStorageBackupHost=gateway2 AccountingStorageType=accounting_storage/slurmdbd AccountingStoreFlags=job_comment JobCompType=jobcomp/none JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/cgroup SlurmctldDebug=debug5 SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurm/slurmd.log NodeName=compute1 CPUs=4 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=3900 Weight=1 NodeName=compute2 CPUs=4 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=3900 Weight=1 PartitionName=gpu_single Nodes=ALL PriorityJobFactor=30 MaxTime=INFINITE State=UP Default=YES PartitionName=cpu_single Nodes=ALL PriorityJobFactor=10 MaxTime=INFINITE State=UP PartitionName=cpu_multi Nodes=ALL MaxTime=INFINITE State=UP slurmdbd.conf: AuthType=auth/munge DebugLevel=4 DbdHost=gateway1 DbdBackupHost=gateway2 LogFile=/var/log/slurm/slurmdbd.log PidFile=/var/run/slurm/slurmdbd.pid PurgeEventAfter=1month PurgeJobAfter=1month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month PurgeTXNAfter=1month PurgeUsageAfter=1month SlurmUser=slurm StorageType=accounting_storage/mysql StorageHost=gateway1 StoragePass=mypassword StorageUser=slurm Best regards, Hiro

1 0