April 2025 - slurm-users - lists.schedmd.com

problem with slurmd dynamic mode with slurmctld in docker
by stef53864 22 Apr '25

22 Apr '25

hi everybody, i try to user dynamic mode with configless mode with slurm 24.11.3 and upgrade to 24.11.4 with slurmd and i found a problem. slurmctld is a container with docker, and my node is outside the container network . slurmctld register my ip with a function getpeeraddr on the slurmctld socket. but my ip connected to the socket come from the docker nat/bridge so slurmctld register my ip bridged ( not my real ip ) that is to say the docker gateway (172.20.0.1) *scontrol show node* *-------------------------* *NodeName=ltlsbubble1 Arch=x86_64 CoresPerSocket=4..NodeAddr=172.20.0.1 NodeHostName=ltlsbubble1 Version=24.11.4* so the node go down after the "not pinging it" timeout i try to update the config *scontrol uupdate NodeName=ltlsbubble1* *NodeAddr=xx.xx.xx.xx* but a the first *scontrol reconfigure * it comes back to : *NodeAddr=172.20.0.1* in normal mode ------------------- *scontrol show nodeNodeName=ltlsbubble1 Arch=x86_64 CoresPerSocket=4..NodeAddr=ltlsbubble1 NodeHostName=ltlsbubble1 Version=24.11.4* in normal mode NodeAddr is the same than NodeName , so it use DNS resolution for communication. to verify my hypothesis, i go to the c code of slurm, identify the register function and replace it with the same mechanism than normal node in src/slurmctld/node_mgr.c i replace : set_node_comm_name(node_ptr, *comm_name*, reg_msg->hostname); by set_node_comm_name(node_ptr, NULL, reg_msg->hostname); i rebuild slutmctld with this patch and try it with dynamic mode , it works like expected *scontrol show nodeNodeName=ltlsbubble1 Arch=x86_64 CoresPerSocket=4..NodeAddr=ltlsbubble1 NodeHostName=ltlsbubble1 Version=24.1* no ip in nodeAddr , but only the nodename, so it use DNS resolution . the node works fine and no goes down for timeout ping so my question : can we have an option to force DNS resolution instead ip discover in Dynamic mode ? ( i try the option cloud_dns, but it not seems the purpose of this option) best regard, Stephane

1 0

pam_slurm_adopt ambiguity in instructions and with ssd
by Robert Kudyba 21 Apr '25

21 Apr '25

In the instructions for pam_slurm_adopt <https://slurm.schedmd.com/pam_slurm_adopt.html#ssh_config>, there are instructions such as: > > Add the following line to the appropriate file in /etc/pam.d, such as > system-auth or sshd (you may use either the "required" or "sufficient" > PAM control flag): This module is configurable. Add these options to the end of the > pam_slurm_adopt line in the appropriate file in /etc/pam.d/ (e.g., sshd > or system-auth): Assuming an OS like CentOS does this mean it should be put in both? slurm.conf on the node has: UsePAM yes slurm.conf has PrologFlags=contain and ProctrackType=proctrack/cgroup I placed the call here only in /etc/pam.d/sshd making sure it is the last line in the account stack. #%PAM-1.0 auth required pam_sepermit.so auth substack system-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so account include system-auth -account required pam_slurm_adopt.so so pam_sss.so is at the bottom of /etc/pam.d/sshd session optional pam_keyinit.so revoke session required pam_limits.so #-session optional pam_systemd.so session optional pam_oddjob_mkhomedir.so umask=0022 skel=/etc/skel session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional pam_sss.so We're testing this on an idle node. I start an interactive srun. However trying to ssh to the node gets: Apr 18 11:13:41 node11 sshd[33355]: Authorized to dk2643, krb5 principal user(a)ouruni.EDU (ssh_gssapi_krb5_cmdok) Apr 18 11:13:41 node11 sshd[33355]: pam_sss(sshd:account): Access denied for user user: 6 (Permission denied) Apr 18 11:13:41 node11 sshd[33355]: fatal: Access denied for user user by PAM account configuration [preauth] Am I missing something?

3 5

Issue with Enforcing GPU Usage Limits in Slurm
by lyz＠simplehpc.com 16 Apr '25

16 Apr '25

Hi, I am currently encountering an issue with Slurm's GPU resource limitation. I have attempted to restrict the number of GPUs a user can utilize by executing the following command: sacctmgr modify user lyz set MaxTRES=gres/gpu=2 This command is intended to limit user 'lyz' to using a maximum of 2 GPUs. However, when the user submits a job using srun, specifying CUDA 0, 1, 2, and 3 in the job script, or os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3", the job still utilizes all 4 GPUs during execution. This indicates that the GPU usage limit is not being enforced as expected. How can I resolve this situation.

4 14

pam_slurm_adopt and multiple jobs on the same worker node
by Massimo Sgaravatto 14 Apr '25

14 Apr '25

Dear all With the pam_slurm_adopt module as far as I understand you can ssh to a worker node if there is at least a job running on the node by that user. If there are multiple jobs, if I am not wrong you will be "mapped" to the last job started on the node. And, if you are using cgroups, you will be "confined" to the resources assigned to this "last" job. Is it possible in some way to specify the job to be mapped, in case there are multiple jobs for that user on the same node ? Thanks, Massimo

2 2

sllurmrestd via unix socket
by Brian Andrus 10 Apr '25

10 Apr '25

All, Maybe someone has seen this. I have slurmrestd running listening on port 8081 as well as a unix socket at /run/slurmrestd/slurmrestd.sock I am able to query the port with curl and do a ping. Everything seems fine. Other commands work as well. "pings": [ { "hostname": "head01", "pinged": "UP", "responding": true, "latency": 1605, "mode": "primary", "primary": true }, { "hostname": "head02", "pinged": "UP", "responding": true, "latency": 2353, "mode": "backup", "primary": false } When I try doing so via socket, however, my two head nodes show 'DOWN' : "pings": [ { "hostname": "head01", "pinged": "DOWN", "latency": 11784, "mode": "primary" }, { "hostname": "head02", "pinged": "DOWN", "latency": 12668, "mode": "backup" } Other commands fail with: "error_number": 1007, "error": "Protocol authentication error", I'll admit, I don't usually use sockets, so I could easily be overlooking something there. Permissions on the socket look right. I am getting json back, so it is connecting. Note: slurmrestd is running under it's own user (not root and not slurmuser). Any ideas? Thanks in advance, Brian Andrus

1 0

Minimum cpu cores per node partition level configuration
by Jeherul Islam 10 Apr '25

10 Apr '25

Dear All, I need to configure the slurm so the user must take a certain minimum number of CPU cores for a particular partition(not system-wide). Otherwise, the job must not run. Any suggestions will be highly appreciated. With Thanks and Regards -- Jeherul Islam

4 4

weird sacct behavior?
by Pierre Abele 10 Apr '25

10 Apr '25

Hi everyone, I am currently stuck with an sacct issue and would appreciate any help/hints/ideas: My users cannot retrieve job data from their currently running jobs through sacct anymore. Running sacct -a as root also reproduces this issue: It does not show running jobs, but both sacct -j <JobID> and squeue -j <JobID> do. AFAICT, this is not intended behavior (?). Also including longer time windows witch sacct -S ... -E did not help. root@slurmmaster:~# sacct -a | grep 154415 # this returns nothing root@slurmmaster:~# sacct -j 154415 JobID JobName Partition Account AllocCPUS State ExitCode ------------ ---------- ---------- ---------- ---------- ---------- -------- 154415 allocation primevo 0 PENDING 0:0 154415.batch batch primevo 2 RUNNING 0:0 154415.exte+ extern primevo 2 RUNNING 0:0 root@slurmmaster:~# squeue -j 154415 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 154415 standard genedrop username R 1:31 1 hpc020 Also, possibly related, we had a slurmdbd crash before this changed. We run Ubuntu Server 24.04 LTS with Slurm 24.05.4, using a MariaDB accounting database hosted on the same machine as the Slurm controller. Does anyone here have any ideas? Best, Pierre -- Pierre Abele, M.Sc. HPC Administrator Max-Planck-Institute for Evolutionary Anthropology Department of Primate Behavior and Evolution Deutscher Platz 6 04103 Leipzig Room: U2.80 E-Mail: pierre_abele(a)eva.mpg.de Phone: +49 (0) 341 3550 245

1 0

Re: slurmctld HA ; backup controller doesn't schedule and start any job
by Hiromasa Watanabe 10 Apr '25

10 Apr '25

Hi all, Fortunately I solved this problem by changing Slurm version from 22.05.9 to 23.11.10. With Slurm 23.11.10, after stopping the primary slurmctld and slurmdbd, when I submit a job with sbatch while backup slurmctld and slurmdbd are running, the job becomes scheduled and runs. I don't know why, but this is OK. Thanks, Hiro

1 0

errors while trying to setup slurmdbd.
by Steven Jones 09 Apr '25

09 Apr '25

root@vuwunicohpcdbp1 ~]# systemctl status slurmdbd × slurmdbd.service - Slurm DBD accounting daemon Loaded: loaded (/usr/lib/systemd/system/slurmdbd.service; enabled; preset: disabled) Active: failed (Result: exit-code) since Thu 2025-04-10 10:28:52 NZST; 2min 33s ago Duration: 85ms Process: 2413 ExecStart=/usr/sbin/slurmdbd -D -s $SLURMDBD_OPTIONS (code=exited, status=1/FAILURE) Main PID: 2413 (code=exited, status=1/FAILURE) CPU: 11ms Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz systemd[1]: Started Slurm DBD accounting daemon. Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz slurmdbd[2413]: slurmdbd: accounting_storage/as_mysql: _check_mysql_concat_is_sane: MySQL server version > Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz slurmdbd[2413]: slurmdbd: accounting_storage/as_mysql: init: Accounting storage MYSQL plugin loaded Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz slurmdbd[2413]: slurmdbd: fatal: This host not configured to run SlurmDBD ((vuwunicohpcdbp1 or vuwunicohp> Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz systemd[1]: slurmdbd.service: Main process exited, code=exited, status=1/FAILURE Apr 10 10:28:52 vuwunicohpcdbp1.ods.vuw.ac.nz systemd[1]: slurmdbd.service: Failed with result 'exit-code'. lines 1-14/14 (END) This is a current RHEL9.5 [root@vuwunicohpcdbp1 ~]# rpm -qi mariadb Name : mariadb Epoch : 3 Version : 10.5.27 Release : 1.el9_5 Architecture: x86_64 Install Date: Fri 04 Apr 2025 11:13:14 AM NZDT Group : Unspecified Size : 18784786 License : GPLv2 and LGPLv2 Signature : RSA/SHA256, Fri 06 Dec 2024 10:47:26 PM NZDT, Key ID 199e2f91fd431d51 Source RPM : mariadb-10.5.27-1.el9_5.src.rpm Build Date : Thu 05 Dec 2024 03:22:03 PM NZDT Build Host : x86-64-04.build.eng.rdu2.redhat.com Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> Vendor : Red Hat, Inc. URL : http://mariadb.org Summary : A very fast and robust SQL database server Description : MariaDB is a community developed fork from MySQL - a multi-user, multi-threaded SQL database server. It is a client/server implementation consisting of a server daemon (mariadbd) and many different client programs and libraries. The base package contains the standard MariaDB/MySQL client programs and utilities. [root@vuwunicohpcdbp1 ~]# ========= # Example slurmdbd.conf file. # # See the slurmdbd.conf man page for more information. # # Archive info #ArchiveJobs=yes #ArchiveDir="/tmp" #ArchiveSteps=yes #ArchiveScript= #JobPurge=12 #StepPurge=1 # # Authentication info AuthType=auth/munge #AuthInfo=/var/run/munge/munge.socket.2 # # slurmDBD info DbdAddr=127.0.0.1 #DbdAddr=localhost DbdHost=vuwunicoslurmrp1.ods.vuw.ac.nz #DbdHost=localhost DbdPort=6819 SlurmUser=slurm #MessageTimeout=300 DebugLevel=verbose #DefaultQOS=normal,standby LogFile=/var/log/slurm/slurmdbd.log PidFile=/var/run/slurmdbd/slurmdbd.pid #PluginDir=/usr/lib/slurm #PrivateData=accounts,users,usage,jobs #TrackWCKey=yes # # Database info StorageType=accounting_storage/mysql #StorageHost=localhost StorageHost=127.0.0.1 #StoragePort=1234 StoragePort=3306 #StoragePort=6819 #StoragePass=xxxxx StoragePass=xxxxxx #StoragePass=aPIkaEqxLrA StorageUser=slurm StorageLoc=slurm_acct_db #ssj # ##ssj fini ============ This ran in test just fine but the slurm version is slightly newer in this case, [root@vuwunicohpcdbp1 admjonesst1]# rpm -qa |grep slurm slurm-24.11.3-1.el9.x86_64 slurm-slurmdbd-24.11.3-1.el9.x86_64 [root@vuwunicohpcdbp1 admjonesst1]# rpm -qi slurm-slurmdbd-24.11.3-1.el9.x86_64 Name : slurm-slurmdbd Version : 24.11.3 Release : 1.el9 Architecture: x86_64 Install Date: Mon 07 Apr 2025 01:58:32 PM NZST Group : System Environment/Base Size : 2959288 License : GPLv2+ Signature : (none) Source RPM : slurm-24.11.3-1.el9.src.rpm Build Date : Mon 07 Apr 2025 12:54:21 PM NZST Build Host : vuwunicoslurmp1.ods.vuw.ac.nz URL : https://slurm.schedmd.com/ Summary : Slurm database daemon Description : Slurm database daemon. Used to accept and process database RPCs and upload database changes to slurmctld daemons on each cluster [root@vuwunicohpcdbp1 admjonesst1]# regards Steven

2 2

pam error, related to accounting?
by David Bremner 09 Apr '25

09 Apr '25

Recently I enabled accounting on my tiny (1 compute node, one head node) slurm cluster. slurmdbd.conf looks like AuthType=auth/munge DbdHost=vertex DbdPort=6819 SlurmUser=slurm StorageHost=localhost StorageType=accounting_storage/mysql StorageUser=slurm StoragePass=#elided# StoragePort=3306 StorageLoc=slurm_acct_db PurgeEventAfter=30days PurgeJobAfter=30days PurgeResvAfter=30days PurgeStepAfter=30days PurgeSuspendAfter=30days Some potentially relevant parts of slurmd.conf are: UsePAM=1 ProctrackType=proctrack/cgroup SelectTypeParameters=CR_Core_Memory AccountingStorageType=accounting_storage/slurmdbd ClusterName=simplex JobCompType=jobcomp/none JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/cgroup Since I enabled accounting the output from every job (whether started by srun or sbatch) finishes with slurmstepd-simplex: error: pam_close_session: Cannot make/remove an entry for the specified session /etc/pamd.d/slurm has the following content auth required pam_localuser.so auth required pam_shells.so account required pam_unix.so account required pam_access.so session required pam_unix.so session optional pam_systemd.so Accounting seems to be working OK. Does anyone know what PAM related code paths could be triggered by enabling accounting? d

1 0

2025

2024

slurm-users April 2025 ----- 2025 ----- June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users April 2025