- slurm-users - lists.schedmd.com

Help with slurmdbd and Slurmctld
by stth 18 Jun '24

18 Jun '24

Hello, I wanted help with a problem I have. I just updated my operation system and the slurmdbd and slurmctld is not working anymore. I get this Errors: slurmdbd: fatal: Database schema is too old for this version of Slurm to upgrade. slurmctld: fatal: Can not recover last_tres state, incompatible version, got 9472 need >= 9728 <= 10240,start with '-i' to ignore this. Warning: using -i will lose the data that can't be recovered. I hope someone can help !

3 3

Unsupported RPC version by slurmctld 19.05.3 from client slurmd 22.05.11
by aladin＠jscc.ru 17 Jun '24

17 Jun '24

Hello! I have a question. I have the server 19.05.3. No chance to upgrade it. Have I any chance to connect new client 22.05.11? Here is log from server: [2024-06-14T15:20:40.345] debug: Unsupported RPC version 9728 msg type 0(0) [2024-06-14T15:20:40.345] error: g_slurm_auth_unpack: remote plugin_id 12340565 not found [2024-06-14T15:20:40.345] error: slurm_unpack_received_msg: Invalid Protocol Version 9728 from uid=-1 at 10.82.0.6:52170 [2024-06-14T15:20:40.345] … [View More]

5 4

Node (anti?) Feature / attribute
by David Magda 17 Jun '24

17 Jun '24

Hello, What I’m looking for is a way for a node to continue to be in the same partition, and have the same QoS(es), but only be chosen if a particular capability is being asked for. This is because we are rolling something (OS upgrade) out slowly to a small batch of nodes at first, and then more and more over time, and do not want to interrupt users’ workflows: we want them to default the ‘current’ nodes and only land on the ‘special’ ones if requested. (At a certain point the ‘special’ ones … [View More]

4 6

Issue with starting slurmctld
by Rafał Lalik 17 Jun '24

17 Jun '24

Hello, I have encountered issues with running slurmctld. From logs, I see these errors: [2024-06-14T17:37:57.587] slurmctld version 24.05.0 started on cluster laura [2024-06-14T17:37:57.587] error: plugin_load_from_file: dlopen(/usr/lib64/slurm/jobacct_gather_cgroup.so): /usr/lib64/slurm/jobacct_gather_cgroup.so: undefined symbol: xcpuinfo_init [2024-06-14T17:37:57.587] error: Couldn't load specified plugin name for jobacct_gather/cgroup: Dlopen of plugin file failed [2024-06-14T17:37:57.… [View More]

2 2

Ill-formed config from the online configurator
by Rafał Lalik 17 Jun '24

17 Jun '24

The online configurator for JobAcctGatherType set to anything that none, generates config file with a such entry: #JobAcctGatherTypejobacct_gather/cgroup= or #JobAcctGatherTypejobacct_gather/linux= Regards, Rafał

1 0

cgroup issue on non-systemd system
by Rafał Lalik 14 Jun '24

14 Jun '24

Hello, per documentation, it is possible to run slurm on non systemd system with IgnoreSystemd=yes in cgroup.conf. However I had an error with slurmd: error: common_file_write_content: unable to open '/sys/fs/cgroup/system.slice/cgroup.subtree_control' for writing: No such file or directory error: Cannot enable cpuset in /sys/fs/cgroup/system.slice/cgroup.subtree_control: No such file or directory error: common_file_write_content: unable to open '/sys/fs/cgroup/system.slice/cgroup.… [View More]subtree_control' for writing: No such file or directory error: Cannot enable memory in /sys/fs/cgroup/system.slice/cgroup.subtree_control: No such file or directory error: common_file_write_content: unable to open '/sys/fs/cgroup/system.slice/cgroup.subtree_control' for writing: No such file or directory error: Cannot enable cpu in /sys/fs/cgroup/system.slice/cgroup.subtree_control: No such file or directory error: Could not create scope directory /sys/fs/cgroup/system.slice/slurmstepd.scope: No such file or directory error: Couldn't load specified plugin name for cgroup/v2: Plugin init() callback failed error: cannot create cgroup context for cgroup/v2 error: Unable to initialize cgroup plugin error: slurmd initialization failed I could link it to the fact that '/sys/fs/cgroup/system.slice' does not exists when creating '/sys/fs/cgroup/system.slice/slurmstepd.scope' subdir. I propose small patch to the slurm which allows to create the path recursively (aka `mkdir -p` from shell): (also here: https://github.com/rlalik/slurm/tree/recurse_mkdir) diff --git a/src/plugins/cgroup/v2/cgroup_v2.c b/src/plugins/cgroup/v2/cgroup_v2.c index a18b9e62bd..6b7e2a2e4a 100644 --- a/src/plugins/cgroup/v2/cgroup_v2.c +++ b/src/plugins/cgroup/v2/cgroup_v2.c @@ -743,11 +743,40 @@ static int _init_stepd_system_scope(pid_t pid) return SLURM_SUCCESS; } +/* This create path recursively.*/ +bool recurse_mkdir(const char *dirname, int mode) +{ + const char *p = dirname; + char *temp = calloc(1, strlen(dirname)+1); + int ret = 0; + + while ((p = strchr(p, '/')) != NULL) { + /* Skip empty elements. Could be just multiple separators + * which is okay. */ + if (p != dirname && *(p-1) == '/') { + p++; + continue; + } + /* Put the path up to this point into a temporary to + * pass to the make directory function. */ + memcpy(temp, dirname, p-dirname); + temp[p-dirname] = '\0'; + p++; + if ((ret = mkdir(temp, mode)) != 0) { + if (errno != EEXIST) { + break; + } + } + } + free(temp); + return ret; +} + static int _init_new_scope(char *scope_path) { int rc; - rc = mkdir(scope_path, 0755); + rc = recurse_mkdir(scope_path, 0755); if (rc && (errno != EEXIST)) { error("Could not create scope directory %s: %m", scope_path); return SLURM_ERROR; [View Less]

1 0

Debian RPM build for arm64?
by Christopher Harrop - NOAA Affiliate 14 Jun '24

14 Jun '24

Hello, Are the instructions for building Debian RPMs found at https://slurm.schedmd.com/quickstart_admin.html#debuild expected to work on ARM machines? I am having trouble with the "debuild -b -uc -us” step. #10 29.01 configure: exit 1 #10 29.01 dh_auto_configure: error: cd obj-aarch64-linux-gnu && ../configure --build=aarch64-linux-gnu --prefix=/usr --includedir=\${prefix}/include --mandir=\${prefix}/share/man --infodir=\${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --… [View More]disable-silent-rules --libdir=\${prefix}/lib/aarch64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --sysconfdir=/etc/slurm --disable-debug --with-slurmrestd --with-pmix --enable-pam --with-systemdsystemunitdir=/lib/systemd/system/ SUCMD=/bin/su SLEEP_CMD=/bin/sleep returned exit code 1 #10 29.01 make[1]: *** [debian/rules:21: override_dh_auto_configure] Error 25 #10 29.01 make[1]: Leaving directory '/tmp/slurm-23.11.7' #10 29.02 make: *** [debian/rules:6: build] Error 2 #10 29.02 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2 #10 29.02 debuild: fatal error at line 1182: #10 29.02 dpkg-buildpackage -us -uc -ui -b failed #10 ERROR: process "/bin/sh -c cd /tmp && wget https://download.schedmd.com/slurm/slurm-23.11.7.tar.bz2 && tar -xaf slurm-23.11.7.tar.bz2 && cd slurm-23.11.7 && mk-build-deps -t \"apt-get -o Debug::pkgProblemResolver=yes -y\" -i debian/control && debuild -b -uc -us && cd .. && ARCH=$(dpkg --print-architecture) && dpkg --install slurm-smd_23.11.7-1_${ARCH}.deb && dpkg --install slurm-smd-client_23.11.7-1_${ARCH}.deb && dpkg --install slurm-smd-dev_23.11.7-1_${ARCH}.deb && dpkg --install slurm-smd-doc_23.11.7-1_all.deb && dpkg --install slurm-smd-libnss-slurm_23.11.7-1_${ARCH}.deb && dpkg --install slurm-smd-libpam-slurm-adopt_23.11.7-1_${ARCH}.deb && dpkg --install slurm-smd-libpmi0_23.11.7-1_${ARCH}.deb && dpkg --install slurm-smd-libpmi2-0_23.11.7-1_${ARCH}.deb && dpkg --install slurm-smd-libslurm-perl_23.11.7-1_${ARCH}.deb && dpkg --install slurm-smd-sackd_23.11.7-1_${ARCH}.deb && dpkg --install slurm-smd-sview_23.11.7-1_${ARCH}.deb" did not complete successfully: exit code: 29 Chris --------------------------------------------------------------------------------------------------- Christopher W. Harrop voice: (720) 649-0316 NOAA Global Systems Laboratory, R/GSL6 fax: (303) 497-7259 325 Broadway Boulder, CO 80303 [View Less]

3 3

Re: slurmstepd: error: task_g_set_affinity: Operation not permitted
by Christopher Harrop - NOAA Affiliate 14 Jun '24

14 Jun '24

There is a permission problem somewhere, but I don’t know where. If I run as root, it works: admin@slurmfrontend:~$ srun hostname srun: error: task 0 launch failed: Slurmd could not execve job slurmstepd: error: task_g_set_affinity: Operation not permitted slurmstepd: error: _exec_wait_child_wait_for_parent: failed: No error admin@slurmfrontend:~$ sudo srun hostname slurmnode1 admin@slurmfrontend:~$ sudo srun -N 3 hostname slurmnode1 slurmnode3 slurmnode2 admin@slurmfrontend:~$ Chris ------… [View More]

2 1

Re: Limit GPU depending on type
by Gestió Servidors 14 Jun '24

14 Jun '24

Hi, because of my real scenario (in mi first post I explained my testing scenario), with several differents users of differents types (researchers, university students and/or teachers, etc), I have distributed my GPUs in 3 differents partitions: * PartitionName=cuda-staff.q Nodes=gpu-[1-4] OverSubscribe=No MaxTime=INFINITE State=UP AllocNodes=node[0-22],node-login,node-login-bak AllowGroups=caos,profesor * PartitionName=cuda-int.q Nodes=gpu-[2,4] OverSubscribe=No MaxTime=30:00 State=… [View More]

1 0

Slurm version 23.11.8 is now available
by Marshall Garey 13 Jun '24

13 Jun '24

We are pleased to announce the availability of Slurm version 23.11.8. The 23.11.8 release fixes some potential crashes in slurmctld, slurmrestd, and slurmd when using less common features; two issues in auth/slurm; and a few other minor bugs. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Marshall > -- Fix slurmctld crash when reconfiguring with a PrologSlurmctld is running. > -- Fix slurmctld crash after a job has been resized. > -- Fix slurmctld and … [View More]slurmdbd potentially stopping instead of performing a > logrotate when recieving SIGUSR2 when using auth/slurm. > -- Fix not having a disabled value for keepalive CommunicationParameters in > slurm.conf when these parameters are not set. This can log an error when > setting a socket, for example during slurmdbd registration with ctld. > -- switch/hpe_slingshot - Fix slurmctld crash when upgrading from 23.02. > -- Fix "Could not find group" errors from validate_group() when using > AllowGroups with large /etc/group files. > -- slurmrestd - Prevent a slurmrestd segfault when parsing the crontab field, > which was never usable. Now it explicitly ignores the value and emits a > warning if it is used for the following endpoints: > 'POST /slurm/v0.0.39/job/{job_id}' > 'POST /slurm/v0.0.39/job/submit' > 'POST /slurm/v0.0.40/job/{job_id}' > 'POST /slurm/v0.0.40/job/submit' > -- Fix getting user environment when using sbatch with "--get-user-env" or > "--export=" when there is a user profile script that reads /proc. > -- Prevent slurmd from crashing if acct_gather_energy/gpu is configured but > GresTypes is not configured. > -- Do not log the following errors when AcctGatherEnergyType plugins are used > but a node does not have or cannot find sensors: > "error: _get_joules_task: can't get info from slurmd" > "error: slurm_get_node_energy: Zero Bytes were transmitted or received" > However, the following error will continue to be logged: > "error: Can't get energy data. No power sensors are available. Try later" > -- Fix cloud nodes not being able to forward to nodes that restarted with new > IP addresses. > -- sacct - Fix printing of job group for job steps. > -- Fix error in scrontab jobs when using slurm.conf:PropagatePrioProcess=1. > -- Fix slurmctld crash on a batch job submission with "--nodes 0,...". > -- Fix dynamic IP address fanout forwarding when using auth/slurm. [View Less]

1 0

slurmstepd: error: task_g_set_affinity: Operation not permitted
by Christopher Harrop - NOAA Affiliate 13 Jun '24

13 Jun '24

Hi, I am building a containerized Slurm cluster with Ubuntu 20.04 and have it almost working. The daemons start, and an “sinfo” command shows compute nodes up and available: admin@slurmfrontend:~$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST slurmpar* up infinite 3 idle slurmnode[1-3] admin@slurmfrontend:~$ However, if I try to use “srun” to test a job submission it fails saying it could not execve the job: admin@slurmfrontend:~$ srun hostname srun: error: task 0 … [View More]launch failed: Slurmd could not execve job slurmstepd: error: task_g_set_affinity: Operation not permitted slurmstepd: error: _exec_wait_child_wait_for_parent: failed: No error admin@slurmfrontend:~$ If I go to the slurmnode1 container where the job should run, and look at the slurmd log, all I see is this: admin@slurmnode1:/$ sudo cat /var/log/slurmd.log [2024-06-13T14:58:36.238] CPU frequency setting not configured for this node [2024-06-13T14:58:36.239] warning: Core limit is only 0 KB [2024-06-13T14:58:36.239] slurmd version 23.11.7 started [2024-06-13T14:58:36.243] slurmd started on Thu, 13 Jun 2024 14:58:36 +0000 [2024-06-13T14:58:36.243] CPUs=8 Boards=1 Sockets=8 Cores=1 Threads=1 Memory=47926 TmpDisk=59767 Uptime=71713 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) [2024-06-13T14:58:47.230] launch task StepId=1.0 request from UID:1000 GID:1000 HOST:172.20.0.2 PORT:50618 [2024-06-13T14:58:47.230] task/affinity: lllp_distribution: JobId=1 implicit auto binding: sockets,one_thread, dist 8192 [2024-06-13T14:58:47.230] task/affinity: _task_layout_lllp_cyclic: _task_layout_lllp_cyclic [2024-06-13T14:58:47.230] task/affinity: _lllp_generate_cpu_bind: _lllp_generate_cpu_bind jobid [1]: mask_cpu,one_thread, 0x01 [2024-06-13T14:58:47.243] [1.0] error: task_g_set_affinity: Operation not permitted [2024-06-13T14:58:47.243] [1.0] error: _exec_wait_child_wait_for_parent: failed: No error [2024-06-13T14:58:47.244] [1.0] error: job_manager: exiting abnormally: Slurmd could not execve job [2024-06-13T14:58:47.247] [1.0] stepd_cleanup: done with step (rc[0xfb4]:Slurmd could not execve job, cleanup_rc[0xfb4]:Slurmd could not execve job) admin@slurmnode1:/$ I’ve installed by following the instructions for building/installing the Debian RPMs and can see that all the daemons are up and running. I have this slurm.conf (on all nodes): admin@slurmfrontend:~$ grep -v '#' /etc/slurm/slurm.conf ClusterName=cluster SlurmctldHost=slurmmaster MpiDefault=none ProctrackType=proctrack/linuxproc ReturnToService=1 SlurmdParameters=config_overrides SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=root StateSaveLocation=/var/spool/slurmctld SwitchType=switch/none TaskPlugin=task/affinity InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 SchedulerType=sched/backfill SelectType=select/cons_tres AccountingStorageType=accounting_storage/none JobCompType=jobcomp/none JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/none SlurmctldDebug=info SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurmd.log NodeName=slurmnode[1-3] CPUs=8 State=UNKNOWN PartitionName=slurmpar Nodes=ALL Default=YES MaxTime=INFINITE State=UP admin@slurmfrontend:~$ And I have this group.conf (on all nodes): admin@slurmfrontend:~$ grep -v '#' /etc/slurm/cgroup.conf CgroupPlugin=cgroup/v1 ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes admin@slurmfrontend:~$ Does anyone have any clues about where to look for why “srun” can’t run a job and where the "task_g_set_affinity: Operation not permitted” may be coming from? Chris --------------------------------------------------------------------------------------------------- Christopher W. Harrop voice: (720) 649-0316 NOAA Global Systems Laboratory, R/GSL6 fax: (303) 497-7259 325 Broadway Boulder, CO 80303 [View Less]

1 0

Limit GPU depending on type
by Gestió Servidors 13 Jun '24

13 Jun '24

Hello, I would like to know if it would be possible to limit, using "sacctmgr", use of a certain type of GPU according the name I have assigned in "gres.conf" file. For example, my small cluster has 3 GPUs nodes sharing 2 GPUs each one. Two of that GPUs are the same model but they are located in different servers. Because of my scenario, I would like to limit users to user only one of that GPU type and not allowing to use both of them. For example: * gpu-node-1: * GTX1080 * … [View More]

2 1

srun hostname - Socket timed out on send/recv operation
by Arnuld 11 Jun '24

11 Jun '24

I have two machines. When I run "srum hostname" on one machine (it's both a controller and a node) then I get the hostname fine but I get socket timed out error in these two situations: 1) "srun hostname" on 2nd machine (it's a node) 2) "srun -N 2 hostname" on controller "scontrol show node" shows both mach2 and mach4. "sinfo" shows both nodes too. Also the job gets stuck forever in CG state after the error. Here is the output: $ srun -N 2 hostname mach2 srun: error: slurm_receive_msgs: [[… [View More]

1 1

sbatch: Node count specification invalid - when only specifying --ntasks
by George Leaver 11 Jun '24

11 Jun '24

Hello, Previously we were running 22.05.10 and could submit a "multinode" job using only the total number of cores to run, not the number of nodes. For example, in a cluster containing only 40-core nodes (no hyperthreading), Slurm would determine two nodes were needed with only: sbatch -p multinode -n 80 --wrap="...." Now in 23.02.1 this is no longer the case - we get: sbatch: error: Batch job submission failed: Node count specification invalid At least -N 2 is must be used (-n 80 can be … [View More]

2 3

Issue about selecting cpus for optimization
by Purvesh Parmar 10 Jun '24

10 Jun '24

Hi, We have 16 nodes cluster with DGX-A100 (80 GB). We have 128 cores of each node separated in to a separate partition for cpu only jobs and 8 GPUs and 128 cores in other partitions for cpugpu jobs. We want to ensure that only selected 128 cores should be part of the cpu partition. (NUMA / Symmetry) for optimization. How to achieve it? cores parameter in gres.conf help? Regards, Purvesh

1 0

scontrol create partition fails
by Long, Daniel S. 10 Jun '24

10 Jun '24

Hi, I need to temporarily dedicated one of our compute nodes to a single account. To do this, I was going to create a new partition but I'm running into an error where scontrol create partition outputs "scontrol: error: Invalid input: partition Request aborted" regardless of what parameters I give it. As far as I can tell, this should be allowed; the man page for scontrol has a whole section titled "Partitions - Specficiations for Create, Update, and Delete Commands". What am I missing? … [View More]

2 2

Software builds using slurm
by Duane Ellis 10 Jun '24

10 Jun '24

I have been lurking here for a while hoping to see some examples that would help but have not fit several months We have a slurm system setup for xilnix FPGA builds (hdl) I want to use this for sw builds too What I seem to see is slurm talks about cpus, GPUs and memory etc I am looking for a “run my make file (or shell script) on any available node” In our case we have 3 top level jobs A B and C These can all run in parallel and are independent (ie bootloader, linux kernel, and the Linux … [View More]

3 2

Re: sbatch: Node count specification invalid - when only specifying --ntasks
by George Leaver 10 Jun '24

10 Jun '24

Noam, Thanks for the suggestion but no luck: sbatch -p multinode -n 80 --ntasks-per-core=1 --wrap="..." sbatch: error: Batch job submission failed: Node count specification invalid sbatch -p multinode -n 2 -c 40 --ntasks-per-core=1 --wrap="..." sbatch: error: Batch job submission failed: Node count specification invalid sbatch -p multinode -N 2 -n 80 --ntasks-per-core=1 --wrap="..." Submitted batch job I guess that the MinNodes=2 in the partition def is now being enforced somewhat more … [View More]

1 0

cpu distribution question
by Alan Stange 08 Jun '24

08 Jun '24

All, I have a very simple slurm cluster. It's just a single system with 2 sockets and 16 cores in each socket. I would like to be able to submit a simple task into this cluster, and to have the cpus assigned to that task allocated round robin across the two sockets. Everything I try is putting all the cpus for this single task on the same socket. I have not specified any CpuBind options in the slurm.conf file. For example, if I try $ srun -c 4 --pty bash … [View More]

3 3

maxrss reported by sachet is wrong
by Feng Zhang 07 Jun '24

07 Jun '24

Hi All, I am having trouble calculating the real RSS memory usage by some kind of users' jobs. Which the sacct returned wrong numbers. Rocky Linux release 8.5, Slurm 21.08 (slurm.conf) ProctrackType=proctrack/cgroup JobAcctGatherType=jobacct_gather/linux The troubling jobs are like: 1. python spawn multithreading 96 threads; 2. Each thread uses SKlearn which again spawns 96 threads using openmp. Which is obviously over running the node, and I want to address it. The node has 300GB RAM, … [View More]

2 1

need to set From: address for slurm
by Vanhorn, Mike 07 Jun '24

07 Jun '24

All, When the slurm daemon is sending out emails, they are coming from “slurm(a)servername.subdomain.domain.edu<mailto:slurm@servername.subdomain.domain.edu>”. This has worked okay in the past, but due to a recent mail server change (over which I have no control whatsoever) this will no longer work. Now, the From: address is going to have to be something like “slurm-servername(a)domain.xn--edu-9o0a , or, at least something that ends in “(a)domain.xn--edu-9o0a (the subdomain being present … [View More]

2 2

Re: Not being able to ssh to node with running job
by Ratnasamy, Fritz 06 Jun '24

06 Jun '24

So the squeue issue was resolved and was due to the partition being hidden. Unhiding it solves the problem. However, the ssh issue remains (looks like both were separate issues). The pam_slurm_adopt is working on all the other nodes but not on the new ones. Any idea how to solve this? Best, *Fritz Ratnasamy* Data Scientist Information Technology On Thu, Jun 6, 2024 at 2:11 PM Ratnasamy, Fritz via slurm-users < slurm-users(a)lists.schedmd.com> wrote: > As admin on the cluster, … [View More]

1 0

Re: JWKS File Update - Reread File
by Pedro Daniel Da Rocha Santos 06 Jun '24

06 Jun '24

Thanks for the reply. I was not aware of that change in 23.11. I am still using Slurm 23.02.7. Best regards, Pedro D.R. Santos Specialist, Ph.D., High Performance Computing Vestas Technology Centre Porto Vestas Wind Systems A/S Vestas will as part of your communication, interaction and business relationship with us collect and process personal data about you. You can read more about Vestas' collection and processing of your personal data and your rights as a data subject in our Privacy … [View More]

1 0

Not being able to ssh to node with running job
by Ratnasamy, Fritz 06 Jun '24

06 Jun '24

As admin on the cluster, we do not observe any issue on our newly added gpu nodes. However, for regular users, they are not seeing their jobs running on these gpu nodes when running squeue -u <username> ( it is however showing as running status with sacct) and they are not able to ssh to these newly added nodes when they have a running job on it. I am not sure if these 2 are related (not being to ssh to mgpu node with a running job on it and not listing a job with squeue for a user on … [View More]

1 0

JWKS File Update - Reread File
by Pedro Daniel Da Rocha Santos 06 Jun '24

06 Jun '24

Hello everyone, I am currently implementing a service to monitor for jwks key rotation. If a change is detected, the jwks file is updated. Is it enough to use "scontrol reconfigure" to reread the jwks file? Or does it require a full restart of the slurmctl daemon? Thanks in advance for the inputs. Best regards, Pedro D.R. Santos Specialist, Ph.D., High Performance Computing Vestas Technology Centre Porto Vestas Wind Systems A/S +351 924 241 907 pddro(a)vestas.com<mailto:pddro@vestas.com… [View More]

2 1

diagnosing why interactive/non-interactive job waits are so long with State=MIXED
by Robert Kudyba 05 Jun '24

05 Jun '24

At the moment we have 2 nodes that are having long wait times. Generally this is when the nodes are fully allocated. What would be the other reasons if there is still enough available memory and CPU available, that a job would take so long? Slurm version is 23.02.4 via Bright Computing. Note the compute nodes have hyperthreading enabled but that should be irrelevant. Is there a way to determine what else could be holding jobs up? srun --pty -t 0-01:00:00 --nodelist=node001 --gres=gpu:1 -A … [View More]

3 5

Re: Problems with gres.conf
by Gestió Servidors 05 Jun '24

05 Jun '24

Hi, my GPU testing system (named “gpu-node”) is a simple computer with one socket and a processor " Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz". Executing "lscpu", I can see there are 4 cores per socket, 2 threads per core and 8 CPUs: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): … [View More]

1 0

Problems with gres.conf
by Gestió Servidors 04 Jun '24

04 Jun '24

Hello, I am trying to rewrite my gres.conf file. Before changes, this file was just like this: NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-3 … [View More]AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-4 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-7 # you can seee that nodes node-gpu-1 and node-gpu-2 have two GPUs each one, whereas nodes node-gpu-3 and node-gpu-4 have only one GPU each one And my slurmd.conf was this: [...] AccountingStorageTRES=gres/gpu GresTypes=gpu NodeName=node-gpu-1 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 CPUs=8 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 NodeName=node-worker-[0-22] CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=47000 [...] With this configuration, all seems works fine, except slurmctld.log reports: [...] error: _node_config_validate: gres/gpu: invalid GRES core specification (0-11) on node node-gpu-3 error: _node_config_validate: gres/gpu: invalid GRES core specification (12-23) on node node-gpu-1 error: _node_config_validate: gres/gpu: invalid GRES core specification (12-23) on node node-gpu-2 error: _node_config_validate: gres/gpu: invalid GRES core specification (0-7) on node node-gpu-4 [...] However, even these errors, users can submit jobs and request GPUs resources. Now, I have tried to reconfigure gres.conf and slurmd.conf in this way: gres.conf: Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 # there is no NodeName attribute slurmd.conf: [...] NodeName=node-gpu-1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 NodeName=node-worker-[0-22] SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=47000 # there is no CPUs attribute [...] With this new configuration, nodes with GPU start correctly slurmd.service daemon, but nodes without GPU (node-worker-[0-22]) can't start slurmd.service daemon and returns this error: [...] error: Waiting for gres.conf file /dev/nvidia0 fatal: can't stat gres.conf file /dev/nvidia0: No such file or directory [...] It seems SLURM is waiting that "node-workers" have also an nvidia GPU but not, theses nodes haven't GPU... So, where is my configuration error? I have read in https://slurm.schedmd.com/gres.conf.html about syntax and examples but it seems I'm doing some wrong. Thanks!! [View Less]

2 1

Tracking GPU utilization with sacct
by Robertson, Jordan 03 Jun '24

03 Jun '24

Hi all, I’m trying to pull (and understand) some GPU usage metrics for historical purposes, and dug into sacct’s TRES reporting a bit. We have AccountingStorageTRES=gres/gpu set in slurm.conf so we do see gres/gpuutil and gres/gpumem numbers available, but I’m struggling to find Slurm-side documentation that describes the units of these values. In looking at the code for gpu_nvml.c it seems the “nvmlDeviceGetProcessUtilization” function is being used and returns units in percentages, but I’m … [View More]

1 0

sbatch and --nodes
by Michael DiDomenico 31 May '24

31 May '24

its friday and i'm either doing something silly or have a misconfig somewhere, i can't figure out which when i run sbatch --nodes=1 --cpus-per-task=1 --array=1-100 --output test_%A_%a.txt --wrap 'uname -n' sbatch doesn't seem to be adhering to the --nodes param. when i look at my output files it's spreading them across more nodes. in the simple case above it's 50/50, but if i through a random sleep in, it'll be more. and if i expand the array it'll use even more nodes. i'm using con/tres … [View More]

2 1

Re: Container Jobs "hanging"
by Joshua Randall 31 May '24

31 May '24

Hi Sean, I appear to be having the same issue that you are having with OCI container jobs running forever / appearing to hang. I haven't figured it out yet, but perhaps we can compare notes and determine what aspect of configuration we both share. Like you, I was following the examples in https://slurm.schedmd.com/containers.html and originally encountered the issue with an alpine container image running the `uptime` command, but I have also confirmed the issue with other images including … [View More]ubuntu and with other processes. I always get the same results - the container process runs to completion and exits, but then the slurm job continues to run until it is cancelled or killed. I have slurm v23.11.6 and am using the nvidia-container-runtime, what slurm version and runtime are you using? My oci.conf is: ``` $ cat /etc/slurm/oci.conf EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)=" RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)=" RunTimeQuery="nvidia-container-runtime --rootless=true --root=/run/user/%U/ state %n.%u.%j.%s.%t" RunTimeKill="nvidia-container-runtime --rootless=true --root=/run/user/%U/ kill -a %n.%u.%j.%s.%t" RunTimeDelete="nvidia-container-runtime --rootless=true --root=/run/user/%U/ delete --force %n.%u.%j.%s.%t" RunTimeRun="nvidia-container-runtime --rootless=true --root=/run/user/%U/ run %n.%u.%j.%s.%t -b %b" ``` Hope that we can get to the bottom of this and resolve our issues with OCI containers! Josh. --- Hello. I am new to this list and Slurm overall. I have a lot of experience in computer operations, including Kubernetes, but I am currently exploring Slurm in some depth. I have set up a small cluster and, in general, have gotten things working, but when I try to run a container job, it runs the command but then appears to hang as if the job container is still running. So, running the following works, but it never returns to the prompt unless I use [Control-C]. $ srun --container /shared_fs/shared/oci_images/alpine uptime 19:21:47 up 20:43, 0 users, load average: 0.03, 0.25, 0.15 I'm unsure if something is misconfigured or if I'm misunderstanding how this should work, but any help and/or pointers would be greatly appreciated. Thanks! Sean -- slurm-users mailing list -- slurm...(a)lists.schedmd.com To unsubscribe send an email to slurm-us...(a)lists.schedmd.com -- Dr. Joshua C. Randall Principal Software Engineer Altos Labs email: jrandall(a)altoslabs.com -- Altos Labs UK Limited | England | Company reg 13484917 Registered address: 3rd Floor 1 Ashley Road, Altrincham, Cheshire, United Kingdom, WA14 2DT [View Less]

1 1

slurmdbd not connecting to mysql (mariadb)
by Radhouane Aniba 31 May '24

31 May '24

Hi everyone I am trying to get slurmdbd to run on my local home server but I am really struggling. Note : am a novice slurm user my slurmdbd always times out even though all the details in the conf file are correct My log looks like this [2024-05-29T20:51:30.088] Accounting storage MYSQL plugin loaded [2024-05-29T20:51:30.088] debug2: ArchiveDir = /tmp [2024-05-29T20:51:30.088] debug2: ArchiveScript = (null) [2024-05-29T20:51:30.088] debug2: AuthAltTypes = (null) [2024-05-29T20:51:30.088] … [View More]debug2: AuthInfo = (null) [2024-05-29T20:51:30.088] debug2: AuthType = auth/munge [2024-05-29T20:51:30.088] debug2: CommitDelay = 0 [2024-05-29T20:51:30.088] debug2: DbdAddr = localhost [2024-05-29T20:51:30.088] debug2: DbdBackupHost = (null) [2024-05-29T20:51:30.088] debug2: DbdHost = head-node [2024-05-29T20:51:30.088] debug2: DbdPort = 7032 [2024-05-29T20:51:30.088] debug2: DebugFlags = (null) [2024-05-29T20:51:30.088] debug2: DebugLevel = 6 [2024-05-29T20:51:30.088] debug2: DebugLevelSyslog = 10 [2024-05-29T20:51:30.088] debug2: DefaultQOS = (null) [2024-05-29T20:51:30.088] debug2: LogFile = /var/log/slurmdbd.log [2024-05-29T20:51:30.088] debug2: MessageTimeout = 100 [2024-05-29T20:51:30.088] debug2: Parameters = (null) [2024-05-29T20:51:30.088] debug2: PidFile = /run/slurmdbd.pid [2024-05-29T20:51:30.088] debug2: PluginDir = /usr/lib/x86_64-linux-gnu/slurm-wlm [2024-05-29T20:51:30.088] debug2: PrivateData = none [2024-05-29T20:51:30.088] debug2: PurgeEventAfter = 1 months* [2024-05-29T20:51:30.088] debug2: PurgeJobAfter = 12 months* [2024-05-29T20:51:30.088] debug2: PurgeResvAfter = 1 months* [2024-05-29T20:51:30.088] debug2: PurgeStepAfter = 1 months [2024-05-29T20:51:30.088] debug2: PurgeSuspendAfter = 1 months [2024-05-29T20:51:30.088] debug2: PurgeTXNAfter = 12 months [2024-05-29T20:51:30.088] debug2: PurgeUsageAfter = 24 months [2024-05-29T20:51:30.088] debug2: SlurmUser = root(0) [2024-05-29T20:51:30.089] debug2: StorageBackupHost = (null) [2024-05-29T20:51:30.089] debug2: StorageHost = localhost [2024-05-29T20:51:30.089] debug2: StorageLoc = slurm_acct_db [2024-05-29T20:51:30.089] debug2: StoragePort = 3306 [2024-05-29T20:51:30.089] debug2: StorageType = accounting_storage/mysql [2024-05-29T20:51:30.089] debug2: StorageUser = slurm [2024-05-29T20:51:30.089] debug2: TCPTimeout = 2 [2024-05-29T20:51:30.089] debug2: TrackWCKey = 0 [2024-05-29T20:51:30.089] debug2: TrackSlurmctldDown= 0 [2024-05-29T20:51:30.089] debug2: acct_storage_p_get_connection: request new connection 1 [2024-05-29T20:51:30.089] debug2: Attempting to connect to localhost:3306 [2024-05-29T20:51:30.090] slurmdbd version 19.05.5 started [2024-05-29T20:51:30.090] debug2: running rollup at Wed May 29 20:51:30 2024 [2024-05-29T20:51:30.091] debug2: Everything rolled up [2024-05-29T20:51:49.673] Terminate signal (SIGINT or SIGTERM) received [2024-05-29T20:51:49.673] debug: rpc_mgr shutting down my config file looks like this ArchiveEvents=yes ArchiveJobs=yes ArchiveResvs=yes ArchiveSteps=no ArchiveSuspend=no ArchiveTXN=no ArchiveUsage=no PurgeEventAfter=1month PurgeJobAfter=12month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month PurgeTXNAfter=12month PurgeUsageAfter=24month # Authentication info AuthType=auth/munge # slurmDBD info DbdAddr=localhost DbdHost=head-node DbdPort=7032 SlurmUser=root MessageTimeout=100 DebugLevel=5 #DefaultQOS=normal,standby LogFile=/var/log/slurmdbd.log PidFile=/run/slurmdbd.pid #PrivateData=accounts,users,usage,jobs #TrackWCKey=yes # # Database info StorageType=accounting_storage/mysql StorageHost=localhost StoragePort=3306 StoragePass=slurmdbpass StorageUser=slurm StorageLoc=slurm_acct_db I used standard names and passwords to get started and I will change later but everytime I try to start slurmdbd.service it crashes and I have that log that I shared with you I use these versions slurmdbd -V slurm-wlm 19.05.5 mysql Ver 15.1 Distrib 10.3.39-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2 Everything else Is working properly except I cannot get slurmdbd to work and at this point I exhausted all my possible trials :) looking for some expert insights :) Any idea what I am doing wrong here ? Also I didn't compile any slurm package. I used the binary from apt repos Any help will be appreciated Cheers Rad -- [View Less]

7 12

Re: slurmdbd not connecting to mysql (mariadb)
by Radhouane Aniba 31 May '24

31 May '24

manually running it through sudo slurmdbd -D /path/to/conf is very quick on my fresh install trying to start the slurmdbd through systemctl take 3 minutes and then crashes and fail Is there an alternative to systemctl to start the slurmdbd in the background ? But most importantly I wanted to know why it takes so long through systemctl. Maybe I can increase the timeout limit ? On Thu, May 30, 2024 at 11:54 PM Ryan Novosielski <novosirj(a)rutgers.edu> wrote: > It may take longer to … [View More]start than systemd allows for. How long does it take > to start from the command line? It’s common to need to run it manually for > upgrades to complete. > > -- > #BlackLivesMatter > ____ > || \\UTGERS, |---------------------------*O*--------------------------- > ||_// the State | Ryan Novosielski - novosirj(a)rutgers.edu > || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus > || \\ of NJ | Office of Advanced Research Computing - MSB > A555B, Newark > `' > > On May 30, 2024, at 20:24, Radhouane Aniba via slurm-users < > slurm-users(a)lists.schedmd.com> wrote: > > Ok I made some progress here. > > I removed and purged slurmdbd mysql mariadb etc .. and started from > scratch. > I added the recommended mysqld requirements > > Started slurmdbd manually : sudo slurmdbd -D /path/to/conf and everything > worked well > > When I tried to start the service sudo systemctl start slurmdbd.service > it didnt work > > sudo systemctl status slurmdbd.service > ● slurmdbd.service - Slurm DBD accounting daemon > Loaded: loaded (/etc/systemd/system/slurmdbd.service; enabled; vendor > preset: enabled) > Active: failed (Result: timeout) since Fri 2024-05-31 00:21:30 UTC; > 2min 5s ago > Process: 6258 ExecStart=/usr/sbin/slurmdbd -D > /etc/slurm-llnl/slurmdbd.conf (code=exited, status=0/SUCCESS) > > May 31 00:20:00 hannibal-hn systemd[1]: Starting Slurm DBD accounting > daemon... > May 31 00:21:30 hannibal-hn systemd[1]: slurmdbd.service: start operation > timed out. Terminating. > May 31 00:21:30 hannibal-hn systemd[1]: slurmdbd.service: Failed with > result 'timeout'. > May 31 00:21:30 hannibal-hn systemd[1]: Failed to start Slurm DBD > accounting daemon. > > Even though it is the same command ?! > > Any idea ? > > > On Thu, May 30, 2024 at 5:02 PM Radhouane Aniba <aradwen(a)gmail.com> wrote: > >> Thank you Ahmet and Brian, >> >> Ahmet, which conf in particular slurmdbd is readiugn from, I parsed all >> the cnf files for mysql and I cannot find the data it is displaying here >> >> slurmdbd: debug2: Attempting to connect to localhost:3306 >> slurmdbd: debug2: innodb_buffer_pool_size: 134217728 >> slurmdbd: debug2: innodb_log_file_size: 50331648 >> slurmdbd: debug2: innodb_lock_wait_timeout: 50 >> slurmdbd: error: Database settings not recommended values: >> innodb_buffer_pool_size innodb_lock_wait_timeout >> >> >> sudo tree /etc/mysql/* >> /etc/mysql/conf.d >> ├── mysql.cnf >> └── mysqldump.cnf >> /etc/mysql/debian.cnf >> /etc/mysql/debian-start >> /etc/mysql/FROZEN >> /etc/mysql/mariadb.cnf >> /etc/mysql/mariadb.conf.d >> ├── 50-client.cnf >> ├── 50-mysql-clients.cnf >> ├── 50-mysqld_safe.cnf >> └── 50-server.cnf >> /etc/mysql/my.cnf >> /etc/mysql/my.cnf.fallback >> /etc/mysql/mysql.cnf >> /etc/mysql/mysql.conf.d >> ├── mysql.cnf >> └── mysqld.cnf >> >> On Thu, May 30, 2024 at 12:21 PM Brian Andrus via slurm-users < >> slurm-users(a)lists.schedmd.com> wrote: >> >>> That SIGTERM message means something is telling slurmdbd to quit. >>> >>> Check your cron jobs, maintenance scripts, etc. Slurmdbd is being told >>> to shutdown. If you are running in the foreground, a ^C does that. If you >>> run a kill or killall on it, you will get that same message. >>> >>> Brian Andrus >>> On 5/30/2024 6:53 AM, Radhouane Aniba via slurm-users wrote: >>> >>> Yes I can connect to my database using mysql --user=slurm >>> --password=slurmdbpass slurm_acct_db and there is no firewall blocking >>> mysql after checking the firewall question >>> >>> ALso here is the output of slurmdbd -D -vvv (note I can only run this as >>> sudo ) >>> >>> sudo slurmdbd -D -vvv >>> slurmdbd: debug: Log file re-opened >>> slurmdbd: debug: Munge authentication plugin loaded >>> slurmdbd: debug2: mysql_connect() called for db slurm_acct_db >>> slurmdbd: debug2: Attempting to connect to localhost:3306 >>> slurmdbd: debug2: innodb_buffer_pool_size: 134217728 >>> slurmdbd: debug2: innodb_log_file_size: 50331648 >>> slurmdbd: debug2: innodb_lock_wait_timeout: 50 >>> slurmdbd: error: Database settings not recommended values: >>> innodb_buffer_pool_size innodb_lock_wait_timeout >>> slurmdbd: Accounting storage MYSQL plugin loaded >>> slurmdbd: debug2: ArchiveDir = /tmp >>> slurmdbd: debug2: ArchiveScript = (null) >>> slurmdbd: debug2: AuthAltTypes = (null) >>> slurmdbd: debug2: AuthInfo = (null) >>> slurmdbd: debug2: AuthType = auth/munge >>> slurmdbd: debug2: CommitDelay = 0 >>> slurmdbd: debug2: DbdAddr = localhost >>> slurmdbd: debug2: DbdBackupHost = (null) >>> slurmdbd: debug2: DbdHost = hannibal-hn >>> slurmdbd: debug2: DbdPort = 7032 >>> slurmdbd: debug2: DebugFlags = (null) >>> slurmdbd: debug2: DebugLevel = 6 >>> slurmdbd: debug2: DebugLevelSyslog = 10 >>> slurmdbd: debug2: DefaultQOS = (null) >>> slurmdbd: debug2: LogFile = /var/log/slurmdbd.log >>> slurmdbd: debug2: MessageTimeout = 100 >>> slurmdbd: debug2: Parameters = (null) >>> slurmdbd: debug2: PidFile = /run/slurmdbd.pid >>> slurmdbd: debug2: PluginDir = /usr/lib/x86_64-linux-gnu/slurm-wlm >>> slurmdbd: debug2: PrivateData = none >>> slurmdbd: debug2: PurgeEventAfter = 1 months* >>> slurmdbd: debug2: PurgeJobAfter = 12 months* >>> slurmdbd: debug2: PurgeResvAfter = 1 months* >>> slurmdbd: debug2: PurgeStepAfter = 1 months >>> slurmdbd: debug2: PurgeSuspendAfter = 1 months >>> slurmdbd: debug2: PurgeTXNAfter = 12 months >>> slurmdbd: debug2: PurgeUsageAfter = 24 months >>> slurmdbd: debug2: SlurmUser = root(0) >>> slurmdbd: debug2: StorageBackupHost = (null) >>> slurmdbd: debug2: StorageHost = localhost >>> slurmdbd: debug2: StorageLoc = slurm_acct_db >>> slurmdbd: debug2: StoragePort = 3306 >>> slurmdbd: debug2: StorageType = accounting_storage/mysql >>> slurmdbd: debug2: StorageUser = slurm >>> slurmdbd: debug2: TCPTimeout = 2 >>> slurmdbd: debug2: TrackWCKey = 0 >>> slurmdbd: debug2: TrackSlurmctldDown= 0 >>> slurmdbd: debug2: acct_storage_p_get_connection: request new connection >>> 1 >>> slurmdbd: debug2: Attempting to connect to localhost:3306 >>> slurmdbd: slurmdbd version 19.05.5 started >>> slurmdbd: debug2: running rollup at Thu May 30 13:50:08 2024 >>> slurmdbd: debug2: Everything rolled up >>> >>> >>> It goes like this for some time and then it crashes with this message >>> >>> slurmdbd: Terminate signal (SIGINT or SIGTERM) received >>> slurmdbd: debug: rpc_mgr shutting down >>> >>> >>> On Thu, May 30, 2024 at 8:18 AM mercan <ahmet.mercan(a)uhem.itu.edu.tr> >>> wrote: >>> >>>> Did you try to connect database using mysql command? >>>> >>>> mysql --user=slurm --password=slurmdbpass slurm_acct_db >>>> >>>> C. Ahmet Mercan >>>> >>>> On 30.05.2024 14:48, Radhouane Aniba via slurm-users wrote: >>>> >>>> Thank you Ahmet, >>>> I dont have a firewall active. >>>> And because slurmdbd cannot connect to the database I am not able to >>>> getting it to be activated through systemctl I will share the output for >>>> slurmdbd -D -vvv shortly but overall it is always saying trying to connect >>>> to the db and then retries a couple of times and crashes >>>> >>>> R. >>>> >>>> >>>> >>>> >>>> On Thu, May 30, 2024 at 2:51 AM mercan <ahmet.mercan(a)uhem.itu.edu.tr> >>>> wrote: >>>> >>>>> Hi; >>>>> >>>>> Did you check can you connect db with your conf parameters from >>>>> head-node: >>>>> >>>>> mysql --user=slurm --password=slurmdbpass slurm_acct_db >>>>> >>>>> Also, check and stop firewall and selinux, if they are running. >>>>> >>>>> Last, you can stop slurmdbd, then run run terminal with: >>>>> >>>>> slurmdbd -D -vvv >>>>> >>>>> Regards; >>>>> >>>>> C. Ahmet Mercan >>>>> >>>>> On 30.05.2024 00:05, Radhouane Aniba via slurm-users wrote: >>>>> >>>>> Hi everyone >>>>> I am trying to get slurmdbd to run on my local home server but I am >>>>> really struggling. >>>>> Note : am a novice slurm user >>>>> my slurmdbd always times out even though all the details in the conf >>>>> file are correct >>>>> >>>>> My log looks like this >>>>> >>>>> [2024-05-29T20:51:30.088] Accounting storage MYSQL plugin loaded >>>>> [2024-05-29T20:51:30.088] debug2: ArchiveDir = /tmp >>>>> [2024-05-29T20:51:30.088] debug2: ArchiveScript = (null) >>>>> [2024-05-29T20:51:30.088] debug2: AuthAltTypes = (null) >>>>> [2024-05-29T20:51:30.088] debug2: AuthInfo = (null) >>>>> [2024-05-29T20:51:30.088] debug2: AuthType = auth/munge >>>>> [2024-05-29T20:51:30.088] debug2: CommitDelay = 0 >>>>> [2024-05-29T20:51:30.088] debug2: DbdAddr = localhost >>>>> [2024-05-29T20:51:30.088] debug2: DbdBackupHost = (null) >>>>> [2024-05-29T20:51:30.088] debug2: DbdHost = head-node >>>>> [2024-05-29T20:51:30.088] debug2: DbdPort = 7032 >>>>> [2024-05-29T20:51:30.088] debug2: DebugFlags = (null) >>>>> [2024-05-29T20:51:30.088] debug2: DebugLevel = 6 >>>>> [2024-05-29T20:51:30.088] debug2: DebugLevelSyslog = 10 >>>>> [2024-05-29T20:51:30.088] debug2: DefaultQOS = (null) >>>>> [2024-05-29T20:51:30.088] debug2: LogFile = /var/log/slurmdbd.log >>>>> [2024-05-29T20:51:30.088] debug2: MessageTimeout = 100 >>>>> [2024-05-29T20:51:30.088] debug2: Parameters = (null) >>>>> [2024-05-29T20:51:30.088] debug2: PidFile = /run/slurmdbd.pid >>>>> [2024-05-29T20:51:30.088] debug2: PluginDir = >>>>> /usr/lib/x86_64-linux-gnu/slurm-wlm >>>>> [2024-05-29T20:51:30.088] debug2: PrivateData = none >>>>> [2024-05-29T20:51:30.088] debug2: PurgeEventAfter = 1 months* >>>>> [2024-05-29T20:51:30.088] debug2: PurgeJobAfter = 12 months* >>>>> [2024-05-29T20:51:30.088] debug2: PurgeResvAfter = 1 months* >>>>> [2024-05-29T20:51:30.088] debug2: PurgeStepAfter = 1 months >>>>> [2024-05-29T20:51:30.088] debug2: PurgeSuspendAfter = 1 months >>>>> [2024-05-29T20:51:30.088] debug2: PurgeTXNAfter = 12 months >>>>> [2024-05-29T20:51:30.088] debug2: PurgeUsageAfter = 24 months >>>>> [2024-05-29T20:51:30.088] debug2: SlurmUser = root(0) >>>>> [2024-05-29T20:51:30.089] debug2: StorageBackupHost = (null) >>>>> [2024-05-29T20:51:30.089] debug2: StorageHost = localhost >>>>> [2024-05-29T20:51:30.089] debug2: StorageLoc = slurm_acct_db >>>>> [2024-05-29T20:51:30.089] debug2: StoragePort = 3306 >>>>> [2024-05-29T20:51:30.089] debug2: StorageType = >>>>> accounting_storage/mysql >>>>> [2024-05-29T20:51:30.089] debug2: StorageUser = slurm >>>>> [2024-05-29T20:51:30.089] debug2: TCPTimeout = 2 >>>>> [2024-05-29T20:51:30.089] debug2: TrackWCKey = 0 >>>>> [2024-05-29T20:51:30.089] debug2: TrackSlurmctldDown= 0 >>>>> [2024-05-29T20:51:30.089] debug2: acct_storage_p_get_connection: >>>>> request new connection 1 >>>>> [2024-05-29T20:51:30.089] debug2: Attempting to connect to >>>>> localhost:3306 >>>>> [2024-05-29T20:51:30.090] slurmdbd version 19.05.5 started >>>>> [2024-05-29T20:51:30.090] debug2: running rollup at Wed May 29 >>>>> 20:51:30 2024 >>>>> [2024-05-29T20:51:30.091] debug2: Everything rolled up >>>>> [2024-05-29T20:51:49.673] Terminate signal (SIGINT or SIGTERM) >>>>> received >>>>> [2024-05-29T20:51:49.673] debug: rpc_mgr shutting down >>>>> >>>>> >>>>> >>>>> my config file looks like this >>>>> >>>>> ArchiveEvents=yes >>>>> ArchiveJobs=yes >>>>> ArchiveResvs=yes >>>>> ArchiveSteps=no >>>>> ArchiveSuspend=no >>>>> ArchiveTXN=no >>>>> ArchiveUsage=no >>>>> PurgeEventAfter=1month >>>>> PurgeJobAfter=12month >>>>> PurgeResvAfter=1month >>>>> PurgeStepAfter=1month >>>>> PurgeSuspendAfter=1month >>>>> PurgeTXNAfter=12month >>>>> PurgeUsageAfter=24month >>>>> # Authentication info >>>>> AuthType=auth/munge >>>>> # slurmDBD info >>>>> DbdAddr=localhost >>>>> DbdHost=head-node >>>>> DbdPort=7032 >>>>> SlurmUser=root >>>>> MessageTimeout=100 >>>>> DebugLevel=5 >>>>> #DefaultQOS=normal,standby >>>>> LogFile=/var/log/slurmdbd.log >>>>> PidFile=/run/slurmdbd.pid >>>>> #PrivateData=accounts,users,usage,jobs >>>>> #TrackWCKey=yes >>>>> # >>>>> # Database info >>>>> StorageType=accounting_storage/mysql >>>>> StorageHost=localhost >>>>> StoragePort=3306 >>>>> StoragePass=slurmdbpass >>>>> StorageUser=slurm >>>>> StorageLoc=slurm_acct_db >>>>> I used standard names and passwords to get started and I will change >>>>> later >>>>> >>>>> but everytime I try to start slurmdbd.service it crashes and I have >>>>> that log that I shared with you >>>>> >>>>> I use these versions >>>>> >>>>> slurmdbd -V >>>>> slurm-wlm 19.05.5 >>>>> mysql Ver 15.1 Distrib 10.3.39-MariaDB, for debian-linux-gnu (x86_64) >>>>> using readline 5.2 >>>>> Everything else Is working properly except I cannot get slurmdbd to >>>>> work and at this point I exhausted all my possible trials :) looking for >>>>> some expert insights :) >>>>> >>>>> >>>>> Any idea what I am doing wrong here ? Also I didn't compile any slurm >>>>> package. I used the binary from apt repos >>>>> >>>>> Any help will be appreciated >>>>> >>>>> Cheers >>>>> >>>>> Rad >>>>> >>>>> -- >>>>> >>>>> >>>>> >>>> >>> >>> -- >>> *Rad Aniba, PhD* >>> >>> >>> >>> -- >>> slurm-users mailing list -- slurm-users(a)lists.schedmd.com >>> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com >>> >> >> >> -- >> *Rad Aniba, PhD* >> >> > > -- > *Rad Aniba, PhD* > > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > > > -- *Rad Aniba, PhD* [View Less]

2 3

Slurm version 24.05.0 is now available
by Marshall Garey 30 May '24

30 May '24

We are pleased to announce the availability of Slurm 24.05.0. To highlight some new features in 24.05: - Isolated Job Step management. Enabled on a job-by-job basis with the --stepmgr option, or globally through SlurmctldParameters=enable_stepmgr. - Federation - Allow for client command operation while SlurmDBD is unavailable. - New MaxTRESRunMinsPerAccount and MaxTRESRunMinsPerUser QOS limits. - New USER_DELETE reservation flag. - New Flags=rebootless option on Features for node_features/… [View More]

1 0

Executing srun -n X where X is greater than total CPU in entire cluster
by Dan Healy 30 May '24

30 May '24

Hi there, SLURM community, I swear I've done this before, but now it's failing on a new cluster I'm deploying. We have 6 compute nodes with 64 cpu each (384 CPU total). When I run `srun -n 500 hostname`, the task gets queued since there's not 500 available CPU. Wasn't there an option that allows for this to be run where the first 384 tasks execute, and then the remaining execute when resources free up? Here's my conf: # Slurm Cgroup Configs used on controllers and workersslurm_cgroup_config:… [View More]

2 2

Jobs showing running but not running
by Sushil Mishra 29 May '24

29 May '24

Hi All, I'm managing a cluster with Slurm, consisting of 4 nodes. One of the compute nodes appears to be experiencing issues. While the front node's 'squeue' command indicates that jobs are running, upon connecting to the problematic node, I observe no active processes and GPUs are not being utilized. [sushil@ccbrc ~]$ sinfo -Nel Wed May 29 12:00:08 2024 NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON gag 1 defq* mixed 48 2:… [View More]

3 2

dynamical configuration || meta configuration mgmt
by Heckes, Frank 29 May '24

29 May '24

Hello all, I’m sorry if this has been asked and answered before, but I couldn’t find anything related. Does anyone know whether a framework of sorts exists that allow to change certain SLURM configuration parameters provided some conditions in the batch system’s state are detected and of course are revert if the state became the old one again? (To be more concrete: We like to raise or unset maxjobPU to run as much small jobs as possible to allocate all nodes as soon as certain threshold … [View More]

2 1

Configuring sacct to report state=OUT_OF_MEMORY
by Lee 29 May '24

29 May '24

Hello, *Background :* I am working on a small cluster that is managed by Base Command Manager v10.0 using Slurm 23.02.7 with Ubuntu 22.04.2. I have a small testing script that simply consumes memory and processors. I run my test script, it consumes more memory than allocated by Slurm and as expected it gets killed by OOM killer. In /var/log/slurmd, I see entries like : [2024-05-29T08:53:04.975] Launching batch job 65 for UID 1001 [2024-05-29T08:53:05.016] [65.batch] task/cgroup: … [View More]_memcg_initialize: job: alloc=10868MB mem.limit=10868MB memsw.limit=10868MB job_swappiness=1 [2024-05-29T08:53:05.016] [65.batch] task/cgroup: _memcg_initialize: step: alloc=10868MB mem.limit=10868MB memsw.limit=10868MB job_swappiness=1 [2024-05-29T08:53:19.530] [65.batch] task/cgroup: task_cgroup_memory_check_oom: StepId=65.batch hit memory+swap limit at least once during execution. This may or may not result in some failure. [2024-05-29T08:53:19.563] [65.batch] done with job Inspecting with sacct, I see : $ sacct -j 65 --format="jobid,jobname,state,exitcode" JobID JobName State ExitCode ------------ ---------- ---------- -------- 65 wrap FAILED 9:0 65.batch batch FAILED 9:0 Based on my previous experience with a RHEL based Slurm cluster, I would expect the state to be listed as OUT_OF_MEMORY and the exitcode to be 0:125. *Question : * 1. How do I configure [slurm,cgroup].conf such that when Slurm kills a job due to exceeding the allocated memory, sacct reports the state as OUT_OF_MEMORY? See below for my configuration : *Configuration:* /etc/default/grub : ~# grep -v "^#" /etc/default/grub GRUB_DEFAULT=0 GRUB_TIMEOUT_STYLE=hidden GRUB_TIMEOUT=10 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="" GRUB_CMDLINE_LINUX="biosdevname=0 cgroup_enable=memory swapaccount=1" GRUB_GFXMODE="1024x768,800x600,auto" GRUB_BACKGROUND="/boot/grub/bcm.png" slurm.conf : ~# grep -v "^#" /cm/shared/apps/slurm/var/etc/bcm10-slurm/slurm.conf SlurmUser=slurm SlurmctldPort=6817 SlurmdPort=6818 AuthType=auth/munge SlurmdSpoolDir=/cm/local/apps/slurm/var/spool SwitchType=switch/none MpiDefault=pmix SlurmctldPidFile=/var/run/slurmctld.pid SlurmdPidFile=/var/run/slurmd.pid ProctrackType=proctrack/cgroup ReturnToService=2 TaskPlugin=task/cgroup InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=300 SlurmdTimeout=300 Waittime=0 SlurmctldDebug=info SlurmctldLogFile=/var/log/slurmctld SlurmdDebug=info SlurmdLogFile=/var/log/slurmd JobAcctGatherType=jobacct_gather/cgroup JobAcctGatherFrequency=30 AccountingStorageType=accounting_storage/slurmdbd AccountingStorageUser=slurm SlurmctldHost=bcm10-h01 AccountingStorageHost=master NodeName=bcm10-n[01,02] Procs=4 CoresPerSocket=4 RealMemory=15988 SocketsPerBoard=1 ThreadsPerCore=1 Boards=1 MemSpecLimit=5120 Feature=location=local PartitionName="defq" Default=YES MinNodes=1 DefaultTime=UNLIMITED MaxTime=UNLIMITED AllowGroups=ALL PriorityJobFactor=1 PriorityTier=1 OverSubscribe=NO PreemptMode=OFF AllowAccounts=ALL AllowQos=ALL Nodes=bcm10-n[01,02] ClusterName=bcm10-slurm SchedulerType=sched/backfill StateSaveLocation=/cm/shared/apps/slurm/var/cm/statesave/bcm10-slurm PrologFlags=Alloc GresTypes=gpu Prolog=/cm/local/apps/cmd/scripts/prolog Epilog=/cm/local/apps/cmd/scripts/epilog SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory cgroup.conf ~# grep -v "^#" /cm/shared/apps/slurm/var/etc/bcm10-slurm/cgroup.conf CgroupMountpoint="/sys/fs/cgroup" CgroupAutomount=no ConstrainCores=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes ConstrainDevices=yes AllowedRamSpace=100.00 AllowedSwapSpace=0.00 MemorySwappiness=1 MaxRAMPercent=100.00 MaxSwapPercent=100.00 MinRAMSpace=30 Best regards, Lee [View Less]

1 0

sbatch problem
by Mihai Ciubancan 29 May '24

29 May '24

Hello, My name is Mihai and a have an issue with a small GPU cluster manage with slurm 22.05.11. I got 2 different output when I'm trying to find out the name of the nodes(one correct and one wrong). The script is: #!/bin/bash #SBATCH --job-name=test #SBATCH --output=/data/mihai/res.txt #SBATCH --partition=eli #SBATCH --nodes=2 srun echo Running on host: $(hostname) srun hostname srun sleep 15 And the output look like this: cat res.txt Running on host: mihai-x8640 Running on host: mihai-… [View More]

2 6

slurm_bufs_sendto failed
by shaobo liu 29 May '24

29 May '24

slurm 23.11.0 , the slurmd error is as follows，what is the reason for this? [2024-05-29T09:21:32.450] task/affinity: task_p_slurmd_batch_request: task_p_slurmd_batch_request: 32259 [2024-05-29T09:21:32.451] task/affinity: batch_bind: job 32259 CPU input mask for node: 0xFFFFFFFFFFFFFFFF [2024-05-29T09:21:32.451] task/affinity: batch_bind: job 32259 CPU final HW mask for node: 0xFFFFFFFFFFFFFFFF [2024-05-29T09:21:32.451] Launching batch job 32259 for UID 66003 [2024-05-29T09:21:32.868] launch … [View More]task StepId=32259.0 request from UID:66003 GID:66003 HOST:192.168.87.45 PORT:31104 [2024-05-29T09:21:32.868] task/affinity: lllp_distribution: JobId=32259 implicit auto binding: cores,one_thread, dist 2 [2024-05-29T09:21:32.868] task/affinity: _task_layout_lllp_cyclic: _task_layout_lllp_cyclic [2024-05-29T09:21:32.868] task/affinity: _lllp_generate_cpu_bind: _lllp_generate_cpu_bind jobid [32259]: mask_cpu,one_thread, 0x0000000000000001,0x0000000100000000,0x0000000000000002,0x0000000200000000,0x0000000000000004,0x0000000400000000,0x0000000000000008,0x0000000800000000,0x0000000000000010,0x0000001000000000,0x0000000000000020,0x0000002000000000,0x0000000000000040,0x0000004000000000,0x0000000000000080,0x0000008000000000,0x0000000000000100,0x0000010000000000,0x0000000000000200,0x0000020000000000,0x0000000000000400,0x0000040000000000,0x0000000000000800,0x0000080000000000,0x0000000000001000,0x0000100000000000,0x0000000000002000,0x0000200000000000,0x0000000000004000,0x0000400000000000,0x0000000000008000,0x0000800000000000,0x0000000000010000,0x0001000000000000,0x0000000000020000,0x0002000000000000,0x0000000000040000,0x0004000000000000,0x0000000000080000,0x0008000000000000,0x0000000000100000,0x0010000000000000,0x0000000000200000,0x0020000000000000,0x0000000000400000,0x0040000000000000,0x0000000000800000,0x0080000000000000,0x0000000001000000,0x0100000000000000,0x0000000002000000,0x0200000000000000,0x0000000004000000,0x0400000000000000,0x0000000008000000,0x0800000000000000,0x0000000010000000,0x1000000000000000,0x0000000020000000,0x2000000000000000,0x0000000040000000,0x4000000000000000,0x0000000080000000,0x8000000000000000 [2024-05-29T09:32:04.541] [32259.batch] done with step [2024-05-29T09:33:31.000] error: slurm_send_node_msg: [socket:[7270627]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:34:02.000] error: slurm_send_node_msg: [socket:[7270629]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:34:03.016] [32259.extern] done with step [2024-05-29T09:34:41.000] error: slurm_send_node_msg: [socket:[7270635]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:35:13.000] error: slurm_send_node_msg: [socket:[7270637]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:35:52.000] error: slurm_send_node_msg: [socket:[7270639]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:36:01.000] error: slurm_send_node_msg: [socket:[7270630]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:36:24.000] error: slurm_send_node_msg: [socket:[7270641]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:37:03.000] error: slurm_send_node_msg: [socket:[7270644]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:37:31.000] [32259.0] error: *** STEP 32259.0 STEPD TERMINATED ON node045 AT 2024-05-29T09:37:30 DUE TO JOB NOT RUNNING *** [2024-05-29T09:37:33.000] [32259.0] stepd_cleanup: done with step (rc[0x0]:No error, cleanup_rc[0xfbb]:Job step not running) [2024-05-29T09:37:35.000] error: slurm_send_node_msg: [socket:[7270646]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:38:14.000] error: slurm_send_node_msg: [socket:[7270649]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:38:46.000] error: slurm_send_node_msg: [socket:[7270651]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:39:25.000] error: slurm_send_node_msg: [socket:[7270657]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [View Less]

1 0

Final Call for SLUG Presenters & Early Bird Registration!
by Victoria Hobson 28 May '24

28 May '24

Slurm User Group (SLUG) 2024 is set for September 12-13 at the University of Oslo in Oslo, Norway. Registration information and a high-level schedule can be found here:https://slug24.splashthat.com/ The last day to register at the early bird pricing is this Friday, May 31st. Friday is also the deadline to submit a presentation abstract. We do not intend to extend this deadline. If you are interested in presenting your own usage, developments, site report, tutorial, etc about Slurm, please … [View More]

1 0

slurmdbd archive format
by O'Neal, Doug (NIH/NCI) [C] 28 May '24

28 May '24

My organization needs to access historic job information records for metric reporting and resource forecasting. slurmdbd is archiving only the job information, which should be sufficient for our numbers, but is using the default archive script. In retrospect, this data should have been migrated to a secondary MariaDB instance, but that train has passed. The format of the archive files is not well documented. Does anyone have a program (python/C/whatever) that will read a job_table_archive file … [View More]

2 2

Fwd: Re: Performance Discrepancy between Slurm and Direct mpirun for VASP Jobs.
by Hongyi Zhao 28 May '24

28 May '24

---------- Forwarded message --------- From: Hermann Schwärzler <hermann.schwaerzler(a)uibk.ac.at> Date: Tue, May 28, 2024 at 4:10 PM Subject: Re: [slurm-users] Re: Performance Discrepancy between Slurm and Direct mpirun for VASP Jobs. To: Hongyi Zhao <hongyi.zhao(a)gmail.com> Hi Zhao, On 5/28/24 03:08, Hongyi Zhao wrote: [...] > > What's the complete content of cli_filter.lua and where should I put this file? [...] Below you find the complete content of our cli_filter.lua.… [View More]

1 0

Performance Discrepancy between Slurm and Direct mpirun for VASP Jobs.
by Hongyi Zhao 27 May '24

27 May '24

Dear Slurm Users, I am experiencing a significant performance discrepancy when running the same VASP job through the Slurm scheduler compared to running it directly with mpirun. I am hoping for some insights or advice on how to resolve this issue. System Information: Slurm Version: 21.08.5 OS: Ubuntu 22.04.4 LTS (Jammy) Job Submission Script: #!/usr/bin/env bash #SBATCH -N 1 #SBATCH -D . #SBATCH --output=%j.out #SBATCH --error=%j.err ##SBATCH --time=2-00:00:00 #SBATCH --ntasks=36 #SBATCH -… [View More]-mem=64G echo '#######################################################' echo "date = $(date)" echo "hostname = $(hostname -s)" echo "pwd = $(pwd)" echo "sbatch = $(which sbatch | xargs realpath -e)" echo "" echo "WORK_DIR = $WORK_DIR" echo "SLURM_SUBMIT_DIR = $SLURM_SUBMIT_DIR" echo "SLURM_JOB_NUM_NODES = $SLURM_JOB_NUM_NODES" echo "SLURM_NTASKS = $SLURM_NTASKS" echo "SLURM_NTASKS_PER_NODE = $SLURM_NTASKS_PER_NODE" echo "SLURM_CPUS_PER_TASK = $SLURM_CPUS_PER_TASK" echo "SLURM_JOBID = $SLURM_JOBID" echo "SLURM_JOB_NODELIST = $SLURM_JOB_NODELIST" echo "SLURM_NNODES = $SLURM_NNODES" echo "SLURMTMPDIR = $SLURMTMPDIR" echo '#######################################################' echo "" module purge > /dev/null 2>&1 module load vasp ulimit -s unlimited mpirun vasp_std Performance Observation: When running the job through Slurm: werner@x13dai-t:~/Public/hpc/servers/benchmark/Cr72_3x3x3K_350eV_10DAV$ grep LOOP OUTCAR LOOP: cpu time 14.4893: real time 14.5049 LOOP: cpu time 14.3538: real time 14.3621 LOOP: cpu time 14.3870: real time 14.3568 LOOP: cpu time 15.9722: real time 15.9018 LOOP: cpu time 16.4527: real time 16.4370 LOOP: cpu time 16.7918: real time 16.7781 LOOP: cpu time 16.9797: real time 16.9961 LOOP: cpu time 15.9762: real time 16.0124 LOOP: cpu time 16.8835: real time 16.9008 LOOP: cpu time 15.2828: real time 15.2921 LOOP+: cpu time 176.0917: real time 176.0755 When running the job directly with mpirun: werner@x13dai-t:~/Public/hpc/servers/benchmark/Cr72_3x3x3K_350eV_10DAV$ mpirun -n 36 vasp_std werner@x13dai-t:~/Public/hpc/servers/benchmark/Cr72_3x3x3K_350eV_10DAV$ grep LOOP OUTCAR LOOP: cpu time 9.0072: real time 9.0074 LOOP: cpu time 9.0515: real time 9.0524 LOOP: cpu time 9.1896: real time 9.1907 LOOP: cpu time 10.1467: real time 10.1479 LOOP: cpu time 10.2691: real time 10.2705 LOOP: cpu time 10.4330: real time 10.4340 LOOP: cpu time 10.9049: real time 10.9055 LOOP: cpu time 9.9718: real time 9.9714 LOOP: cpu time 10.4511: real time 10.4470 LOOP: cpu time 9.4621: real time 9.4584 LOOP+: cpu time 110.0790: real time 110.0739 Could you provide any insights or suggestions on what might be causing this performance issue? Are there any specific configurations or settings in Slurm that I should check or adjust to align the performance more closely with the direct mpirun execution? Thank you for your time and assistance. Best regards, Zhao -- Assoc. Prof. Hongsheng Zhao <hongyi.zhao(a)gmail.com> Theory and Simulation of Materials Hebei Vocational University of Technology and Engineering No. 473, Quannan West Street, Xindu District, Xingtai, Hebei province [View Less]

4 10

Building Slurm debian package vs building from source
by Arnuld 23 May '24

23 May '24

We have several nodes, most of which have different Linux distributions (distro for short). Controller has a different distro as well. The only common thing between controller and all the does is that all of them ar x86_64. I can install Slurm using package manager on all the machines but this will not work because controller will have a different version of Slurm compared to the nodes (21.08 vs 23.11) If I build from source then I see two solutions: - build a deb package - build a custom … [View More]

4 7

Slurm Build Error
by Arnuld 23 May '24

23 May '24

Getting this error when I run "make install": echo >>"lib_ref.lo" /bin/bash ../../libtool --tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o lib_ref.la lib_ref.lo -lpthread -lm -lresolv libtool: link: ar cr .libs/lib_ref.a libtool: link: ranlib .libs/lib_ref.a libtool: link: ( cd ".libs" && rm -f "lib_ref.la" && ln -s "../lib_ref.la" " lib_ref.la" ) /bin/bash ../../libtool --… [View More]tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -export-dynamic -o sacctmgr account_functions.o archive_functions.o association_functions.o config_functions.o cluster_functions.o common.o event_functions.o federation_functions.o file_functions.o instance_functions.o runaway_job_functions.o job_functions.o reservation_functions.o resource_functions.o sacctmgr.o qos_functions.o txn_functions.o user_functions.o wckey_functions.o problem_functions.o tres_function.o -Wl,-rpath=/root/slurm-slurm-23-11-7-1/z/lib/slurm -L../../src/api/.libs -lslurmfull -export-dynamic -lreadline -lhistory lib_ref.la -lpthread -lm -lresolv libtool: link: gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o sacctmgr account_functions.o archive_functions.o association_functions.o config_functions.o cluster_functions.o common.o event_functions.o federation_functions.o file_functions.o instance_functions.o runaway_job_functions.o job_functions.o reservation_functions.o resource_functions.o sacctmgr.o qos_functions.o txn_functions.o user_functions.o wckey_functions.o problem_functions.o tres_function.o -Wl,-rpath=/root/slurm-slurm-23-11-7-1/z/lib/slurm -Wl,--export-dynamic -L../../src/api/.libs /root/slurm-slurm-23-11-7-1/src/api/.libs/libslurmfull.a -lreadline -lhistory ./.libs/lib_ref.a -lpthread -lm -lresolv -pthread /usr/bin/ld: sacctmgr.o: warning: relocation against `_binary_usage_txt_end' in read-only section `.text' /usr/bin/ld: sacctmgr.o: in function `_usage': /root/slurm-slurm-23-11-7-1/src/sacctmgr/sacctmgr.c:926:(.text+0x1f): undefined reference to `_binary_usage_txt_start' /usr/bin/ld: /root/slurm-slurm-23-11-7-1/src/sacctmgr/sacctmgr.c:926:(.text+0x26): undefined reference to `_binary_usage_txt_end' /usr/bin/ld: warning: creating DT_TEXTREL in a PIE collect2: error: ld returned 1 exit status make[3]: *** [Makefile:672: sacctmgr] Error 1 make[3]: Leaving directory '/root/slurm-slurm-23-11-7-1/src/sacctmgr' make[2]: *** [Makefile:520: all-recursive] Error 1 make[2]: Leaving directory '/root/slurm-slurm-23-11-7-1/src' make[1]: *** [Makefile:621: all-recursive] Error 1 make[1]: Leaving directory '/root/slurm-slurm-23-11-7-1' make: *** [Makefile:520: all] Error 2 ----------------------------------------------------------------------------------------- I used these config options: ./configure --enable-debug --enable-pam --disable-sview --disable-shared --with-munge --with-json --with-yaml --with-http-parser --with-pmix --with-lz4 --with-hwloc --with-jwt --with-libcurl --with-freeipmi --with-rdkafka --with-bpf --prefix=/root/slurm-slurm-23-11-7-1/z [View Less]

1 0

memory high water mark reporting
by Emyr James 22 May '24

22 May '24

Hi, We are trying out slurm having been running grid engine for a long while. In grid engine, the cgroups peak memory and max_rss are generated at the end of a job and recorded. It logs the information from the cgroup hierarchy as well as doing a getrusage call right at the end on the parent pid of the whole job "container" before cleaning up. With slurm it seems that the only way memory is recorded is by the acct gather polling. I am trying to add something in an epilog script to get the … [View More]

5 10

Slurm version 23.11.7 is now available
by Marshall Garey 21 May '24

21 May '24

We are pleased to announce the availability of Slurm version 23.11.7. The 23.11.7 release fixes a few potential crashes in slurmctld when using less common options on job submission, slurmrestd compatibility with auth/slurm, and some additional minor and moderate severity bugs. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Marshall > -- slurmrestd - Correct OpenAPI specification for > 'GET /slurm/v0.0.40/jobs/state' having response as null. > -- … [View More]Allow running jobs on overlapping partitions if jobs don't specify -s. > -- Fix segfault when requesting a shared gres along with an exclusive > allocation. > -- Fix regression in 23.02 where afternotok and afterok dependencies were > rejected for federated jobs not running on the origin cluster of the > submitting job. > -- slurmctld - Disable job table locking while job state cache is active when > replying to `squeue --only-job-state` or `GET /slurm/v0.0.40/jobs/state`. > -- Fix sanity check when setting tres-per-task on the job allocation as well as > the step. > -- slurmrestd - Fix compatiblity with auth/slurm. > -- Fix issue where TRESRunMins gets off correct value if using > QOS UsageFactor != 1. > -- slurmrestd - Require `user` and `association_condition` fields to be > populated for requests to 'POST /slurmdb/v0.0.40/users_association'. > -- Avoid a slurmctld crash with extra_constraints enabled when a job requests > certain invalid --extra values. > -- `scancel --ctld` and `DELETE /slurm/v0.0/40/jobs` - Fix support for job > array expressions (e.g. 1_[3-5]). Also fix signaling a single pending array > task (e.g. 1_10), which previously signaled the whole array job instead. > -- Fix a possible slurmctld segfault when at some point we failed to create an > external launcher step. > -- Allow the slurmctld to open a connection to the slurmdbd if the first > attempt fails due to a protocol error. > -- mpi/cray_shasta - Fix launch for non-het-steps within a hetjob. > -- sacct - Fix "gpuutil" TRES usage output being incorrect when using --units. > -- Fix a rare deadlock on slurmctld shutdown or reconfigure. > -- Fix issue that only left one thread on each core available when "CPUs=" is > configured to total thread count on multi-threaded hardware and no other > topology info ("Sockets=", "CoresPerSocket", etc.) is configured. > -- Fix the external launcher step not being allocated a VNI when requested. > -- jobcomp/kafka - Fix payload length when producing and sending a message. > -- scrun - Avoid a crash if RunTimeDelete is called before the container > finishes. > -- Save the slurmd's cred_state while reconfiguring to prevent the loss job > credentials. [View Less]

1 0

SLUG Call for Papers Deadline
by Victoria Hobson 21 May '24

21 May '24

Slurm User Group (SLUG) 2024 is set for September 12-13 at the University of Oslo in Oslo, Norway. Registration information and a high-level schedule can be found here:https://slug24.splashthat.com/ The deadline to submit a presentation abstract is Friday, May 31st. We do not intend to extend this deadline. If you are interested in presenting your own usage, developments, site report, tutorial, etc about Slurm, please fill out the following form:https://forms.gle/N7bFo5EzwuTuKkBN7 … [View More]

1 0

Slurm not allocating correct cgroup cpu ids in srun step (possible bug)
by Ashley Wright 21 May '24

21 May '24

Hi, At our site we have recently upgraded to Slurm 23.11.5 and are having trouble with MPI jobs doing srun inside a sbatch'ed script. The cgroup does not appear to be setup correctly for the srun (step_0). As an example $ cat /sys/fs/cgroup/cpuset/slurm/uid_11000..../job..../cpuset.cpus 0,2-3,68-69,96,98-99,164-165 $ cat /sys/fs/cgroup/cpuset/slurm/uid_11000..../job..../step_0/cpuset.cpus 0,2,68,96,98,164 The sbatch is allocated a range of cpus in the cgroup. However, when step_0 is run, … [View More]

1 0

2025

2024

slurm-users ----- 2025 ----- May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users