- slurm-users - lists.schedmd.com

Mailing list upgrade - slurm-users list paused
by Tim Wickberg 30 Jan '24

30 Jan '24

Hey folks - The mailing list will be offline for about an hour as we upgrade the host, upgrade the mailing list software, and change the mail configuration around. As part of these changes, the "From: " field will no longer be the original sender, but instead use the mailing list ID itself. This is to comply with DMARC sending options, and allow us to start DKIM signing messages to ensure deliverability once Google and Yahoo impose new policy changes in February. This is the last post on the current (mailman2) list. I'll send a welcome message on the upgraded (mailman3) list once finished, and when the list is open to new traffic again. - Tim -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 1

Testing to SchedMD folks
by Tim Wickberg 30 Jan '24

30 Jan '24

Hey - this is hopefully a final test before the list is re-enabled for everyone. - Tim

1 0

Socket timed out - tuning
by Reed Dier 29 Jan '24

29 Jan '24

Hoping someone can help point me towards some tweaks to help prevent denial-of-service issues. > sbatch: error: Batch job submission failed: Socket timed out on send/recv operation Root cause is understood, issues with shared storage for the slurmctld’s was impacted, leading to an increase in write latency to the StateSaveLocation. Then with a large enough avalanche of job submissions, things the RPC’s would stack up and stop responding. I’ve been running well with some tweaks sourced from the “high-throughput” guide <https://slurm.schedmd.com/high_throughput.html>. > SchedulerParameters=max_rpc_cnt=400,\ > sched_min_interval=50000,\ > sched_max_job_start=300,\ > batch_sched_delay=6 > KillWait=30 > MessageTimeout=30 I’m assuming that I was running into batch_sched_delay because looking at sdiag after the fact, it was averaging .2s, and total time is 5.5h out of 16h8m18s at the time of the sdiag sample. > ******************************************************* > sdiag output at Thu Jan 25 11:08:18 2024 (1706198898) > Data since Wed Jan 24 19:00:00 2024 (1706140800) > ******************************************************* > REQUEST_SUBMIT_BATCH_JOB ( 4003) count:98400 ave_time:201442 total_time:19821991013 Currently on 22.05.8, but hoping to get to 23.02.7 soon™, and I think this could possible resolve the issue well enough if I’m reading it correctly from the release notes <https://slurm.schedmd.com/archive/slurm-23.02-latest/news.html>? > HIGHLIGHTS > ========== > -- slurmctld - Add new RPC rate limiting feature. This is enabled through > SlurmctldParameters=rl_enable, otherwise disabled by default. > rl_enable <https://slurm.schedmd.com/slurm.conf.html#OPT_rl_enable>Enable per-user RPC rate-limiting support. Client-commands will be told to back off and sleep for a second once the limit has been reached. This is implemented as a "token bucket", which permits a certain degree of "bursty" RPC load from an individual user before holding them to a steady-state RPC load established by the refill period and rate. But given that the hardware seems to be well over provisioned, CPU never drops below 5% idle, it feels like there is more room to squeeze some optimization out of here that I’m missing in the interim, and hoping to get a better overall understanding in the process. I scrape the DBD Agent queue size from sdiag every 30s and the largest value I saw was 115, which is much higher than normal, but should be well below MaxDBDMsgs, where the minimum value is 10000. I would really hope that I didn’t potentially hit a 30s MessageTimeout value, but I guess thats on the table all well, not knowing if that would potentially trigger the sbatch submission failure like that. Just moving the max_rpc_cnt value up seems like an easy button, but also seems like it could have some adverse effects for backfill scheduling, and may be diminishing returns for actually keeping RPCs flowing? > Setting max_rpc_cnt to more than 256 will be only useful to let backfill continue scheduling work after locks have been yielded (i.e. each 2 seconds) if there are a maximum of MAX(max_rpc_cnt/10, 20) RPCs in the queue. i.e. max_rpc_cnt=1000, the scheduler will be allowed to continue after yielding locks only when there are less than or equal to 100 pending RPCs. Obviously, fix the storage is the real solution, but hoping that there may be more goodness to unlock, even if it is as simple as “upgrade to 23.02”. Appreciate any insight, Reed

1 0

Two jobs each with a different partition running on same node?
by Loris Bennett 29 Jan '24

29 Jan '24

Hi, I seem to remember that in the past, if a node was configured to be in two partitions, the actual partition of the node was determined by the partition associated with the jobs running on it. Moreover, at any instance where the node was running one or more jobs, the node could only actually be in a single partition. Was this indeed the case and is it still the case with version Slurm 23.02.7? Cheers, Loris -- Dr. Loris Bennett (Herr/Mr) FUB-IT (ex-ZEDAT), Freie Universität Berlin

2 1

srun only runs one job on a node
by Rike-Benjamin Schuppner 27 Jan '24

27 Jan '24

Hi, Not sure if I do not understand the scheduling algorithm correctly of if this is a bug. The bug seems similar to the one described and fixed here https://github.com/SchedMD/slurm/commit/6a2c99edbf96e50463cc2f16d8e5eb955c8… Given the node definition: NodeName=cluster01 CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=64283 NodeName=cluster02 CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=128814 PartitionName=DEFAULT MaxTime=2-0 MaxNodes=2 PartitionName=mini Nodes=cluster01,cluster02 State=UP and the batch script: for i in {1..6} do srun --exclusive --mem=100 --threads-per-core=1 -N1 -c1 -n1 bash -c 'echo $(hostname) $(date); sleep 10 ; echo done $(hostname) $(date)' & done wait When run with: time sbatch --wait --partition=mini --ntasks=10 --cpus-per-task=1 --mem=1000 mini.slurm I get the following log after 2m 18s: cluster01 Sat Jan 27 02:05:12 PM CET 2024 cluster01 Sat Jan 27 02:05:12 PM CET 2024 cluster01 Sat Jan 27 02:05:13 PM CET 2024 cluster01 Sat Jan 27 02:05:13 PM CET 2024 cluster02 Sat Jan 27 02:05:13 PM CET 2024 done cluster01 Sat Jan 27 02:05:22 PM CET 2024 done cluster01 Sat Jan 27 02:05:22 PM CET 2024 done cluster01 Sat Jan 27 02:05:23 PM CET 2024 done cluster01 Sat Jan 27 02:05:23 PM CET 2024 done cluster02 Sat Jan 27 02:05:23 PM CET 2024 srun: Job 650 step creation temporarily disabled, retrying (Requested nodes are busy) srun: Step created for StepId=650.6 cluster01 Sat Jan 27 02:07:14 PM CET 2024 done cluster01 Sat Jan 27 02:07:24 PM CET 2024 So apparently I am getting 4 sruns on cluster01 (why not 6 when I have 8 CPUs?), but only 1 is scheduled on cluster02. Then it is exhausted and waits endlessly long (almost 2 minutes?) before scheduling again. The step log tells me: slurmctld: debug: laying out the 1 tasks on 1 hosts cluster01 dist 1 slurmctld: STEPS: _step_alloc_lps: JobId=650 StepId=4 node 0 (cluster01) gres_cpus_alloc=0 tasks=1 cpus_per_task=1 slurmctld: STEPS: _pick_step_cores: step JobId=650 StepId=4 requires 1 cores on node 0 with cpus_per_core=1, available cpus from job: 8 slurmctld: STEPS: _pick_step_core: alloc Node:0 Socket:0 Core:3 slurmctld: STEPS: step alloc on job node 0 (cluster01) used 4 of 8 CPUs slurmctld: STEPS: _slurm_rpc_job_step_create: JobId=650 StepId=4 cluster01 usec=5502 slurmctld: STEPS: _pick_step_nodes: JobId=650 Currently running steps use 4 of allocated 8 CPUs on node cluster01 slurmctld: STEPS: _pick_step_nodes: JobId=650 Currently running steps use 1 of allocated 2 CPUs on node cluster02 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=0 has nodes cluster01 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=1 has nodes cluster02 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=2 has nodes cluster01 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=3 has nodes cluster01 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=4 has nodes cluster01 slurmctld: STEPS: _pick_step_nodes: step pick 1-1 nodes, avail:cluster[01-02] idle: picked:NONE slurmctld: STEPS: _pick_step_nodes: step picked 0 of 1 nodes slurmctld: STEPS: Picked nodes cluster01 when accumulating from cluster01 slurmctld: debug: laying out the 1 tasks on 1 hosts cluster01 dist 1 slurmctld: STEPS: _step_alloc_lps: JobId=650 StepId=5 node 0 (cluster01) gres_cpus_alloc=0 tasks=1 cpus_per_task=1 slurmctld: STEPS: _pick_step_cores: step JobId=650 StepId=5 requires 1 cores on node 0 with cpus_per_core=1, available cpus from job: 8 slurmctld: STEPS: unable to pick step cores for job node 0 (cluster01): Requested nodes are busy slurmctld: STEPS: Deallocating 100MB of memory on node 0 (cluster01) now used: 400 of 1000 slurmctld: STEPS: step dealloc on job node 0 (cluster01) used: 4 of 8 CPUs slurmctld: STEPS: _slurm_rpc_job_step_create for JobId=650: Requested nodes are busy So why is it unable to allocate on cluster02? And why does its need to when it is only using 4 of 8 CPUs on cluster01 anyway? Now if I change to --ntasks=8, the algorithm doesn’t even schedule cluster02 at all: cluster01 Sat Jan 27 02:15:30 PM CET 2024 cluster01 Sat Jan 27 02:15:30 PM CET 2024 cluster01 Sat Jan 27 02:15:30 PM CET 2024 cluster01 Sat Jan 27 02:15:30 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 srun: Job 651 step creation temporarily disabled, retrying (Requested nodes are busy) srun: Step created for StepId=651.6 cluster01 Sat Jan 27 02:17:35 PM CET 2024 srun: Job 651 step creation temporarily disabled, retrying (Requested nodes are busy) srun: Step created for StepId=651.7 cluster01 Sat Jan 27 02:17:36 PM CET 2024 done cluster01 Sat Jan 27 02:17:45 PM CET 2024 done cluster01 Sat Jan 27 02:17:46 PM CET 2024 Thanks for your help. Best /rike

1 0

LuaSQLite3 with slurm-lua-spank
by Jackson, Gary L. 26 Jan '24

26 Jan '24

What do I need to do to use the LuaSQLite3 library with Lua SPANK? For what it’s worth, I don’t know anything about Lua. -- Gary

1 0

sinfo: error: resolve_ctls_from_dns_srv: res_nsearch error: Unknown host
by Michael Lewis 26 Jan '24

26 Jan '24

Hi All, I’m trying to get slurm-23.11.3 running on Ubuntu 20.04 and running on a stand alone system. I’m running into an issue I can not find the answer to. After compiling and installing when I fire up slurmctld and slurmd I get an error from sinfo: sinfo: error: resolve_ctls_from_dns_srv: res_nsearch error: Unknown host sinfo: error: fetch_config: DNS SRV lookup failed sinfo: error: _establish_config_source: failed to fetch config sinfo: fatal: Could not establish a configuration source I looks like a DNS issue but the system has no issue resolving to its hostname or localhost. The slurm.conf file is also being read properly as I have the logs directed to a place convenient to me. I see lots have had these same issues but cannot find a clear resolution. I have slurm running on a stand alone system in another lab with and identical setup without issue. Any advice would be greatly appreciated. Thanks, Mike

2 2

Database cluster
by Daniel L'Hommedieu 26 Jan '24

26 Jan '24

Community: What do you do to ensure database reliability in your SLURM environment? We can have multiple controllers and multiple slurmdbds, but my understanding is that slurmdbd can be configured with a single MySQL server, so what do you do? Do you have that “single MySQL server” be a cluster, such as Percona XtraDB? Do you use MySQL replication, then manually switch to slurmdbd to a replication slave if the master goes down? Do you do something else? Thanks. Daniel

6 7

Problem using Podman with scrun on SLURM 23.11.3
by Marcus Lauer 25 Jan '24

25 Jan '24

I am getting an unusual error when trying to run Podman containers using scrun on SLURM 23.11.3 (and 23.11.1 previously). In short, Podman works when not configured to use scrun, but when configured to use scrun it fails. Podman gives this error: scrun: fatal: Unable to request job allocation: Job cannot be submitted without the current working directory specified. The slurmctld logs show this error: _slurm_rpc_allocate_resources: Job cannot be submitted without the current working directory specified. The scrun command does not appear to have a --chdir option so the working directory must be detected automatically, and this appears to be failing. Perhaps someone has encountered this or a similar error before? If anyone has seen this before I would love to hear about your experiences. A bit more detail in case anyone is interested. This is a two-node test system with one head node and one compute node, both running Rocky Linux 8.9. I have created /etc/containers/storage.conf and containers.conf files which are identical to the ones in the SLURM Containers Guide except that I set the rootless_storage_path to the same value as the graphroot in storage.conf. When I get rid of the /etc/containers/containers.conf file I can run containers on the head node. The backing store is ext4, though I have tried NFS (which had other problems). I have tried the vfs and overlay storage drivers but get the same results. On the SLURM side I created an oci.conf which is the same as the "oci.conf example for crun using run (suggested)" from the SLURM Container Guide. I'm using crun because I have forced cgroup v2 on both systems ( systemd.unified_cgroup_hierarchy=1 on the kernel cmdline) and crun seems to support cgroup v2 better. I am using an scrun.lua which is similar to the one in the scrun manual page with some paths modified for my setup. I should also mention that sbatch jobs run just fine and srun works as well. the test cluster seems to be working fine in general. -- Marcus Lauer Systems Administrator CETS Group, Research Support

1 0

Question about CPUs and cores
by Gestió Servidors 25 Jan '24

25 Jan '24

Hi, I want to run a simple test that uses one node and four cores. Also, in my script, I execute a binary that reports me in what core is running one of the four tasks. These are my files: * submit script: #!/bin/bash #SBATCH --job-name=test_jobs # Job name #SBATCH --output=test_job_%j.out # Output file (using array index) #SBATCH --error=test_job_%j.err # Error file (using array index) #SBATCH --ntasks=4 #SBATCH --ntasks-per-core=1 #SBATCH --cpus-per-task=1 # Number of CPU cores per task echo "START : $(date)" srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & # Wait for all jobs to finish before exiting the job submission script wait echo "END : $(date)" * check_cores.c file: #include <stdio.h> #include<unistd.h> struct pstat { int pid; //process id char tcomm[256];//filename of the executable char state[2]; //state (R is running, S is sleeping, D is sleeping in an //uninterruptible wait, Z is zombie, T is traced or stopped) int ppid;// process id of the parent process int pgrp;// pgrp of the process int sid;// session id int tty_nr;// tty the process uses int tty_pgrp;// pgrp of the tty int flags;// task flags int min_flt;// number of minor faults int cmin_flt;// number of minor faults with child's int maj_flt;// number of major faults int cmaj_flt;// number of major faults with child's int utime;// user mode jiffies int stime;// kernel mode jiffies int cutime;// user mode jiffies with child's int cstime;// kernel mode jiffies with child's int priority;// priority level int nice;// nice level int num_threads;// number of threads int it_real_value;// (obsolete, always 0) int start_time;// time the process started after system boot int vsize;// virtual memory size int rss;// resident set memory size int rsslim;// current limit in bytes on the rss int start_code;// address above which program text can run int end_code;// address below which program text can run int start_stack;// address of the start of the stack int esp;// current value of ESP int eip;// current value of EIP int pending;// bitmap of pending signals int blocked;// bitmap of blocked signals int sigign;// bitmap of ignored signals int sigcatch;// bitmap of catched signals int wchan;// address where process went to sleep int i0;// (place holder) int i1;// (place holder) int exit_signal;// signal to send to parent thread on exit int task_cpu;// which CPU the task is scheduled on int rt_priority;// realtime priority int policy;// scheduling policy (man sched_setscheduler) int blkio_ticks;// time spent waiting for block IO int gtime;// guest time of the task in jiffies int cgtime;// guest time of the task children in jiffies } p ; int main() { char name[256]; char state[8]; FILE* f = fopen("/proc/self/stat", "r"); char hostname[1024]; gethostname(hostname, 1024); fscanf(f, "%d%s%s%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d", &p.pid, &p.tcomm, &p.state, &p.ppid, &p.pgrp, &p.sid, &p.tty_nr, &p.tty_pgrp, &p.flags, &p.min_flt, &p.cmin_flt, &p.maj_flt, &p.cmaj_flt, &p.utime, &p.stime, &p.cutime, &p.cstime, &p.priority, &p.nice, &p.num_threads, &p.it_real_value, &p.start_time, &p.vsize, &p.rss, &p.rsslim, &p.start_code, &p.end_code, &p.start_stack, &p.esp, &p.eip, &p.pending, &p.blocked, &p.sigign, &p.sigcatch, &p.wchan, &p.i0, &p.i1, &p.exit_signal, &p.task_cpu, &p.rt_priority, &p.policy, &p.blkio_ticks, &p.gtime, &p.cgtime); sleep (15); printf("HOST %s\n", hostname); printf("CPU %d\n", p.task_cpu); return 0; } When I run this script, I should receive an output like this: HOST clus04.hpc.local CPU 0 HOST clus04.hpc.local CPU 7 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 6 because system has scheduled my job in node "clus04.hpc.local" (a node with two 6-cores processors) and has reserved cores 0-1 of each processor. However, the real output I receive is HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 6 (and others outputs where there are one CPU repeated), so it seems SLURMs is reserving correctly four cores but, then, process is running two or more times in the same core. Slurmctld.log file in server shows this information: [...] [2024-01-25T09:38:42.867] DEBUG: JobId=206129 node clus04 vpus 1 cpus 4 [2024-01-25T09:38:42.867] ==================== [2024-01-25T09:38:42.867] JobId=206129 nhosts:1 ncpus:4 node_req:1 nodes=clus04 [2024-01-25T09:38:42.867] Node[0]: [2024-01-25T09:38:42.867] Mem(MB):0:0 Sockets:2 Cores:6 CPUs:4:0 [2024-01-25T09:38:42.867] Socket[0] Core[0] is allocated [2024-01-25T09:38:42.867] Socket[0] Core[1] is allocated [2024-01-25T09:38:42.867] Socket[1] Core[0] is allocated [2024-01-25T09:38:42.867] Socket[1] Core[1] is allocated [2024-01-25T09:38:42.867] -------------------- [2024-01-25T09:38:42.867] cpu_array_value[0]:4 reps:1 [2024-01-25T09:38:42.867] ==================== [2024-01-25T09:38:42.867] DEBUG: Dump job_resources: nhosts 1 cb 0-1,6-7 [...] [2024-01-25T09:38:58.213] _job_complete: JobId=206129 WEXITSTATUS 0 [2024-01-25T09:38:58.213] deallocate_nodes: JobId=206129 [2024-01-25T09:38:58.213] DEBUG: Dump job_resources: nhosts 1 cb 0-1,6-7 [2024-01-25T09:38:58.213] _job_complete: JobId=206129 done And slurmd.log file in node clus04 shows this information: [...] [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: step physical cores are '0-1,6-7' It seems that SLURM is reserving me correctly four cores for my job but, when inside my script I run four "srun", then each srun can freely take one of the four reserved cores, not exactly one to each task. What am I doing wrong? Thanks a lot!

1 0