- slurm-users - lists.schedmd.com

Slurm billback and sreport
by Chip Seraphine 05 Mar '24

05 Mar '24

Hello, I am attempting to implement a billback model and finding myself stymied by the way that sreport handles job arrays. Basically, when a user submits a large array, their usage includes time that jobs in the back of the array spend waiting their turn. (My #1 user in “sreport user topusage” shows more “used” cpu*minutes than the cluster physically _has_ during that interval.) However, jobs that are idle pending resources are simply regarded as pending; as a result, a “polite” user who submits an array of 1000 jobs running N at a time is penalized over a user who just dumps 1000 loose jobs into the queue. This incentives my users to do exactly what I do not want! Has anyone tried to bill their users based on the results of sreport? If so, how did you work around this problem? What did you use to determine the # of CPU*Minutes that a user actually allocated on during a given interval? -- Chip Seraphine Grid Operations For support please use help-grid in email or slack. This e-mail and any attachments may contain information that is confidential and proprietary and otherwise protected from disclosure. If you are not the intended recipient of this e-mail, do not read, duplicate or redistribute it by any means. Please immediately delete it and any attachments and notify the sender that you have received it by mistake. Unintended recipients are prohibited from taking action on the basis of information in this e-mail or any attachments. The DRW Companies make no representations that this e-mail or any attachments are free of computer viruses or other defects.

3 3

Is SWAP memory mandatory for SLURM
by John Joseph 04 Mar '24

04 Mar '24

Dear All, Good morning I do have a 4 node SLURM instance up and running. Like to know if I disable the SWAP memory, will it effect the SLURM performance Is SWAP a mandatory requirement, I have each node more RAM, if my phsicall RAM is more, is there any need for the SWAP thanks Joseph John

4 3

What is "eligible time"
by Oren Shani 04 Mar '24

04 Mar '24

Hi All, Can someone shed some light on the exact meaning of the "eligible time" attribute in the slurm accounter, or to be more exact on the time between when the jobs becomes "eligible" to the time it actually starts to run? I understand that "eligible time" refers to the time in which the job becomes a candidate for running and that one case in which it will still not run, is that it waits for resources to become available. But are there any other conditions for a job to keep waiting at this point? For example, is fair-share scheduling considered only at this point? Many thanks, Oren

1 0

Question about IB and Ethernet networks
by Dan Healy 04 Mar '24

04 Mar '24

Hi Fellow Slurm Users, This question is not slurm-specific, but it might develop into that. My question relates to understanding how *typical* HPCs are designed in terms of networking. To start, is it typical for there to be a high speed Ethernet *and* Infiniband networks (meaning separate switches, NICs)? I know you can easily setup IP over IB, but is IB usually fully reserved for MPI messages? I’m tempted to spec all new HPCs with only a high speed (200Gbps) IB network, and use IPoIB for all slurm comms with compute nodes. I plan on using BeeGFS for the file system with RDMA. Just looking for some feedback, please. Is this OK? Is there a better way? If yes, please share why it’s better. Thanks, Daniel Healy

6 6

How to get usage data for a QOS
by thomas.hartmann＠th-ht.de 01 Mar '24

01 Mar '24

Hi, so, I figured out that I can give some users priority access for a specific amount of TRES by creating a qos with the GrpTRESMins property and the DenyOnLimit,NoDecay flags. This works nicely. However, I would like to know, how much of this has already been consumed and I have not yet found a way to do this. Like: How can I get the amount of TRES/TRES minutes consumed for a certain QOS? Thanks a lot! Thomas

3 2

MinTRES in QoS and power saving
by Stefan Walter 01 Mar '24

01 Mar '24

I am running a GPU cluster where nodes are mostly off to save on electricity. I have run into the problem that if I set 'MinTRES=gres/gpu=1' in the QoS for user-account associations, waking up nodes on-demand stops working for these users. Jobs are allocated on all running nodes but if a user starts a job that would require a node to be woken up, it remains pending with 'QOSMinGRES'. In the verbose logs, such a job will show this: debug3: TRES Weight: cpu = 1.000000 * 0.000000 = 0.000000 debug3: TRES Weight: mem = 6000.000000 * 0.000000 = 0.000000 debug3: TRES Weight: energy = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: node = 1.000000 * 0.000000 = 0.000000 debug3: TRES Weight: fs/disk = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: vmem = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: pages = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: gres/gpu = 0.000000 * 1.000000 = 0.000000 debug3: TRES Weight: gres/gpu:nvidia_geforce_gtx_1080ti = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: gres/gpumem = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weight: gres/gpuutil = 0.000000 * 0.000000 = 0.000000 debug3: TRES Weighted: SUM(TRES) = 0.000000 debug2: JobId=4256 is being held, QOS local818-test min tres(gres/gpu) per job request 0 exceeds min tres limit 1 The job was started with '-G 1' or '--gres=gpu:1' but notably the TRES value for 'gres/gpu' becomes 0. For jobs that fit on the running nodes the TRES weight remains correct: debug3: TRES Weight: gres/gpu = 1.000000 * 1.000000 = 1.000000 The quick fix was to unset MinTRES for all QoS but I still would force user in some accounts to use GPUs because that is what is billed for them. Is this by design? I was not successful to find any hint that this is expected behavior. Thanks, Stefan

1 0

pty jobs are killed when another job on the same node terminates
by Thomas Hartmann 01 Mar '24

01 Mar '24

Hi, when I start an interactive job like this: srun --pty --mem=3G -c2 bash And then I schedule and run other jobs (can be interactive or non interactive) and one of these jobs that runs on the same node terminates, the interactive job gets killed with this message: srun: error: node01.abc.at: task 0: Killed I attached our slurm config. Does anybody have an idea what is going on here or where I could look to debug? I'm quite new to slurm, so I don't know all the places to look... Thanks a lot in advance! Thomas

4 3

GPU shards not exclusive
by Reed Dier 29 Feb '24

29 Feb '24

I seem to have run into an edge case where I’m able to oversubscribe a specific subset of GPUs on one host in particular. Slurm 22.05.8 Ubuntu 20.04 cgroups v1 (ProctrackType=proctrack/cgroup) It seems to be partly a corner case with a couple of caveats. This host has 2 different GPU types in the same host. All GPUs are configured with 6 shards/gpu. > NodeName=gpu03 CPUs=88 RealMemory=768000 Sockets=2 CoresPerSocket=22 ThreadsPerCore=2 State=UNKNOWN Feature=avx512 Gres=gpu:rtx:2,gpu:p40:6,shard:48 My gres.conf is auto-nvml, with nothing else in the file. > AutoDetect=nvml Schedule and select from slurm.conf > SchedulerType=sched/backfill > SelectType=select/cons_tres > SelectTypeParameters=CR_CPU_Memory > GresTypes=gpu,shard Here’s a copy of slurmd’s log starting up with debug logging: https://pastebin.com/fbriTAZD <https://pastebin.com/fbriTAZD> I checked there to make sure that it was allocating an even 6 shards per GPU as intended, and it did. The only thing odd about the rtx cards is that they have an NVLink bridge between them, but I wouldn’t expect that to have any impact with respect to shards. So far it only appears to manifest if anything is using the named gpu (--gres gpu:rtx:1), but not as a general gres (--gres gpu:1). It also doesn’t occur with any count of named p40 gres either, only with the rtx gres. The below was as “high” as I was able to capture it oversubscribed in very quick testing. You can see there are 2 rtx and 43 shards allocated, which should be an impossible number, if we correctly assume an rtx==6 shards, and with 48 total shards, that should reduce the available pool of shards to 36 shards. > NodeName=gpu03 Arch=x86_64 CoresPerSocket=22 > CPUAlloc=88 CPUEfctv=88 CPUTot=88 CPULoad=0.27 > AvailableFeatures=avx512 > ActiveFeatures=avx512 > Gres=gpu:p40:6(S:0),gpu:rtx:2(S:0),shard:p40:36(S:0),shard:rtx:12(S:0) > NodeAddr=gpu03 NodeHostName=gpu03 Version=22.05.8 > OS=Linux 5.4.0-164-generic #181-Ubuntu SMP Fri Sep 1 13:41:22 UTC 2023 > RealMemory=768000 AllocMem=45056 FreeMem=563310 Sockets=2 Boards=1 > State=ALLOCATED+RESERVED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A > Partitions=gpu,gpu-prod > BootTime=2024-02-02T14:17:07 SlurmdStartTime=2024-02-14T16:28:17 > LastBusyTime=2024-02-14T16:38:03 > CfgTRES=cpu=88,mem=750G,billing=88,gres/gpu=8,gres/gpu:p40=6,gres/gpu:rtx=2,gres/shard=48 > AllocTRES=cpu=88,mem=44G,gres/gpu=2,gres/gpu:rtx=2,gres/shard=43 > CapWatts=n/a > CurrentWatts=0 AveWatts=0 > ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s The above is 1 rtx:2 jobs, and 43 shard:1 jobs, which is as many as I can seem to push through with the given cpu cores on the system, I expect if there were 98-100 cores I could completely oversubscribe it. If I do (1) rtx:2 + (24) shard:2 it actually limits it correctly to 36 shards, which is odd because the case that caused us to discover this was when 2 rtx:1 jobs were running for multiple days, and somehow shards got scheduled on those gpus. > AllocTRES=cpu=40,mem=20G,gres/gpu=2,gres/gpu:rtx=2,gres/shard=36 If I flip it around to shards before rtx, then I’m unable to oversubscribe it. If I add any P40 jobs before the rtx and then shard jobs, also unable to oversubscribe. This sounded familiar to Bug 16484 <https://bugs.schedmd.com/show_bug.cgi?id=16484> although they claimed it was resolved by using auto-nvml which would preclude this node. Here’s my very simple bash script that just srun's a sleep 20 so that it runs long enough to observe it. > #!/bin/bash > #rtx 2 > for((i=1;i<=1;++i)) ; > do > srun --gres gpu:rtx:2 --reservation=shardtest -J "RTX $i" sleep 20 & > done > sleep 1 > #shard 48 > for((i=1;i<=43;++i)) ; > do > srun --gres shard:1 --reservation=shardtest -J "shard $i" sleep 20 & > done > sleep 5 ; scontrol show node gpu03 Curious if anyone has any ideas beyond try a newer release? Thanks, Reed

2 2

sbatch and cgroup v2
by Dietmar Rieder 29 Feb '24

29 Feb '24

Hi, I'm new to slrum, but maybe someone can help me: I'm trying to restrict the CPU usage to the actually requested/allocated resources using cgroup v2. For this I made the following settings in slurmd.conf: ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup,task/affinity And in cgroup.conf CgroupPlugin=cgroup/v2 CgroupAutomount=yes ConstrainCores=yes ConstrainRAMSpace=yes ConstrainDevices=yes AllowedRAMSpace=98 cgroup v2 seems to be active on the compute node: # mount | grep cgroup cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) # cat /sys/fs/cgroup/cgroup.subtree_control cpuset cpu io memory pids # cat /sys/fs/cgroup/system.slice/cgroup.subtree_control cpuset cpu io memory pids Now, when I use sbatch to submit the following test script, the python script which is started from the batch script is utilizing all CPUs (96) at 100% on the allocated node, although I only ask for 4 cpus (--cpus-per-task=4). I'd expect that the task can not use more that these 4. #!/bin/bash #SBATCH --output=/local/users/appadmin/test-%j.log #SBATCH --job-name=test #SBATCH --chdir=/local/users/appadmin #SBATCH --cpus-per-task=4 #SBATCH --ntasks=1 #SBATCH --nodes=1 #SBATCH --mem=64gb #SBATCH --time=4:00:00 #SBATCH --partition=standard #SBATCH --gpus=0 #SBATCH --export #SBATCH --get-user-env=L export PATH=/usr/local/bioinf/jupyterhub/bin:/usr/local/bioinf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/bioinf/miniforge/condabin source .bashrc conda activate test python test.py The python code in test.py is the following using the cpu_load_generator package from [1]: #!/usr/bin/env python import sys from cpu_load_generator import load_single_core, load_all_cores, from_profile load_all_cores(duration_s=120, target_load=1) # generates load on all cores Interestingly, when I use srun to launch an interactive job, and run the python script manually, I see with top that only 4 cpus are running at 100%. And I also python errors thrown when the script tries to start the 5th process (which makes sense): File "/usr/local/bioinf/miniforge/envs/test/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/local/bioinf/miniforge/envs/test/lib/python3.12/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/bioinf/miniforge/envs/test/lib/python3.12/site-packages/cpu_load_generator/_interface.py", line 24, in load_single_core process.cpu_affinity([core_num]) File "/usr/local/bioinf/miniforge/envs/test/lib/python3.12/site-packages/psutil/__init__.py", line 867, in cpu_affinity self._proc.cpu_affinity_set(list(set(cpus))) File "/usr/local/bioinf/miniforge/envs/test/lib/python3.12/site-packages/psutil/_pslinux.py", line 1714, in wrapper return fun(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/bioinf/miniforge/envs/test/lib/python3.12/site-packages/psutil/_pslinux.py", line 2213, in cpu_affinity_set cext.proc_cpu_affinity_set(self.pid, cpus) OSError: [Errno 22] Invalid argument What am I missing, why are the CPU resources not restricted when I use sbatch? Thanks for any input or hint Dietmar [1]: https://pypi.org/project/cpu-load-generator/

3 7

slurmdbd error - Symbol `slurm_conf' has different size in shared object
by Miriam Olmi 29 Feb '24

29 Feb '24

Hi all, I am having some issue with the new version of slurm 23.11.0-1. I had already installed and configured slurm 23.02.3-1 on my cluster and all the services were active and running properly. Following the instructions of the official SLURM webpage, for the moment I upgrated only the slurmdbd service. In principle the cluster should be able to work properly if the slurmdbd has a higher version with respect to slurmctld and slurmd. Unfortunately the slurmdbd service fails to start with the following status: slurmdbd.service - Slurm DBD accounting daemon Loaded: loaded (/usr/lib/systemd/system/slurmdbd.service; enabled; vendor preset: disabled) Active: failed (Result: core-dump) since Wed 2024-02-28 10:05:53 CET; 9min ago Process: 534938 ExecStart=/usr/sbin/slurmdbd -D -s $SLURMDBD_OPTIONS (code=dumped, signal=SEGV) Main PID: 534938 (code=dumped, signal=SEGV) Feb 28 10:05:53 slurm-db systemd[1]: Started Slurm DBD accounting daemon. Feb 28 10:05:53 slurm-db slurmdbd[534938]: /usr/sbin/slurmdbd: Symbol `slurm_conf' has different size in shared object, consider re-linking Feb 28 10:05:53 slurm-db systemd[1]: slurmdbd.service: Main process exited, code=dumped, status=11/SEGV Feb 28 10:05:53 slurm-db systemd[1]: slurmdbd.service: Failed with result 'core-dump'. Can anyone help me? Thanks in advance, Miriam

2 4