- slurm-users - lists.schedmd.com

Re: slurmstepd: error: task_g_set_affinity: Operation not permitted
by Christopher Harrop - NOAA Affiliate 14 Jun '24

14 Jun '24

There is a permission problem somewhere, but I don’t know where. If I run as root, it works: admin@slurmfrontend:~$ srun hostname srun: error: task 0 launch failed: Slurmd could not execve job slurmstepd: error: task_g_set_affinity: Operation not permitted slurmstepd: error: _exec_wait_child_wait_for_parent: failed: No error admin@slurmfrontend:~$ sudo srun hostname slurmnode1 admin@slurmfrontend:~$ sudo srun -N 3 hostname slurmnode1 slurmnode3 slurmnode2 admin@slurmfrontend:~$ Chris --------------------------------------------------------------------------------------------------- Christopher W. Harrop voice: (720) 649-0316 NOAA Global Systems Laboratory, R/GSL6 fax: (303) 497-7259 325 Broadway Boulder, CO 80303

2 1

Re: Limit GPU depending on type
by Gestió Servidors 14 Jun '24

14 Jun '24

Hi, because of my real scenario (in mi first post I explained my testing scenario), with several differents users of differents types (researchers, university students and/or teachers, etc), I have distributed my GPUs in 3 differents partitions: * PartitionName=cuda-staff.q Nodes=gpu-[1-4] OverSubscribe=No MaxTime=INFINITE State=UP AllocNodes=node[0-22],node-login,node-login-bak AllowGroups=caos,profesor * PartitionName=cuda-int.q Nodes=gpu-[2,4] OverSubscribe=No MaxTime=30:00 State=UP AllocNodes=node[0-22] * PartitionName=cuda-ext.q Nodes=gpu-[1,3] OverSubscribe=No MaxTime=30:00 State=UP AllocNodes=node[0-22],node-login,node-login-bak Explanation: * In “cuda-staff.q”, only teachers could submit and they can submit from each lab node or each login node. * In “cuda-int.q” everybody can submit, but only from lab nodes. * In “cuda-ex.q” everybody can also submit, but in this case, from lab nodes and login nodes. Since now, I have not used “QoS”... but I’m going to install a new data/user server and I want to reconfigure SLURM. If I distributed GPUs in the way that Gerhard Strangar explains (both similar RTX3080) in a partition with restricted QoS and all other GPUs in other partition, some GPUs that now are restricted for “inside lab user” will be accessible from “outside lab user”. So I think (all teachers want this way ☹ ) I must have these partitions distribution. However, if I apply a QoS limiting only one GPU in each partition. it woulb be possible that a user could user one RTX3080 from outside lab and the other RTX3080 from inside lab… and this is what I want to deny. I will reread documentation. All help will be appreciated, of course!!!! Thanks.

1 0

Slurm version 23.11.8 is now available
by Marshall Garey 13 Jun '24

13 Jun '24

We are pleased to announce the availability of Slurm version 23.11.8. The 23.11.8 release fixes some potential crashes in slurmctld, slurmrestd, and slurmd when using less common features; two issues in auth/slurm; and a few other minor bugs. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Marshall > -- Fix slurmctld crash when reconfiguring with a PrologSlurmctld is running. > -- Fix slurmctld crash after a job has been resized. > -- Fix slurmctld and slurmdbd potentially stopping instead of performing a > logrotate when recieving SIGUSR2 when using auth/slurm. > -- Fix not having a disabled value for keepalive CommunicationParameters in > slurm.conf when these parameters are not set. This can log an error when > setting a socket, for example during slurmdbd registration with ctld. > -- switch/hpe_slingshot - Fix slurmctld crash when upgrading from 23.02. > -- Fix "Could not find group" errors from validate_group() when using > AllowGroups with large /etc/group files. > -- slurmrestd - Prevent a slurmrestd segfault when parsing the crontab field, > which was never usable. Now it explicitly ignores the value and emits a > warning if it is used for the following endpoints: > 'POST /slurm/v0.0.39/job/{job_id}' > 'POST /slurm/v0.0.39/job/submit' > 'POST /slurm/v0.0.40/job/{job_id}' > 'POST /slurm/v0.0.40/job/submit' > -- Fix getting user environment when using sbatch with "--get-user-env" or > "--export=" when there is a user profile script that reads /proc. > -- Prevent slurmd from crashing if acct_gather_energy/gpu is configured but > GresTypes is not configured. > -- Do not log the following errors when AcctGatherEnergyType plugins are used > but a node does not have or cannot find sensors: > "error: _get_joules_task: can't get info from slurmd" > "error: slurm_get_node_energy: Zero Bytes were transmitted or received" > However, the following error will continue to be logged: > "error: Can't get energy data. No power sensors are available. Try later" > -- Fix cloud nodes not being able to forward to nodes that restarted with new > IP addresses. > -- sacct - Fix printing of job group for job steps. > -- Fix error in scrontab jobs when using slurm.conf:PropagatePrioProcess=1. > -- Fix slurmctld crash on a batch job submission with "--nodes 0,...". > -- Fix dynamic IP address fanout forwarding when using auth/slurm.

1 0

slurmstepd: error: task_g_set_affinity: Operation not permitted
by Christopher Harrop - NOAA Affiliate 13 Jun '24

13 Jun '24

Hi, I am building a containerized Slurm cluster with Ubuntu 20.04 and have it almost working. The daemons start, and an “sinfo” command shows compute nodes up and available: admin@slurmfrontend:~$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST slurmpar* up infinite 3 idle slurmnode[1-3] admin@slurmfrontend:~$ However, if I try to use “srun” to test a job submission it fails saying it could not execve the job: admin@slurmfrontend:~$ srun hostname srun: error: task 0 launch failed: Slurmd could not execve job slurmstepd: error: task_g_set_affinity: Operation not permitted slurmstepd: error: _exec_wait_child_wait_for_parent: failed: No error admin@slurmfrontend:~$ If I go to the slurmnode1 container where the job should run, and look at the slurmd log, all I see is this: admin@slurmnode1:/$ sudo cat /var/log/slurmd.log [2024-06-13T14:58:36.238] CPU frequency setting not configured for this node [2024-06-13T14:58:36.239] warning: Core limit is only 0 KB [2024-06-13T14:58:36.239] slurmd version 23.11.7 started [2024-06-13T14:58:36.243] slurmd started on Thu, 13 Jun 2024 14:58:36 +0000 [2024-06-13T14:58:36.243] CPUs=8 Boards=1 Sockets=8 Cores=1 Threads=1 Memory=47926 TmpDisk=59767 Uptime=71713 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) [2024-06-13T14:58:47.230] launch task StepId=1.0 request from UID:1000 GID:1000 HOST:172.20.0.2 PORT:50618 [2024-06-13T14:58:47.230] task/affinity: lllp_distribution: JobId=1 implicit auto binding: sockets,one_thread, dist 8192 [2024-06-13T14:58:47.230] task/affinity: _task_layout_lllp_cyclic: _task_layout_lllp_cyclic [2024-06-13T14:58:47.230] task/affinity: _lllp_generate_cpu_bind: _lllp_generate_cpu_bind jobid [1]: mask_cpu,one_thread, 0x01 [2024-06-13T14:58:47.243] [1.0] error: task_g_set_affinity: Operation not permitted [2024-06-13T14:58:47.243] [1.0] error: _exec_wait_child_wait_for_parent: failed: No error [2024-06-13T14:58:47.244] [1.0] error: job_manager: exiting abnormally: Slurmd could not execve job [2024-06-13T14:58:47.247] [1.0] stepd_cleanup: done with step (rc[0xfb4]:Slurmd could not execve job, cleanup_rc[0xfb4]:Slurmd could not execve job) admin@slurmnode1:/$ I’ve installed by following the instructions for building/installing the Debian RPMs and can see that all the daemons are up and running. I have this slurm.conf (on all nodes): admin@slurmfrontend:~$ grep -v '#' /etc/slurm/slurm.conf ClusterName=cluster SlurmctldHost=slurmmaster MpiDefault=none ProctrackType=proctrack/linuxproc ReturnToService=1 SlurmdParameters=config_overrides SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=root StateSaveLocation=/var/spool/slurmctld SwitchType=switch/none TaskPlugin=task/affinity InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 SchedulerType=sched/backfill SelectType=select/cons_tres AccountingStorageType=accounting_storage/none JobCompType=jobcomp/none JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/none SlurmctldDebug=info SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurmd.log NodeName=slurmnode[1-3] CPUs=8 State=UNKNOWN PartitionName=slurmpar Nodes=ALL Default=YES MaxTime=INFINITE State=UP admin@slurmfrontend:~$ And I have this group.conf (on all nodes): admin@slurmfrontend:~$ grep -v '#' /etc/slurm/cgroup.conf CgroupPlugin=cgroup/v1 ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes admin@slurmfrontend:~$ Does anyone have any clues about where to look for why “srun” can’t run a job and where the "task_g_set_affinity: Operation not permitted” may be coming from? Chris --------------------------------------------------------------------------------------------------- Christopher W. Harrop voice: (720) 649-0316 NOAA Global Systems Laboratory, R/GSL6 fax: (303) 497-7259 325 Broadway Boulder, CO 80303

1 0

Limit GPU depending on type
by Gestió Servidors 13 Jun '24

13 Jun '24

Hello, I would like to know if it would be possible to limit, using "sacctmgr", use of a certain type of GPU according the name I have assigned in "gres.conf" file. For example, my small cluster has 3 GPUs nodes sharing 2 GPUs each one. Two of that GPUs are the same model but they are located in different servers. Because of my scenario, I would like to limit users to user only one of that GPU type and not allowing to use both of them. For example: * gpu-node-1: * GTX1080 * RTX3080 * gpu-node-2: * GTX750 * GTX680 * gpu-node-3: * RTX2070 * RTX3080 What I want is users could user all of them but simultaniously, a user only could use one of the RTX3080. Using QoS I have created a new "qos" with "sacctmgr add qos test-gpu-limit MaxTRESPerUser=gres/gpu=1", but with this new QoS, users are limited to use only one GPU, even they need to use different GPUs models. I have tried with "sacctmgr add qos test-gpu-limit MaxTRESPerUser=gres/gpu:RTX3080:1" but system returns this error "sacctmgr: error: slurmdb_format_tres_str: no TRES id found for gres/gpu:RTX3080:1". So, could be possible to apply limits I want to apply? Thanks. -- [cid:image003.jpg@01DABD6A.8DFD6D30]<http://www.uab.cat/> Daniel Ruiz Molina Tècnic Mitjà Informàtic Arquitectura de Computadors i Sistemes Operatius Escola d'Enginyeria Edifici Q - Despatx QC/3052 - Carrer de les Sitges Campus de la UAB · 08193 Bellaterra (Cerdanyola del Vallès) · Barcelona · Spain +34 93 581 35 44 www.uab.cat<http://www.uab.cat/> Daniel Ruiz at UAB<https://tinyurl.com/yd95zb8j> [cid:image004.jpg@01DABD6A.8DFD6D30]<www.linkedin.com/in/daniel-ruiz-molina-50a83b27> Aquest missatge s'adreça exclusivament a la persona destinatària i pot contenir informació privada o confidencial. Si l'heu rebut per error, comuniqueu-nos-ho i destruïu-lo, i tingueu present que no teniu autorització per fer-ne cap ús. Abans d'imprimir aquest missatge penseu en el medi ambient.

2 1

srun hostname - Socket timed out on send/recv operation
by Arnuld 11 Jun '24

11 Jun '24

I have two machines. When I run "srum hostname" on one machine (it's both a controller and a node) then I get the hostname fine but I get socket timed out error in these two situations: 1) "srun hostname" on 2nd machine (it's a node) 2) "srun -N 2 hostname" on controller "scontrol show node" shows both mach2 and mach4. "sinfo" shows both nodes too. Also the job gets stuck forever in CG state after the error. Here is the output: $ srun -N 2 hostname mach2 srun: error: slurm_receive_msgs: [[mach4]:6818] failed: Socket timed out on send/recv operation srun: error: Task launch for StepId=2222.0 failed on node hpc4: Socket timed out on send/recv operation srun: error: Application launch failed: Socket timed out on send/recv operation srun: Job step aborted Output form "squeue" 3 seconds apart Tue Jun 11 05:09:56 2024 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 2222 poxo hostname arnuld R 0:19 2 mach4,mach2 Tue Jun 11 05:09:59 2024 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 2222 poxo hostname arnuld CG 0:20 1 mach4

1 1

sbatch: Node count specification invalid - when only specifying --ntasks
by George Leaver 11 Jun '24

11 Jun '24

Hello, Previously we were running 22.05.10 and could submit a "multinode" job using only the total number of cores to run, not the number of nodes. For example, in a cluster containing only 40-core nodes (no hyperthreading), Slurm would determine two nodes were needed with only: sbatch -p multinode -n 80 --wrap="...." Now in 23.02.1 this is no longer the case - we get: sbatch: error: Batch job submission failed: Node count specification invalid At least -N 2 is must be used (-n 80 can be added) sbatch -p multinode -N 2 -n 80 --wrap="...." The partition config was, and is, as follows (MinNodes=2 to reject small jobs submitted to this partition - we want at least two nodes requested) PartitionName=multinode State=UP Nodes=node[081-245] DefaultTime=168:00:00 MaxTime=168:00:00 PreemptMode=OFF PriorityTier=1 DefMemPerCPU=4096 MinNodes=2 QOS=multinode Oversubscribe=EXCLUSIVE Default=NO All nodes are of the form NodeName=node245 NodeAddr=node245 State=UNKNOWN Procs=40 Sockets=2 CoresPerSocket=20 ThreadsPerCore=1 RealMemory=187000 slurm.conf has EnforcePartLimits = ANY SelectType = select/cons_tres TaskPlugin = task/cgroup,task/affinity A few fields from: sacctmgr show qos multinode Name|Flags|MaxTRES multinode|DenyOnLimit|node=5 The sbatch/srun man page states: -n, --ntasks .... If -N is not specified, the default behavior is to allocate enough nodes to satisfy the requested resources as expressed by per-job specification options, e.g. -n, -c and --gpus. I've had a look through release notes back to 22.05.10 but can't see anything obvious (to me). Has this behaviour changed? Or, more likely, what have I missed ;-) ? Many thanks, George -- George Leaver Research Infrastructure, IT Services, University of Manchester http://ri.itservices.manchester.ac.uk | @UoM_eResearch

2 3

Issue about selecting cpus for optimization
by Purvesh Parmar 10 Jun '24

10 Jun '24

Hi, We have 16 nodes cluster with DGX-A100 (80 GB). We have 128 cores of each node separated in to a separate partition for cpu only jobs and 8 GPUs and 128 cores in other partitions for cpugpu jobs. We want to ensure that only selected 128 cores should be part of the cpu partition. (NUMA / Symmetry) for optimization. How to achieve it? cores parameter in gres.conf help? Regards, Purvesh

1 0

scontrol create partition fails
by Long, Daniel S. 10 Jun '24

10 Jun '24

Hi, I need to temporarily dedicated one of our compute nodes to a single account. To do this, I was going to create a new partition but I'm running into an error where scontrol create partition outputs "scontrol: error: Invalid input: partition Request aborted" regardless of what parameters I give it. As far as I can tell, this should be allowed; the man page for scontrol has a whole section titled "Partitions - Specficiations for Create, Update, and Delete Commands". What am I missing? Also, is there a better way to approach this? This is really just a one or two day thing and I'm a little surprised there's no easy way to cordon off a node for a user or a project without spinning up an entire partition. Am I missing something obvious? Thanks for any help you can provide.

2 2

Software builds using slurm
by Duane Ellis 10 Jun '24

10 Jun '24

I have been lurking here for a while hoping to see some examples that would help but have not fit several months We have a slurm system setup for xilnix FPGA builds (hdl) I want to use this for sw builds too What I seem to see is slurm talks about cpus, GPUs and memory etc I am looking for a “run my make file (or shell script) on any available node” In our case we have 3 top level jobs A B and C These can all run in parallel and are independent (ie bootloader, linux kernel, and the Linux root file system via buildroot) Job A (boot) is actually about 7 small builds that are independent I am looking for a means to fork n jobs (ie job A B and C above) across the cluster and wait/collect the std output of those n jobs and the exit status Job A would then fork and build 7 to 8 sub jobs When they are done it would assemble the result into what Xilinix calls boot.bin Job B is a Linux kernel build Job C is buildroot so there are several (n=50) smaller builds ie bash, busybody, and other tools like python for the target agian each of these can be executed in parallel Really do not (cannot) re architect my build to be a slurm only build because it also need to be able to run without slurm ie build everything on my laptop without slurm present In that case the jobs would run serially and take an hour or so the hope is by parallelizing the sw build jobs our overall cycle time will improve It would also be nice if the slurm cluster would adapt to the available nodes automatically Our hope is we can run our lab pcs as duel boot they normally boot windows but we can duel boot them into Linux and they become a compile node and auto join the cluster and the cluster sees them as going off line when somebody reboots the machine back to windows Sent from my iPhone

3 2