- slurm-users - lists.schedmd.com

salloc not starting shell despite LaunchParameters=use_interactive_step
by Loris Bennett 06 Sep '24

06 Sep '24

Hi, With $ salloc --version slurm 23.11.10 and $ grep LaunchParameters /etc/slurm/slurm.conf LaunchParameters=use_interactive_step the following $ salloc --partition=interactive --ntasks=1 --time=00:03:00 --mem=1000 --qos=standard salloc: Granted job allocation 18928869 salloc: Nodes c001 are ready for job creates a job $ squeue --me JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 18928779 interacti interact loris R 1:05 1 c001 but causes the terminal to block. From a second terminal I can log into the compute node: $ ssh c001 [13:39:36] loris@c001 (1000) ~ Is that the expected behaviour or should salloc return a shell directly on the compute node (like srun --pty /bin/bash -l used to do)? Cheers, Loris -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin

5 8

Nodelist syntax and semantics
by Jackson, Gary L. 05 Sep '24

05 Sep '24

Is there a description of the “nodelist” syntax and semantics somewhere other than the source code? By “nodelist” I mean expressions like “name[000,099-100]” and how this one, for example, expands to “name000, name099, name100”. -- Gary

2 1

Configuration for nodes with different TmpFs locations and TmpDisk sizes
by Jake Longo 05 Sep '24

05 Sep '24

Hi all, We have a number of machines in our compute cluster that have larger disks available for local data. I would like to add them to the same partition as the rest of the nodes but assign them a larger TmpDisk value which would allow users to request a larger tmp and land on those machines. The main hurdle is that (for reasons beyond my control) the larger local disks are on a special mount point /largertmp whereas the rest of the compute cluster uses the vanilla /tmp. I can't see an obvious way to make this work as the TmpFs value appears to be global only and attempting to set TmpDisk to a value larger than TmpFs for those nodes will put the machine into an invalid state. I couldn't see any similar support tickets or anything in the mail archive but I wouldn't have thought it would be that unusual to do this. Thanks in advance! Jake

2 1

Make a job pending in the plugin
by benjamin.jin＠bos-semi.com 05 Sep '24

05 Sep '24

Hello all, I am tyring to build a custom plugin to force some jobs to be pended. In the official document, `ESLURM*` errors are only valid for `job_submit_lua`. I tried to send `ESLURM_JOB_PENDING`, but it only rejects the job submission. Does anyone know how to pend a job in job_submit plugin? Thanks.

2 1

Bug? sbatch not respecting MaxMemPerNode setting
by Angel de Vicente 05 Sep '24

05 Sep '24

Hello, we found an issue with Slurm 24.05.1 and the MaxMemPerNode setting. Slurm is installed in a single workstation, and thus, the number of nodes is just 1. The relevant sections in slurm.conf read: ,---- | EnforcePartLimits=ALL | PartitionName=short Nodes=..... State=UP Default=YES MaxTime=2-00:00:00 MaxCPUsPerNode=76 MaxMemPerNode=231000 OverSubscribe=FORCE:1 `---- Now, if I submit a job requesting 76 CPUs and each one needing 4000M (for a total of 304000M), Slurm does indeed respect the MaxMemPerNode setting and the job is not submitted in the following cases ("-N 1" is not really necessary, as there is only one node): ,---- | $ sbatch -N 1 -n 1 -c 76 -p short --mem-per-cpu=4000M test.batch | sbatch: error: Batch job submission failed: Memory required by task is not available | | $ sbatch -N 1 -n 76 -c 1 -p short --mem-per-cpu=4000M test.batch | sbatch: error: Batch job submission failed: Memory required by task is not available | | $ sbatch -n 1 -c 76 -p short --mem-per-cpu=4000M test.batch | sbatch: error: Batch job submission failed: Memory required by task is not available `---- But with this submission Slurm is happy: ,---- | $ sbatch -n 76 -c 1 -p short --mem-per-cpu=4000M test.batch | Submitted batch job 133982 `---- and the slurmjobcomp.log file does indeed tell me that the memory went above MaxMemPerNode: ,---- | JobId=133982 UserId=......(10487) GroupId=domain users(2000) Name=test JobState=CANCELLED Partition=short TimeLimit=45 StartTime=2024-09-04T09:11:17 EndTime=2024-09-04T09:11:24 NodeList=...... NodeCnt=1 ProcCnt=76 WorkDir=/tmp/. ReservationName= Tres=cpu=76,mem=304000M,node=1,billing=76 Account=ddgroup QOS=domino WcKey= Cluster=...... SubmitTime=2024-09-04T09:11:17 EligibleTime=2024-09-04T09:11:17 DerivedExitCode=0:0 ExitCode=0:0 `---- What is the best way to report issues like this to the Slurm developers? I thought of adding it to https://support.schedmd.com/ but it is not clear to me if that page is only meant for Slurm users with a Support Contract? Cheers, -- Ángel de Vicente Research Software Engineer (Supercomputing and BigData) Instituto de Astrofísica de Canarias (https://www.iac.es/en)

2 3

Slurm Remote Task Launch
by Bhaskar Chakraborty 04 Sep '24

04 Sep '24

Hello, I have a slurm job which needs to launch multiple tasks across the allocated hosts for the job. My criteria is that most of the tasks need to be launched from within the main task launchedby slurm in the launch compute node. So, if the allocated hosts are h1, h2 & h3 with h1 being the main launcher node then theinitial task, say launchTask, launched in h1 will need to launch RemoteTask1, RemoteTask2 in h2 & h3 at somepoint during execution. Can I use srun from inside launchTask to do so? If yes, what would be the syntax / args?If no, then what alternative I have other than using rsh/ssh which mayn't be available in the cluster. Thanks in advance! Regards,Bhaskar.

1 0

Best practices for tracking jobs started across multiple clusters for accounting purposes.
by Di Bernardini, Fabio 02 Sep '24

02 Sep '24

I need to account for jobs composed of multiple jobs launched on multiple federated (and non-federated) clusters, which therefore have different job IDs. What are the best practices to prevent users from bypassing this tracking? NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale: 10.329,14 EUR i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico

3 4

QOS MaxTRESPU node=X intepretation
by David Magda 30 Aug '24

30 Aug '24

Hello, Have a question on how to interpret a Node=X MaxTRESPU value for a QOS: If (e.g.) X=4, and each node has (say) 64 CPUs or cores: if a particular job needs 32 cores, then would two jobs count as the equivalent of one node (2*32=64)? And if X=4 for the QOS, would that mean that eight of these jobs would be able to run ((4*64)/32=8)? Or does this TRES limit only get activated -N/--nodes is used? Don't see anything obvious: https://slurm.schedmd.com/resource_limits.html https://slurm.schedmd.com/sacctmgr.html https://slurm.schedmd.com/tres.html Thanks for any info. Regards, David

1 0

playing with --nodes=<size_string>
by Matteo Guglielmi 30 Aug '24

30 Aug '24

Hello, I have a cluster with four Intel nodes (node[01-04], Feature=intel) and four Amd nodes (node[05-08], Feature=amd). # job file #SBATCH --ntasks=3 #SBATCH --nodes=2,4 #SBATCH --constraint="[intel|amd]" env | grep SLURM # slurm.conf PartitionName=DEFAULT MinNodes=1 MaxNodes=UNLIMITED # log SLURM_JOB_USER=software SLURM_TASKS_PER_NODE=1(x3) SLURM_JOB_UID=1002 SLURM_TASK_PID=49987 SLURM_LOCALID=0 SLURM_SUBMIT_DIR=/home/software SLURMD_NODENAME=node01 SLURM_JOB_START_TIME=1724932865 SLURM_CLUSTER_NAME=cluster SLURM_JOB_END_TIME=1724933465 SLURM_CPUS_ON_NODE=1 SLURM_JOB_CPUS_PER_NODE=1(x3) SLURM_GTIDS=0 SLURM_JOB_PARTITION=nodes SLURM_JOB_NUM_NODES=3 SLURM_JOBID=26 SLURM_JOB_QOS=lprio SLURM_PROCID=0 SLURM_NTASKS=3 SLURM_TOPOLOGY_ADDR=node01 SLURM_TOPOLOGY_ADDR_PATTERN=node SLURM_MEM_PER_CPU=0 SLURM_NODELIST=node[01-03] SLURM_JOB_ACCOUNT=dalco SLURM_PRIO_PROCESS=0 SLURM_NPROCS=3 SLURM_NNODES=3 SLURM_SUBMIT_HOST=master SLURM_JOB_ID=26 SLURM_NODEID=0 SLURM_CONF=/etc/slurm/slurm.conf SLURM_JOB_NAME=mpijob SLURM_JOB_GID=1002 SLURM_JOB_NODELIST=node[01-03] <<<=== why three nodes? Shouldn't this still be two nodes? Thank you.

2 5

Print Slurm Stats on Login
by Paul Edmon 29 Aug '24

29 Aug '24

We are working to make our users more aware of their usage. One of the ideas we came up with was to having some basic usage stats printed at login (usage over past day, fairshare, job efficiency, etc). Does anyone have any scripts or methods that they use to do this? Before baking my own I was curious what other sites do and if they would be willing to share their scripts and methodology. -Paul Edmon-

10 20