April 2024 - slurm-users - lists.schedmd.com

job_container/tmpfs and srun.
by Phill Harvey-Smith 17 Jan '25

17 Jan '25

Hi all, On our setup we are using job_container/tmpfs to give each job it's own temp space. Since our compute nodes have reasonably sized disks for tasks that do a lot of disk I/O on user's data we have asked users to copy their data to the local disk at the beginning of the task and (if needed) copy it back at the end. This saves lots of NFS thrashing slowing down both the task and the NFS servers. However some of our users are having problems with this, their initial sbatch script will create a temp directory in their private /tmp copy their data to it and then try to srun a program. The srun will fall over as it doesn't seem to have have access to the copied data. I suspect this is because the srun task is getting it's own private /tmp. So my question is, is there a way to have the srun task inherit the /tmp of the initial sbatch? I'll include a sample of the script our user is using below. If any further information is required please feel free to ask. Cheers. Phill. #!/usr/bin/bash #SBATCH --nodes 1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --time=00:00:10 #SBATCH --mem-per-cpu=3999 #SBATCH --output=script_out.log #SBATCH --error=script_error.log # The above options puts the STDOUT and STDERR of sbatch in # log files prefixed with 'script_'. # Create a randomly-named directory under /tmp jobtmpdir=$(mktemp -d) # Register a function to try and cleanup in case of job failure cleanup_handler() { echo "Cleaning up ${jobtmpdir}" rm -rf ${jobtmpdir} } trap 'cleanup_handler' SIGTERM EXIT # Change working directory to this directory cd ${jobtmpdir} # Copy the executable and input files from # where the job was submitted to the temporary directory. cp ${SLURM_SUBMIT_DIR}/a.out . cp ${SLURM_SUBMIT_DIR}/input.txt . # Run the executable, handling the collection of stdout # and stderr ourselves by redirecting to file srun ./a.out 2> task_error.log > task_out.log # Copy output data back to the submit directory. cp output.txt ${SLURM_SUBMIT_DIR} cp task_out.log ${SLURM_SUBMIT_DIR} cp task_error.log ${SLURM_SUBMIT_DIR} # Cleanup cd ${SLURM_SUBMIT_DIR} cleanup_handler

2 1

AllowAccounts partition setting
by Marko Markoc 17 Oct '24

17 Oct '24

Hi All, At one point, when we were running slurm 22, we set up a few new partitions with `AllowAccounts` setting enabled. We limit access to these partitions only to a few accounts. We manually create these accounts and users using the `sacctmgr` command. At that time, everything was working. Only users in these accounts were able to run jobs in that partition. Few months ago we upgraded slurm to 23.02.7 but just a couple of days ago we noticed that anyone could submit jobs to these partitions. In addition to that, you can provide any value to the `--account` directive and slurm will accept and record it in the job accounting database. In our initial testing, you could only provide `--account` value that has been previously created. I'm still looking into this but I wanted to check if there was any change in version 23 that could have impacted this? I don't remember seeing anything in release notes. Thanks, Marko

6 8

Can't schedule on cloud node: State=IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING
by Xaver Stiensmeier 20 Sep '24

20 Sep '24

Dear slurm-user list, I have a cloud node that is powered up and down on demand. Rarely it can happen that slurm's resumeTimeout is reached and the node is therefore powered down. We have set ReturnToService=2 in order to avoid the node being marked down, because the instance behind that node is created on demand and therefore after a failure nothing stops the system to start the node again as it is a different instance. I thought this would be enough, but apparently the node is still marked with "NOT_RESPONDING" which leads to slurm not trying to schedule on it. After a while NOT_RESPONDING is removed, but I would like to move it directly from within my fail script if possible so that the node can return to service immediately and not be blocked by "NOT_RESPONDING". Best regards, Xaver

2 3

Job Invalid Account
by Joe Teumer 15 May '24

15 May '24

We installed slurm 23.11.5 and we are receiving "JobId=n has invalid account" for every sbatch job. We are not using the slurm accounting/user database; we are using uniform UIDs and GIDs across the cluster. The jobs run and complete; can these invalid account errors be ignored or silenced? Job Submission Environment: id joteumer uid=938401109(joteumer) gid=938400513(SPG) groups=938400513(SPG),27(sudo) Slurm Worker Node: id joteumer uid=938401109(joteumer) gid=938400513(SPG) groups=938400513(SPG),27(sudo) slurmctld log: [2024-04-18T09:46:40.000] sched: JobId=18 has invalid account scontrol show job 18 JobId=18 JobName=simplejob.sh UserId=joteumer(938401109) GroupId=SPG(938400513) MCS_label=N/A Priority=1 Nice=0 *Account=(null) *QOS=(null) Submit another sbatch job and update the job to include an Account scontrol update jobid=19 Account=joteumer [2024-04-18T09:56:05.126] _slurm_rpc_submit_batch_job: JobId=19 InitPrio=1 usec=485 [2024-04-18T09:56:06.000] sched: JobId=19 has invalid account [2024-04-18T09:56:17.000] debug: set_job_failed_assoc_qos_ptr: Filling in assoc for JobId=19 Assoc=0 [2024-04-18T09:56:17.000] sched: JobId=19 has invalid account [2024-04-18T09:56:17.588] debug: set_job_failed_assoc_qos_ptr: Filling in assoc for JobId=19 Assoc=0 [2024-04-18T09:56:27.505] _slurm_rpc_update_job: complete JobId=19 uid=0 usec=110 [2024-04-18T09:56:28.000] sched: JobId=19 has invalid account scontrol show job 19 JobId=19 JobName=simplejob.sh UserId=joteumer(938401109) GroupId=SPG(938400513) MCS_label=N/A Priority=1 Nice=0 Account=(null) QOS=(null) JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) 19 SPG simplejob joteumer PENDING 0:00 18:00:00 1 (InvalidAccount)

2 1

any way to allow interactive jobs or ssh in Slurm 23.02 when node is draining?
by Robert Kudyba 13 May '24

13 May '24

We use Bright Cluster Manager with SLurm 23.02 on RHEL9. I know about pam_slurm_adopt https://slurm.schedmd.com/pam_slurm_adopt.html which does not appear to come by default with the Bright 'cm' package of Slurm. Currently ssh to a node gets: Login not allowed: no running jobs and no WLM allocations We have 8 GPUs on a node so when we drain a node, which can have up to a 5 day job, no new jobs can run. And since we have 20+ TB (yes TB) local drives, researchers have their work and files on them to retrieve. Is there a way to use /etc/security/access.conf to work around this at least temporarily until the reboot and then we can revert? Thanks! Rob

2 2

Trouble Running Slurm C Extension Plugin
by Glen MacLachlan 10 May '24

10 May '24

Hi, We have a plugin in Lua that mostly does what we want but there are features available in the C extension that are not available to lua. For that reason, we are attempting to convert to C using the guidance found here: https://slurm.schedmd.com/job_submit_plugins.html#building. We arrived here because the lua plugins don't seem to stretch enough to cover the use case we were looking at, i.e., branching off of the value of alloc_id or, for that matter, get_sid(). The goal is to disallow interactive allocations (i.e., salloc) on specific partitions while allowing it on others. However, we've run into an issue with our C plugin right out of the gate and I've included a minimal reproducer as an example which is basically a "Hello World" type of test (job_submit_disallow_salloc.c, see attached). *Expectation* What we expect to happen is a sort of hello-world result with a message being written to a /tmp/min_repo.log but that does not occur. It seems that the plugin does not get run at all when jobs are submitted. Jobs still run as expected but the plugin seems to be ignored. *Steps* We compile gcc -fPIC -DHAVE_CONFIG_H -I /modules/source/slurm-23.02.4 -g -O2 -pthread -fno-gcse -Werror -Wall -g -O0 -fno-strict-aliasing -MT job_submit_disallow_salloc.lo -MD -MP -MF .deps/job_submit_disallow_salloc.Tpo -c job_submit_disallow_salloc.c -o .libs/job_submit_disallow_salloc.o mv .deps/job_submit_disallow_salloc.Tpo .deps/job_submit_disallow_salloc.Plo and link gcc -shared -fPIC -DPIC .libs/job_submit_disallow_salloc.o -O2 -pthread -O0 -pthread -Wl,-soname -Wl,job_submit_disallow_salloc.so -o job_submit_disallow_salloc.so Check links after copying to /usr/lib64/slurm: ldd /usr/lib64/slurm/job_submit_disallow_salloc.so linux-vdso.so.1 (0x00007ffe467aa000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1c02095000) libc.so.6 => /lib64/libc.so.6 (0x00007f1c01cd0000) /lib64/ld-linux-x86-64.so.2 (0x00007f1c024b7000) Can someone point out what we are doing incorrectly or how we might troubleshoot this issue? Kindest regards, Glen *Reproducer* The minimal reproducer is basically a "hello world" for C extensions which I've pasted below (I've also attached it for convenience): #include <slurm/slurm.h> #include <slurm/slurm_errno.h> #include <stdio.h> #include "src/slurmctld/slurmctld.h" const char plugin_name[] = "Min Reproducer"; const char plugin_type[] = "job_submit/disallow_salloc"; const uint32_t plugin_version = SLURM_VERSION_NUMBER; extern int job_submit(job_desc_msg_t *job_desc, uint32_t submit_uid, char **err_msg) { FILE *fp; fp = fopen("/tmp/min_repo.log", "w"); fprintf(fp,"Hello!"); fclose(fp); return SLURM_SUCCESS; } int job_modify(job_desc_msg_t *job_desc, job_record_t *job_ptr, uint32_t submit_uid, char **err_msg) { return SLURM_SUCCESS; }

3 2

scheduling according time requirements
by Dietmar Rieder 30 Apr '24

30 Apr '24

Hi, is it possible to have slurm scheduling jobs automatical according to the "-t" time requirements to a fitting partition? e.g. 3 partitions PartitionName=standard Nodes=c-[01-10] Default=YES MaxTime=04:00:00 DefaultTime=00:10:00 State=UP OverSubscribe=NO PartitionName=medium Nodes=c-[04-08] Default=NO MaxTime=24:00:00 DefaultTime=04:00:00 State=UP OverSubscribe=NO PartitionName=long Nodes=c-[09-10] Default=NO MaxTime=336:00:00 DefaultTime=24:00:00 State=UP OverSubscribe=NO So in the standard partition which is the default we have all nodes and a max time of 4h, in the medium partition we have 4 nodes with a max time of 24h and in the long partition we have 2 nodes with a max time of 336h. I was hoping that if I submit a job with -t 01:00:00 it can be run on any node (standard partition), whereas when specifying -t 05:00:00 or -t 48:00:00 the job will run on the nodes of the medium or long partition respectively. However, my job will not get scheduled at all when -t is greater than 01:00:00 i.e. ]$ srun --cpus-per-task 1 -t 01:00:01 --pty bash srun: Requested partition configuration not available now srun: job 42095 queued and waiting for resources it will wait forever because the standard partition is selected, I was thinking that slurm would automatically switch to the medium partition. Do I misunderstand something there? Or can this be somehow configured. Thanks so much and sorry for the naive question Dietmar

4 10

Trying to Track Down root Usage
by Jason Simms 29 Apr '24

29 Apr '24

Hello all, Each week, I generate an automated report of the top users by CPU hours. This week, for whatever reason the user root accounted for a massive number of hours: -------------------------------------------------------- Login Proper Name Used Account --------------- -------------------- --------- --------- root root 28224 root I've tried to generate a report on what might account for this, but so far I'm stumped. For example, when I run a typical report: sacct -o jobid,jobname,start,end,NNodes,NCPUS,ReqMem,CPUTime,AveRSS,MaxRSS -S 2024-04-22 -E 2024-04-29 --user=root --units=G Nothing comes up (same if I change --user=root to --account=root). I'd appreciate any thoughts as to how to investigate or understand this. It may well be that I'm missing something fundamental. Warmest regards, Jason -- *Jason L. Simms, Ph.D., M.P.H.* Manager of Research Computing Swarthmore College Information Technology Services (610) 328-8102 Schedule a meeting: https://calendly.com/jlsimms

2 2

HPC Principal System Engineer at the Broad
by Paul Edmon 25 Apr '24

25 Apr '24

A friend ask me to pass this along. Figured some folks on this list might be interested. https://broadinstitute.avature.net/en_US/careers/JobDetail/HPC-Principal-Sy… -Paul Edmon-

1 0

Apply an specific QoS to all users that belongs to an specific account
by Gestió Servidors 23 Apr '24

23 Apr '24

Hi, I would like to know if it is possible to apply an specific QoS to all users that belongs to an specific account. For example, I have created some new users "user_XX" and, also, I have created their new accounts in SLURM with "sacctmgr create account name=Test" and "sacctmgr create user name=user_XX DefaultAccount=Test". After it, I have changed default QoS "normal" to a new QoS "minimal" (with some limits) to account "Test" (sacctmgr modify account where name=Test set qos=minimal), but what I have seen is that users "user_XX" that belongs to "Test" account continue in QoS "normal" (QoS by default), so it seems that users have not inherited QoS applied to their account. Is there any way to do this and am I doing something wrong? Thanks. -- [cid:image003.jpg@01DA9569.AE233AA0]<http://www.uab.cat/> Daniel Ruiz Molina Tècnic Mitjà Informàtic Arquitectura de Computadors i Sistemes Operatius Escola d'Enginyeria Edifici Q - Despatx QC/3052 - Carrer de les Sitges Campus de la UAB · 08193 Bellaterra (Cerdanyola del Vallès) · Barcelona · Spain +34 93 581 35 44 www.uab.cat<http://www.uab.cat/> Daniel Ruiz at UAB<https://tinyurl.com/yd95zb8j> [cid:image004.jpg@01DA9569.AE233AA0]<www.linkedin.com/in/daniel-ruiz-molina-50a83b27> Aquest missatge s'adreça exclusivament a la persona destinatària i pot contenir informació privada o confidencial. Si l'heu rebut per error, comuniqueu-nos-ho i destruïu-lo, i tingueu present que no teniu autorització per fer-ne cap ús. Abans d'imprimir aquest missatge penseu en el medi ambient.

1 0