- slurm-users - lists.schedmd.com

Cloud elastic help
by mark.w.moorcroft＠ama-inc.com 29 Jan '25

29 Jan '25

I have a new Slurm setup in AWS gov cloud that is not quite working. I will list a few factoids and maybe someone can suggest where to look next. The Troubleshooting page really has nothing relevant for elastic cloud deployments. The nodes are getting set to DOWN+CLOUD+POWERED_DOWN. Running a job does not launch a node in this state. I can force the nodes to launch with scontrol POWER_UP. The jobs will claim to run, then re-queue, but never complete. When the nodes boot I see them appear in slurmctl, but it soon reports connection lost. The node slurmd claims to be healthy, but the controller eventually just terminates them. You can ping in both directions with the hostnames. I've done 4 clusters. The first was torque/maui and the rest were slurm, but all were bare metal. This is my first attempt at cloud. We have ITAR data, so I can't use Amazon Parallel Computing because it is not offered in GovCloud. https://cluster-in-the-cloud.readthedocs.io/en/latest/running.html I had to fork this project because so much is obsolete, but it's mostly working for me now. https://github.com/mntbighker [root@mgmt ~]# journalctl -fu slurmctld Jan 29 19:16:15 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: 1 jobs to backfill Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: Updating partition uid access list Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: purge_old_job: job file deletion is falling behind, 1 left to remove Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched: Running job scheduler for full queue. Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: beginning Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: 1 jobs to backfill Jan 29 19:16:50 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sackd_mgr_dump_state: saved state of 0 nodes Jan 29 19:17:15 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: beginning Jan 29 19:17:15 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: 1 jobs to backfill Jan 29 19:17:26 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: POWER: Power save mode: 4 nodes [root@mgmt ~]# scontrol show node many-antelope-c5n-2xlarge-0001 NodeName=many-antelope-c5n-2xlarge-0001 CoresPerSocket=4 CPUAlloc=0 CPUEfctv=8 CPUTot=8 CPULoad=0.00 AvailableFeatures=shape=c5n.2xlarge,ad=None,arch=x86_64 ActiveFeatures=shape=c5n.2xlarge,ad=None,arch=x86_64 Gres=(null) NodeAddr=many-antelope-c5n-2xlarge-0001 NodeHostName=many-antelope-c5n-2xlarge-0001 RealMemory=20034 AllocMem=0 FreeMem=N/A Sockets=1 Boards=1 State=DOWN+CLOUD+POWERED_DOWN ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=production,debug,batch,long BootTime=None SlurmdStartTime=None LastBusyTime=Unknown ResumeAfterTime=None CfgTRES=cpu=8,mem=20034M,billing=8 AllocTRES= CurrentWatts=0 AveWatts=0 Reason=Not responding [slurm@2025-01-29T17:49:26] [root@mgmt ~]# scontrol show jobs JobId=30 JobName=test.sl UserId=mwmoorcroft(1106) GroupId=nssam(1101) MCS_label=N/A Priority=1 Nice=0 Account=nssam QOS=(null) JobState=PENDING Reason=Nodes_required_for_job_are_DOWN,_DRAINED_or_reserved_for_jobs_in_higher_priority_partitions Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=01:00:00 TimeMin=N/A SubmitTime=2025-01-29T18:58:26 EligibleTime=2025-01-29T18:58:26 AccrueTime=2025-01-29T18:58:26 StartTime=Unknown EndTime=Unknown Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-01-29T19:19:15 Scheduler=Backfill:* Partition=production AllocNode:Sid=ip-172-16-2-14.us-gov-east-1.compute.internal:7090 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1-1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=20034M,node=1,billing=1 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=1:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=/mnt/shared/home/mwmoorcroft/test.sl WorkDir=/mnt/shared/home/mwmoorcroft StdErr=/mnt/shared/home/mwmoorcroft/slurm-30.out StdIn=/dev/null StdOut=/mnt/shared/home/mwmoorcroft/slurm-30.out

2 2

Adding a permanent node Feature to slurm in AWS parallelcluster
by luigi＠scorzato.it 28 Jan '25

28 Jan '25

I need to add a node Feature to my SLURM AWS parallel-cluster I understand from here: https://groups.google.com/g/slurm-users/c/LgL6hq3XxzE that to add a node feature permanently I have to edit slurm.conf, because the effects of the command: scontrol update NodeName=[node(s)] AvailableFeatures=[comma separated list of features] will not persist after restart (besides, I could not find how to ADD a feature without deleting the existing AvailableFeatures). However, AWS parallelcluster slurm.conf already includes files that contain lines like: NodeName=nn77[1-100] CPUs=2 RealMemory=186777 State=CLOUD Feature=dynamic,g4dn.large,gpu Weight=1000 Gres=gpu:t4:1 which is automatically generated and should not be edited. What is the recommended way to add a feature without breaking the parallelcluster set up? My I put somewhere something like NodeName=nn77[1-100] Feature+=cascadelake ? thank you!

1 0

Automate job lmod environment setup based on node
by Daniel UB 27 Jan '25

27 Jan '25

Greetings ! We're using Slurm 23.11.8 in a small cluster . The control node shares via NFS the directory /clusterprograms with the compute nodes (using the same name as mountpoint) to provide access to available software. The users are instructed to use lmod<https://lmod.readthedocs.io/en/latest/060_locating.html> (module load/avail/purge) to setup their job's runtime environment. We have the software and modules at /clusterprograms distributed like this: /clusterprograms/ ├── common ├── gpu/ │ ├── cuda8/ │ └── cuda10/ └── cpu/ ├── intel/ └── amd/ We configured the /etc/profile.d/ directory on each node so ,when we access them directly (via ssh) the paths used by module (MODULEPATH) are only these: * /home/<user>/modulefiles * /clusterprograms/common/modules * /clusterprograms/<processor vendor or cuda version>/modules The exception is the control/login node, which only has /home/<user>/modulefiles and /clusterprograms/common/modules The issue we have now is this: We are looking for a safe and practical way to automate the reconfiguration of MODULEPATH<https://lmod.readthedocs.io/en/latest/077_ref_counting.html> in the compute nodes when a job is submitted. For now, the only method that is "working" is the following TaskProlog script, but it forces the users to load the modules inside the scripts that may be called by srun steps, rather than being able to load them in the job base step: #!/bin/bash if [[ -n "${SLURM_JOB_CONSTRAINTS}" ]]; then echo "export ORIGINAL_MPATH=${MODULEPATH}" IFS=',' read -ra JOB_ARCH_CONSTRAINTS <<< "$SLURM_JOB_CONSTRAINTS" for constraint in "${JOB_ARCH_CONSTRAINTS[@]}"; do case "${constraint}" in cuda*) echo "export MODULEPATH=/clusterprograms/gpu/${constraint}:${MODULEPATH}" ;; intel-*) echo "export MODULEPATH=/clusterprograms/cpu/intel/${constraint}:${MODULEPATH}" ;; amd-*) echo "export MODULEPATH=/clusterprograms/cpu/amd/${constraint}:${MODULEPATH}" ;; esac done fi Is there any better way to do this ? How is that it doesn't work if we use Prolog ? I tried using the --export option but it doesn't work, since the control node has MODULEPATH set via the main shell profiles. Many thanks !!! -- Daniel Garrapucho Lévy Técnic informàtic Departament de Física de la Matèria Condensada Facultat de Física Martí i Franquès, 1 08028 Barcelona, SPAIN Despatx V302 Email: daniel.garrapucho(a)ub.edu<mailto:daniel.garrapucho@ub.edu> [https://estatics.web.ub.edu/image/company_logo?img_id=2946262&t=17001439433…] Aquest missatge, i els fitxers adjunts que hi pugui haver, pot contenir informació confidencial o protegida legalment i s’adreça exclusivament a la persona o entitat destinatària. Si no consteu com a destinatari final o no teniu l’encàrrec de rebre’l, no esteu autoritzat a llegir-lo, retenir-lo, modificar-lo, distribuir-lo, copiar-lo ni a revelar-ne el contingut. Si l’heu rebut per error, informeu-ne el remitent i elimineu del sistema tant el missatge com els fitxers adjunts que hi pugui haver. Este mensaje, y los ficheros adjuntos que pueda incluir, puede contener información confidencial o legalmente protegida y está exclusivamente dirigido a la persona o entidad destinataria. Si usted no consta como destinatario final ni es la persona encargada de recibirlo, no está autorizado a leerlo, retenerlo, modificarlo, distribuirlo o copiarlo, ni a revelar su contenido. Si lo ha recibido por error, informe de ello al remitente y elimine del sistema tanto el mensaje como los ficheros adjuntos que pueda contener. This email message and any attachments it carries may contain confidential or legally protected material and are intended solely for the individual or organization to whom they are addressed. If you are not the intended recipient of this message or the person responsible for processing it, then you are not authorized to read, save, modify, send, copy or disclose any part of it. If you have received the message by mistake, please inform the sender of this and eliminate the message and any attachments it carries from your account.

1 0

Problems with slurmdbd on Ubuntu 24.04
by karl＠lovink.net 27 Jan '25

27 Jan '25

Hello, Last weekend, I updated my server to ubuntu 24.04 and instead of using my own compiled version of slurm, I switched to the version (23.11.4) that comes with Ubuntu 24.04. Since then, I have been having problems with slurmdbd. I get the following error message: slurmdbd: error: issue making get_lineage procedure slurmdbd: accounting_storage/as_mysql: init: Accounting storage MYSQL plugin failed slurmdbd: debug2: wsrep_on=0 slurmdbd: error: Couldn't load specified plugin name for accounting_storage/mysql: Plugin init() callback failed slurmdbd: error: cannot create accounting_storage context for accounting_storage/mysql slurmdbd: fatal: Unable to initialize accounting_storage/mysql accounting storage plugin My /etc/slurm/slurmdbd.conf looks like this: AuthType=auth/munge DebugLevel=debug5 DbdHost=localhost DbdPort=3306 StorageLoc=slurm_acct_db StoragePass=<password> StorageType=accounting_storage/mysql StorageUser=slurm SlurmUser=admina PluginDir=/usr/lib/x86_64-linux-gnu/slurm-wlm ArchiveEvents=yes ArchiveJobs=yes ArchiveResvs=yes ArchiveSteps=no ArchiveSuspend=no ArchiveTXN=no ArchiveUsage=no AuthInfo=/var/run/munge/munge.socket.2 PurgeEventAfter=1month PurgeJobAfter=12month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month PurgeTXNAfter=12month PurgeUsageAfter=24month LogFile=/var/log/slurm/slurmdbd.log PidFile=/var/run/slurmdbd.pid NOTE: Removing the PluginDir directive makes no difference. And the Plugin directory contains the accounting/MySQL file: ls -al /usr/lib/x86_64-linux-gnu/slurm-wlm/*mysql* -rw-r--r-- 1 root root 3145992 Apr 1 2024 /usr/lib/x86_64-linux-gnu/slurm-wlm/accounting_storage_mysql.a -rw-r--r-- 1 root root 1071 Apr 1 2024 /usr/lib/x86_64-linux-gnu/slurm-wlm/accounting_storage_mysql.la -rw-r--r-- 1 root root 501848 Apr 1 2024 /usr/lib/x86_64-linux-gnu/slurm-wlm/accounting_storage_mysql.so I checked already the threat on: https://groups.google.com/g/slurm-users/c/wRV8o7qqGi4 Any idea what is going on here? Regards, Karl

3 4

Re: Problems with slurmdbd on Ubuntu 24.04
by karl＠lovink.net 27 Jan '25

27 Jan '25

Hi Stpehan, It is there: root@vrfcanalyzer:/usr/lib/x86_64-linux-gnu/slurm-wlm# ls -al *mysql* -rw-r--r-- 1 root root 3145992 Apr 1 2024 accounting_storage_mysql.a -rw-r--r-- 1 root root 1071 Apr 1 2024 accounting_storage_mysql.la -rw-r--r-- 1 root root 501848 Apr 1 2024 accounting_storage_mysql.so -rw-r--r-- 1 root root 213700 Apr 1 2024 jobcomp_mysql.a -rw-r--r-- 1 root root 994 Apr 1 2024 jobcomp_mysql.la -rw-r--r-- 1 root root 44240 Apr 1 2024 jobcomp_mysql.so dpkg -S jobcomp_mysql.so slurm-wlm-mysql-plugin: /usr/lib/x86_64-linux-gnu/slurm-wlm/jobcomp_mysql.so root@vrfcanalyzer:/usr/lib/x86_64-linux-gnu/slurm-wlm# -----Original Message----- From: Stephan Roth <stephan.roth(a)isg.ee.ethz.ch> Sent: Monday, 27 January 2025 13:23 To: karl(a)lovink.net; slurm-users(a)lists.schedmd.com Subject: Re: [slurm-users] Problems with slurmdbd on Ubuntu 24.04 Hello Karl, I quickly downloaded slurm-wlm-mysql-plugin_23.11.4-1.2ubuntu5_amd64.deb, unpacked it and saw these contents:ä usr/ ├── lib │ └── x86_64-linux-gnu │ └── slurm-wlm │ ├── accounting_storage_mysql.so │ └── jobcomp_mysql.so └── share ├── doc │ └── slurm-wlm-mysql-plugin │ ├── changelog.Debian.gz -> ../slurm-wlm-basic-plugins/changelog.Debian.gz │ └── copyright └── lintian └── overrides └── slurm-wlm-mysql-plugin So it seems to me you're somehow missing /usr/lib/x86_64-linux-gnu/slurm-wlm/jobcomp_mysql.so Does that help? Best, Stephan On 27.01.25 12:00, karl--- via slurm-users wrote: > Hello, > > Last weekend, I updated my server to ubuntu 24.04 and instead of using > my own compiled version of slurm, I switched to > > the version (23.11.4) that comes with Ubuntu 24.04. Since then, I have > been having problems with slurmdbd. > > I get the following error message: > > slurmdbd: error: issue making get_lineage procedure > > slurmdbd: accounting_storage/as_mysql: init: Accounting storage MYSQL > plugin failed > > slurmdbd: debug2: wsrep_on=0 > > slurmdbd: error: Couldn't load specified plugin name for > accounting_storage/mysql: Plugin init() callback failed > > slurmdbd: error: cannot create accounting_storage context for > accounting_storage/mysql > > slurmdbd: fatal: Unable to initialize accounting_storage/mysql > accounting storage plugin > > My /etc/slurm/slurmdbd.conf looks like this: > > AuthType=auth/munge > > DebugLevel=debug5 > > DbdHost=localhost > > DbdPort=3306 > > StorageLoc=slurm_acct_db > > StoragePass=<password> > > StorageType=accounting_storage/mysql > > StorageUser=slurm > > SlurmUser=admina > > PluginDir=/usr/lib/x86_64-linux-gnu/slurm-wlm > > ArchiveEvents=yes > > ArchiveJobs=yes > > ArchiveResvs=yes > > ArchiveSteps=no > > ArchiveSuspend=no > > ArchiveTXN=no > > ArchiveUsage=no > > AuthInfo=/var/run/munge/munge.socket.2 > > PurgeEventAfter=1month > > PurgeJobAfter=12month > > PurgeResvAfter=1month > > PurgeStepAfter=1month > > PurgeSuspendAfter=1month > > PurgeTXNAfter=12month > > PurgeUsageAfter=24month > > LogFile=/var/log/slurm/slurmdbd.log > > PidFile=/var/run/slurmdbd.pid > > NOTE: Removing the PluginDir directive makes no difference. > > And the Plugin directory contains the accounting/MySQL file: > > ls -al /usr/lib/x86_64-linux-gnu/slurm-wlm/*mysql* > > -rw-r--r-- 1 root root 3145992 Apr 1 2024 /usr/lib/x86_64-linux-gnu/ > slurm-wlm/accounting_storage_mysql.a > > -rw-r--r-- 1 root root 1071 Apr 1 2024 /usr/lib/x86_64-linux-gnu/slurm- > wlm/accounting_storage_mysql.la > > -rw-r--r-- 1 root root 501848 Apr 1 2024 /usr/lib/x86_64-linux-gnu/ > slurm-wlm/accounting_storage_mysql.so > > I checked already the threat on: https://groups.google.com/g/slurm- > users/c/wRV8o7qqGi4 <https://groups.google.com/g/slurm-users/c/wRV8o7qqGi4> > > Any idea what is going on here? > > Regards, > > Karl > > > -- ETH Zurich Stephan Roth Systems Administrator IT Services Group (ISG) D-ITET ETF D 104 Sternwartstrasse 7 8092 Zurich Phone +41 44 632 30 59 stephan.roth(a)isg.ee.ethz.ch www.isg.ee.ethz.ch Working days: Mon, Tue, Thu, Fri

1 0

Slurm version 24.11.1 is now available
by Marshall Garey 24 Jan '25

24 Jan '25

We are pleased to announce the availability of Slurm version 24.11.1. This fixes a few possible crashes of the slurmctld and slurmrestd; a regression in 24.11 which caused file transfers to a job with sbcast to not join the job container namespace; mpi apps using Intel OPA, PSM2 and OMPI 5.x when ran through srun; and various minor to moderate bugs. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.11.1 > ========================== > -- With client commands MIN_MEMORY will show mem_per_tres if specified. > -- Fix errno message about bad constraint > -- slurmctld - Fix crash and possible split brain issue if the > backup controller handles an scontrol reconfigure while in control > before the primary resumes operation. > -- Fix stepmgr not getting dynamic node addrs from the controller > -- stepmgr - avoid "Unexpected missing socket" errors. > -- Fix `scontrol show steps` with dynamic stepmgr > -- Deny jobs using the "R:" option of --signal if PreemptMode=OFF > globally. > -- Force jobs using the "R:" option of --signal to be preemptable > by requeue or cancel only. If PreemptMode on the partition or QOS is off > or suspend, the job will default to using PreemptMode=cancel. > -- If --mem-per-cpu exceeds MaxMemPerCPU, the number of cpus per > task will always be increased even if --cpus-per-task was specified. This > is needed to ensure each task gets the expected amount of memory. > -- Fix compilation issue on OpenSUSE Leap 15 > -- Fix jobs using more nodes than needed when not using -N > -- Fix issue with allocation being allocated less resources > than needed when using --gres-flags=enforce-binding. > -- select/cons_tres - Fix errors with MaxCpusPerSocket partition > limit. Used cpus/cores weren't counted properly, nor limiting free ones > to avail, when the socket was partially allocated, or the job request > went beyond this limit. > -- Fix issue when jobs were preempted for licenses even if there > were enough licenses available. > -- Fix srun ntasks calculation inside an allocation when nodes are > requested using a min-max range. > -- Print correct number of digits for TmpDisk in sdiag. > -- Fix a regression in 24.11 which caused file transfers to a job > with sbcast to not join the job container namespace. > -- data_parser/v0.0.40 - Prevent a segfault in the slurmrestd when > dumping data with v0.0.40+complex data parser. > -- Remove logic to force lowercase GRES names. > -- data_parser/v0.0.42 - Prevent the association id from always > being dumped as NULL when parsing in complex mode. Instead it will now > dump the id. This affects the following endpoints: > GET slurmdb/v0.0.42/association > GET slurmdb/v0.0.42/associations > GET slurmdb/v0.0.42/config > -- Fixed a job requeuing issue that merged job entries into the > same SLUID when all nodes in a job failed simultaneously. > -- When a job completes, try to give idle nodes to reservations with > the REPLACE flag before allowing them to be allocated to jobs. > -- Avoid expensive lookup of all associations when dumping or > parsing for v0.0.42 endpoints. > -- Avoid expensive lookup of all associations when dumping or > parsing for v0.0.41 endpoints. > -- Avoid expensive lookup of all associations when dumping or > parsing for v0.0.40 endpoints. > -- Fix segfault when testing jobs against nodes with invalid gres. > -- Fix performance regression while packing larger RPCs. > -- Document the new mcs/label plugin. > -- job_container/tmpfs - Fix Xauthoirty file being created > outside the container when EntireStepInNS is enabled. > -- job_container/tmpfs - Fix spank_task_post_fork not always > running in the container when EntireStepInNS is enabled. > -- Fix a job potentially getting stuck in CG on permissions > errors while setting up X11 forwarding. > -- Fix error on X11 shutdown if Xauthority file was not created. > -- slurmctld - Fix memory or fd leak if an RPC is recieved that > is not registered for processing. > -- Inject OMPI_MCA_orte_precondition_transports when using PMIx. This fixes > mpi apps using Intel OPA, PSM2 and OMPI 5.x when ran through srun. > -- Don't skip the first partition_job_depth jobs per partition. > -- Fix gres allocation issue after controller restart. > -- Fix issue where jobs requesting cpus-per-gpu hang in queue. > -- switch/hpe_slingshot - Treat HTTP status forbidden the same as > unauthorized, allowing for a graceful retry attempt.

3 2

Slurmd cannot find cgroup plugin
by Webb, Jordan 20 Jan '25

20 Jan '25

Hi all, When trying to start slurmd, it is failing with cgroup issues. Any suggestions on where to troubleshoot this issue? x8000c0s0b0n0:~ # slurmd -V slurm 24.11.0 x8000c0s0b0n0:~ # slurmd -D -vvv slurmd: debug: Log file re-opened slurmd: debug2: hwloc_topology_init slurmd: debug2: hwloc_topology_load slurmd: debug2: hwloc_topology_export_xml slurmd: debug: CPUs:288 Boards:1 Sockets:4 CoresPerSocket:72 ThreadsPerCore:1 slurmd: error: Couldn't find the specified plugin name for cgroup/v2 looking at all files slurmd: error: cannot find cgroup plugin for cgroup/v2 slurmd: error: cannot create cgroup context for cgroup/v2 slurmd: error: Unable to initialize cgroup plugin slurmd: error: slurmd initialization failed x8000c0s0b0n0:~ # mount | grep cgroup cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) x8000c0s0b0n0:~ # grep cgroup /etc/slurm/slurm.conf ProctrackType=proctrack/cgroup TaskPlugin=task/affinity,task/cgroup Thanks! Jordan

1 1

Re: Slurm "showpartitions" tool has been updated
by Ole Holm Nielsen 17 Jan '25

17 Jan '25

Hi Robert, What's your version of Slurm? We run 24.05.5. The showpartitions tools compacts the list of nodenames using "scontrol show hostlistsorted <nodelist>". Can you please check your scontrol manual page if "hostlistsorted" is documented? Does the command work without -N? Thanks, Ole On 17-01-2025 19:54, Robert Kudyba wrote: > Hello Ole, > > If you'd like me to post this on the Git page LMK. Any idea why this > error invalid encode argument with -N? > ./showpartitions -N > Print partitions nodelists > Partition statistics for cluster axon at Fri Jan 17 13:51:43 EST 2025 > Partition #Nodes #CPU_cores Cores_pending Job_Nodes > MaxJobTime Cores Mem/Node NODELIST > Name State Total Idle Total Idle Resorc Other Min Max Day- > hr:mn /node (GB) > nklab up 4 0 352 125 0 0 1 infin > 10-00:00 48 191+invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > ctn up 3 1 168 152 0 0 1 infin > 10-00:00 56 128 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > issa up 5 1 384 128 0 0 1 infin > 10-00:00 64 191+invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > naplab up 3 1 320 231 0 0 1 infin > 10-00:00 64 515+invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > lkumar up 3 2 192 191 0 0 1 infin > 10-00:00 64 515 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > u19moc3 up 1 1 96 96 0 0 1 infin > 10-00:00 96 257 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > abbott up 3 2 192 187 0 0 1 infin > 10-00:00 64 515 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > miller up 3 2 192 190 0 0 1 infin > 10-00:00 64 515 invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > burst up 25 10 1896 1300 0 0 1 infin > 5-00:00 48 128+invalid encode argument > scontrol [<OPTION>] [<COMMAND>] > zrc up 6 1 376 190 0 0 1 infin > 10-00:00 48 128+invalid encode argument > > On Fri, Jan 17, 2025 at 7:31 AM Ole Holm Nielsen via slurm-users <slurm- > users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com>> wrote: > > Dear Slurm users, > > The "showpartitions" tool prints a Slurm cluster partition status > overview > with 1 line per partition. Color coding helps locating under- or > over-utilized partitions. > > The Slurm "showpartitions" tool has now been updated with a new "-N" > flag > which prints the nodename list for each partition. > > You can download this tool from > https://urldefense.proofpoint.com/v2/url? > u=https-3A__github.com_OleHolmNielsen_Slurm-5Ftools_tree_master_partitions&d=DwICAg&c=aqMfXOEvEJQh2iQMCb7Wy8l0sPnURkcqADc2guUW8IM&r=X0jL9y0sL4r4iU_qVtR3lLNo4tOL1ry_m7-psV3GejY&m=SvjV7A7QvPs0H4AmOiKvni-pI3DeHZmNgrCPf9ije3ftTN6STgomGCGHyuspdUFz&s=e2VRHqxuaDlLYggVv8eW9tkQqfXU332ucgmnqiU3Toc&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_OleHolmNiel…> > > If you have any issues or suggestions, please write to > Ole.H.Nielsen(a)fysik.dtu.dk <mailto:Ole.H.Nielsen@fysik.dtu.dk> or > open an issue on GitHub. > > Best regards, > Ole > > -- > Ole Holm Nielsen > PhD, Senior HPC Officer > Department of Physics, Technical University of Denmark > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > <mailto:slurm-users@lists.schedmd.com> > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > <mailto:slurm-users-leave@lists.schedmd.com> >

2 1

Slurm "showpartitions" tool has been updated
by Ole Holm Nielsen 17 Jan '25

17 Jan '25

Dear Slurm users, The "showpartitions" tool prints a Slurm cluster partition status overview with 1 line per partition. Color coding helps locating under- or over-utilized partitions. The Slurm "showpartitions" tool has now been updated with a new "-N" flag which prints the nodename list for each partition. You can download this tool from https://github.com/OleHolmNielsen/Slurm_tools/tree/master/partitions If you have any issues or suggestions, please write to Ole.H.Nielsen(a)fysik.dtu.dk or open an issue on GitHub. Best regards, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

1 0

job_container/tmpfs and srun.
by Phill Harvey-Smith 17 Jan '25

17 Jan '25

Hi all, On our setup we are using job_container/tmpfs to give each job it's own temp space. Since our compute nodes have reasonably sized disks for tasks that do a lot of disk I/O on user's data we have asked users to copy their data to the local disk at the beginning of the task and (if needed) copy it back at the end. This saves lots of NFS thrashing slowing down both the task and the NFS servers. However some of our users are having problems with this, their initial sbatch script will create a temp directory in their private /tmp copy their data to it and then try to srun a program. The srun will fall over as it doesn't seem to have have access to the copied data. I suspect this is because the srun task is getting it's own private /tmp. So my question is, is there a way to have the srun task inherit the /tmp of the initial sbatch? I'll include a sample of the script our user is using below. If any further information is required please feel free to ask. Cheers. Phill. #!/usr/bin/bash #SBATCH --nodes 1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --time=00:00:10 #SBATCH --mem-per-cpu=3999 #SBATCH --output=script_out.log #SBATCH --error=script_error.log # The above options puts the STDOUT and STDERR of sbatch in # log files prefixed with 'script_'. # Create a randomly-named directory under /tmp jobtmpdir=$(mktemp -d) # Register a function to try and cleanup in case of job failure cleanup_handler() { echo "Cleaning up ${jobtmpdir}" rm -rf ${jobtmpdir} } trap 'cleanup_handler' SIGTERM EXIT # Change working directory to this directory cd ${jobtmpdir} # Copy the executable and input files from # where the job was submitted to the temporary directory. cp ${SLURM_SUBMIT_DIR}/a.out . cp ${SLURM_SUBMIT_DIR}/input.txt . # Run the executable, handling the collection of stdout # and stderr ourselves by redirecting to file srun ./a.out 2> task_error.log > task_out.log # Copy output data back to the submit directory. cp output.txt ${SLURM_SUBMIT_DIR} cp task_out.log ${SLURM_SUBMIT_DIR} cp task_error.log ${SLURM_SUBMIT_DIR} # Cleanup cd ${SLURM_SUBMIT_DIR} cleanup_handler

2 1

2025

2024

slurm-users ----- 2025 ----- May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users