- slurm-users - lists.schedmd.com

Tool for profiling resource usage by slurm jobs
by Nicolas Granger 08 Jan '24

08 Jan '24

Hi all, Happy new year everyone! I've been looking for a simple tool that reports how much resources are actually consumed by a job to help my colleagues and I adjust job requirements. I could not find such a tool, or the ones mentioned on this ML were not easy to install and use, so I have written a new one: https://github.com/CEA-LIST/sprofile It's a simple python script which parses cgroup and nvml data from the nvidia driver. It reports duration, cpu load, peak RAM, GPU load and peak GPU memory like so: |-- sprofile report (node03) -- Time: 0:00:03 / 1:00:00 CPU load: 2.0 / 4.0 RAM peak mem: 7G / 8G GPU load: 0.2 / 2.0 GPU peak mem: 7G / 40G| The requirements are to use the slurm cgroup plugin and to enable accounting on the GPU (nvidia-smi --accounting-mode=1). I hope you find this useful and let me know I you find bugs or want to contribute. Regards, Nicolas Granger

1 0

A fairshare policy that spans multiple clusters
by David Baker 06 Jan '24

06 Jan '24

Hello, We are soon to install new Slurm cluster at our site. That means that we will have a total of three clusters running Slurm. Only two, that is the new clusters, will share a common file system. The original cluster has its own file system is independent of the new arrivals. If possible, we would like to try to prevent users from making significant user of all the clusters and get a 'triple whammy'. In other words, is there any way to share the fairshare information between the clusters so that a user's use of one of the clusters impacts their usage on the other clusters – if that makes sense. Does anyone have any thoughts on this question, please? Am I correct in thinking that federating clusters is related to my question? Do I gather correctly, however, that federation only works if there is a common database on a shared file system? Best regards, David

2 1

slurmdb endpoints for slurmrestd
by Jackson, Gary L. 06 Jan '24

06 Jan '24

I get an HTTP 404 when I try to GET /slurmdb/v0.0.39/clusters or any other /slurmdb endpoint. I get this against multiple versions of Slurm, including 23.11.1. Using GET against /slurm/v0.0.39/ping works just fine. Is there something I need to do to turn slurmdb endpoints on? -- Gary

1 0

Slurp for sw builds
by Duane Ellis 05 Jan '24

05 Jan '24

In my case I would like to use a slurm cluster for a sw ci/cd like solution for building sw images My current scripted full system build takes 3-5 hours and is done serially we could easily find places where we can choose to build things in parallel hence the idea is to spawn parallel builds on the Linux slurm cluster example: we have a list of images we iterate over in a for loop to build each thing the steps are: cd somedir then type make or run a shell script in that directory The last step after the for loop would be wait for all of the child builds to complete Once all child jobs are done we have a single job that combines or packages all the intermediate images Really want to use slurm because our FPGA team will have a giant slurm linux cluster for Xilinix FPGA builds and those nodes can do what we need for sw purposes (reusing the existing cluster is a huge win for us) My question is this: Can somebody point me to some sw build examples for or using slurm? All I can seem to find is how to install I see the srun and sbatch command man pages but no good examples Bonus would be something that integrates into a gitlab runner example or Jenkins in some way All I can seem to find is how to install and administer slurm not a how to use slurm Sent from my iPhone

3 2

Fwd: Fairshare: users not added
by Alex Ninaber 04 Jan '24

04 Jan '24

Hi all, A problem on slurm-23.02.4-1, 10.6.16-MariaDB; Maria and Slurmctld in active/active, SlurmDB in active/off, shared IP. Shared spool via Gluster. DB is an upgraded version of Slurm from somewhere 2017 (upgraded various times). The question is whether we should give up and start from scratch or if there's an easy fix. Problem: whenever we add a new user and add it to sacctmgr, the user shows up properly in sacct/mgr – but never shows up with the sshare commands after running some jobs. After restarting slurm a couple of times it shows up. Problem seems to be there also in the previous version. Only error we can see in slurmdb log: [2023-12-21T09:43:30.586] error: slurm_persist_conn_open: Something happened with the receiving/processing of the persistent connection init message to 10.141.255.253:6817 : (null) [2023-12-21T09:43:30.586] error: slurmdb_send_accounting_update_persist: Unable to open connection to registered cluster cluster. [2023-12-21T09:43:30.586] error: slurm_receive_msg: No response to persist_init [2023-12-21T09:43:30.586] error: update cluster: No error to cluster at 10.141.255.253(6817) [2023-12-21T09:43:30.586] debug2: DBD_FINI: CLOSE:1 COMMIT:0 [2023-12-21T09:43:30.586] debug4: accounting_storage/as_mysql: acct_storage_p_commit: got 0 commits AccountingStorageType=accounting_storage/slurmdbd # jobaccounting JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/linux SlurmctldTimeout=60 SlurmdTimeout=60 TCPTimeout=60 MessageTimeout=60 Best regards, Alex

1 0

All nodes within one partition reboot unexpectedly
by Jinglei Hu 02 Jan '24

02 Jan '24

Hi all, I had a slurm partition gpu_gmx with the following configuration (Slurm version: 20.11.9): > NodeName=node[09-11] Gres=gpu:rtx4080:1 Sockets=1 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=62000 State=UNKNOWN > NodeName=node[12-14] Gres=gpu:rtx4070ti:1 Sockets=1 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=62000 State=UNKNOWN > PartitionName=gpu_gmx Nodes=node[09-14] Default=NO MaxTime=UNLIMITED State=UP A job running on node11 had a problem, which then triggered the reboot of all nodes (node[09-14]) within the same partition (cat /var/log/slurmctld.log): > [2023-12-26T23:04:23.200] Batch JobId=25061 missing from batch node node11 (not found BatchStartTime after startup), Requeuing job > [2023-12-26T23:04:23.200] _job_complete: JobId=25061 WTERMSIG 126 > [2023-12-26T23:04:23.200] _job_complete: JobId=25061 cancelled by node failure > [2023-12-26T23:04:23.200] _job_complete: requeue JobId=25061 due to node failure > [2023-12-26T23:04:23.200] _job_complete: JobId=25061 done > [2023-12-26T23:04:23.200] validate_node_specs: Node node11 unexpectedly rebooted boot_time=1703603052 last response=1703602983 > [2023-12-26T23:04:23.222] validate_node_specs: Node node09 unexpectedly rebooted boot_time=1703603052 last response=1703602983 > [2023-12-26T23:04:23.579] Batch JobId=25060 missing from batch node node10 (not found BatchStartTime after startup), Requeuing job > [2023-12-26T23:04:23.579] _job_complete: JobId=25060 WTERMSIG 126 > [2023-12-26T23:04:23.579] _job_complete: JobId=25060 cancelled by node failure > [2023-12-26T23:04:23.579] _job_complete: requeue JobId=25060 due to node failure > [2023-12-26T23:04:23.579] _job_complete: JobId=25060 done > [2023-12-26T23:04:23.579] validate_node_specs: Node node10 unexpectedly rebooted boot_time=1703603052 last response=1703602983 > [2023-12-26T23:04:23.581] validate_node_specs: Node node14 unexpectedly rebooted boot_time=1703603051 last response=1703602983 > [2023-12-26T23:04:23.654] validate_node_specs: Node node13 unexpectedly rebooted boot_time=1703603052 last response=1703602983 > [2023-12-26T23:04:24.681] validate_node_specs: Node node12 unexpectedly rebooted boot_time=1703603053 last response=1703602983 > [2023-12-27T04:46:42.461] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=25060 uid 0 > [2023-12-27T04:46:43.822] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=25061 uid 0 The operating systems are CentOS 7.9.2009 on master node, and CentOS 8.5.2111 on node[09-14]. Does anyone have a similar experience and have a clue how to resolve this? Thanks in advance. Best, Jinglei

1 0

2025

2024

slurm-users