- slurm-users - lists.schedmd.com

Re: Change primary alloc node
by Bhaskar Chakraborty 31 Oct '24

31 Oct '24

Hello, Just to add some context here. We plan to use slurm for developing a sched solution which interacts with a backend system. Now, the backend system has pieces of h/w which require specific host in the allocation to be the primary/master host wherein the initial task would be launched, this in turn is driven by the job's placement orientation on the h/w itself. So, our primary task should launch in the asked primary host while secondary / remote tasks would subsequently get started on … [View More]

3 2

Why AllowAccounts not work in slurm-23.11.6
by daijiangkuicgo＠gmail.com 30 Oct '24

30 Oct '24

I have set AllowAccounts=sunlabc5hpc,root, but it doesn’t seem to work. User c010637 is not part of the sunlabc5hpc account but is still able to use the sunlabc5hpc partition. I have tried setting EnforcePartLimits to ALL, ANY, and NO, but none of these options resolved the issue. [c010637@sl-login ~]$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST cpu* up infinite 3 mix sl-c[0035,0042-0043] cpu* up infinite 1 idle sl-c0036 gpu up … [View More]

5 9

Re: 转发: What is the safe upgrade path when upgrade from slurm21.08 and mariadb5.5?
by taleintervenor＠sjtu.edu.cn 30 Oct '24

30 Oct '24

Thanks for all your help. So it seems we can skip the trouble of compiling SLURM over different mariadb versions. Tianyang Zhang SJTU Network Information Center 发件人: Sid Young <sid.young(a)gmail.com> 发送时间: 2024年10月30日 7:19 收件人: Andrej Sec <andrej.sec(a)savba.sk> 抄送: taleintervenor(a)sjtu.edu.cn; slurm-users(a)lists.schedmd.com 主题: Re: [slurm-users] Re: 转发: What is the safe upgrade path when upgrade from slurm21.08 and mariadb5.5? I recently upgraded from 20.11 to 24.… [View More]05.2, before moving the cluster from CentOS 7.9 to Oracle Linux 8.10 The DB upgrade should be pretty simple, do a mysqldump first, then uninstall the old DB, change the repo's and install the new DB version. It should recognise the DB files on disk and access them. Do another DB backup on the new DB version. then roll through the Slurm upgrades. I picked the first and last version of each release, and systematically went through each node till it was done. First the slurm controller node, then the compute nodes. To avoid Job loss, drain the nodes or you end up with a situation where the slurmd can't talk to the running slurmstepd and the job(s) gets lost. (Shows as a "Protocol Error"). Ole sent me a link to this guide which mostly worked. https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/#upgrade-slurm… Sid Young W: https://off-grid-engineering.com On Tue, Oct 29, 2024 at 6:33 PM Andrej Sec via slurm-users <slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> > wrote: Hi, we are facing a similar task. We have a Slurm 22.05 / MariaDB 5.5.68 environment and want to upgrade to a newer version. According to the documentation, it’s recommended to upgrade from 22.05 to a maximum of 23.11 in one step. With the MariaDB upgrade, there’s a challenge between 10.1 and 10.2+ due to incompatible changes (https://mariadb.com/kb/en/changes-improvements-in-mariadb-10-2). This upgrade, as I understand from the documentation, requires at least slurm 22.05, where it is automatically handled by the slurmdbd service. In the test lab, we performed the following tests: a. Incremental upgrade - according to MariaDB recommendations: 1. Upgrade MariaDB 5.5.68 -> 10.1.48 -> 10.2.44 2. Start the Slurm suite 22.05, checking content after each MariaDB upgrade step. During the 10.1 -> 10.2 upgrade, the slurmdbd service automatically converted the database to the required format. We had enabled general.log in MariaDB, allowing detailed inspection of database changes during conversion. 3. Upgrade slurmdbd to version 23.11 4. Upgrade slurmctld to version 23.11 5. Upgrade slurmd to version 23.11 6. Check the database content and compare tests before and after the upgrade (we used various reports with scontrol, sreport, sacct, sacctmgr for verification). b. Direct MariaDB upgrade from 5.5.68 to 10.2.44 using the same approach. According to the tests, this resulted in the same state as the incremental approach. PS: If you proceed with the upgrade, I would appreciate it if you could let us know about any potential challenges you encountered. Andrej Sec nscc, Bratislava, Slovakia _____ Od: "hermes via slurm-users" <slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> > Komu: slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> Odoslané: pondelok, 28. október 2024 8:48:19 Predmet: [slurm-users] =?eucgb2312_cn?q?=D7=AA=B7=A2=3A_What_is_the_safe_upgrade_path_when_upgrade_from_slurm21=2E08_and_mariadb5=2E5=3F?= Hi everyone: We are currently running business on SLURM21.08 and mariadb5.5. When talking about the upgrade, we need to keep all the users and jobs history data. And we see the official document wrote: “When upgrading an existing accounting database to MariaDB 10.2.1 or later from an older version of MariaDB or any version of MySQL, ensure you are running slurmdbd 22.05.7 or later. These versions will gracefully handle changes to MariaDB default values that can cause problems for slurmdbd.” So is this mean we have to firstly build SLURM>22.05 over mariadb5.5, and do the SLURM upgrade. Then upgrade the mariadb to newer version, and rebuild the same version of SLURM over new mariadb-devel? And is it safe to jump directly from mariadb5.5 to latest version? How can we check whether the slurm have correctly inherited the historical data? Thanks, Tianyang Zhang SJTU Network Information Center -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com> -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com> [View Less]

1 0

转发: What is the safe upgrade path when upgrade from slurm21.08 and mariadb5.5?
by taleintervenor＠sjtu.edu.cn 29 Oct '24

29 Oct '24

Hi everyone: We are currently running business on SLURM21.08 and mariadb5.5. When talking about the upgrade, we need to keep all the users and jobs history data. And we see the official document wrote: “When upgrading an existing accounting database to MariaDB 10.2.1 or later from an older version of MariaDB or any version of MySQL, ensure you are running slurmdbd 22.05.7 or later. These versions will gracefully handle changes to MariaDB default values that can cause problems for … [View More]

3 2

Job pre / post submit scripts
by Bhaskar Chakraborty 29 Oct '24

29 Oct '24

Hi, Is there an option in slurm to launch a custom script at the time of job submission through sbatchor salloc? The script should run with submit user permission in submit area. The idea is that we need to enquire something which characterises our job’s requirement like CPUslots, memory etc from a central server and we do need read access to user area prior to that. In our use case the user doesn’t necessarily know beforehand what kind of resource his job needs.(Hence, the need for such a … [View More]

4 4

Scheduling oddity with multiple GPU types in same partition
by Kevin M. Hildebrand 29 Oct '24

29 Oct '24

We have a 'gpu' partition with 30 or so nodes, some with A100s, some with H100s, and a few others. It appears that when (for example) all of the A100 GPUs are in use, if there are additional jobs requesting A100 GPUs pending, and those jobs have the highest priority in the partition, then jobs submitted for H100s won't run even if there are idle H100s. This is a small subset of our present pending queue- the four bottom jobs should be running, but aren't. The top pending job shows reason '… [View More]

2 1

I need to limit the number of jobs per user per partition
by Stephen Connolly 28 Oct '24

28 Oct '24

I am unable to limit the number of jobs per user per partition. I have searched the internet, the forums and the slurm documentation. I created a partition with a QOS having MaxJobsPU=1 and MaxJobsPA=1 Created a user stephen with account=stephen and MaxJobs=1 However if I sbatch a test job (sleep 180) multiple times they all run concurrently. I am at a loss of what else to do. Help would be very appreciated . Thank you -- Stephen Connolly JSI Data Systems Ltd 613-727-9353 stephen(a)jsidata.ca

2 1

jobs dropping
by Mihai Ciubancan 24 Oct '24

24 Oct '24

Hello, We are trying to run some PiconGPU codes on a machine with 8x100H, susing slurm. But the jobs don't run, and are not in the queue. In slurmd logs I have: [2024-10-24T09:50:40.934] CPU_BIND: _set_batch_job_limits: Memory extracted from credential for StepId=1079.batch job_mem_limit= 648000 [2024-10-24T09:50:40.934] Launching batch job 1079 for UID 1009 [2024-10-24T09:50:40.938] debug: acct_gather_energy/none: init: AcctGatherEnergy NONE plugin loaded [2024-10-24T09:50:40.938] debug:… [View More] acct_gather_profile/none: init: AcctGatherProfile NONE plugin loaded [2024-10-24T09:50:40.938] debug: acct_gather_interconnect/none: init: AcctGatherInterconnect NONE plugin loaded [2024-10-24T09:50:40.938] debug: acct_gather_filesystem/none: init: AcctGatherFilesystem NONE plugin loaded [2024-10-24T09:50:40.939] debug: gres/gpu: init: loaded [2024-10-24T09:50:41.022] [1079.batch] debug: cgroup/v2: init: Cgroup v2 plugin loaded [2024-10-24T09:50:41.026] [1079.batch] debug: CPUs:192 Boards:1 Sockets:2 CoresPerSocket:48 ThreadsPerCore:2 [2024-10-24T09:50:41.026] [1079.batch] debug: jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded [2024-10-24T09:50:41.026] [1079.batch] CPU_BIND: Memory extracted from credential for StepId=1079.batch job_mem_limit=648000 step_mem_limit=648000 [2024-10-24T09:50:41.027] [1079.batch] debug: laying out the 8 tasks on 1 hosts mihaigpu2 dist 2 [2024-10-24T09:50:41.027] [1079.batch] gres_job_state gres:gpu(7696487) type:(null)(0) job:1079 flags: [2024-10-24T09:50:41.027] [1079.batch] total_gres:8 [2024-10-24T09:50:41.027] [1079.batch] node_cnt:1 [2024-10-24T09:50:41.027] [1079.batch] gres_cnt_node_alloc[0]:8 [2024-10-24T09:50:41.027] [1079.batch] gres_bit_alloc[0]:0-7 of 8 [2024-10-24T09:50:41.027] [1079.batch] debug: Message thread started pid = 459054 [2024-10-24T09:50:41.027] [1079.batch] debug: Setting slurmstepd(459054) oom_score_adj to -1000 [2024-10-24T09:50:41.027] [1079.batch] debug: switch/none: init: switch NONE plugin loaded [2024-10-24T09:50:41.027] [1079.batch] debug: task/cgroup: init: core enforcement enabled [2024-10-24T09:50:41.027] [1079.batch] debug: task/cgroup: task_cgroup_memory_init: task/cgroup/memory: total:2063720M allowed:100%(enforced), swap:0%(permissive), max:100%(2063720M) max+swap:100%(4127440M) min:30M kmem:100%(2063720M permissive) min:30M [2024-10-24T09:50:41.027] [1079.batch] debug: task/cgroup: init: memory enforcement enabled [2024-10-24T09:50:41.027] [1079.batch] debug: task/cgroup: init: Tasks containment cgroup plugin loaded [2024-10-24T09:50:41.027] [1079.batch] cred/munge: init: Munge credential signature plugin loaded [2024-10-24T09:50:41.027] [1079.batch] debug: job_container/none: init: job_container none plugin loaded [2024-10-24T09:50:41.030] [1079.batch] debug: spank: opening plugin stack /etc/slurm/plugstack.conf [2024-10-24T09:50:41.030] [1079.batch] debug: task/cgroup: task_cgroup_cpuset_create: job abstract cores are '0-63' [2024-10-24T09:50:41.030] [1079.batch] debug: task/cgroup: task_cgroup_cpuset_create: step abstract cores are '0-63' [2024-10-24T09:50:41.030] [1079.batch] debug: task/cgroup: task_cgroup_cpuset_create: job physical CPUs are '0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,96,98,100,102,104,106,108,110,112,114,116,118,120,122,124,126,128,130,132,134,136,138,140,142,144,146,148,150,152,154,156,158' [2024-10-24T09:50:41.030] [1079.batch] debug: task/cgroup: task_cgroup_cpuset_create: step physical CPUs are '0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,96,98,100,102,104,106,108,110,112,114,116,118,120,122,124,126,128,130,132,134,136,138,140,142,144,146,148,150,152,154,156,158' [2024-10-24T09:50:41.031] [1079.batch] task/cgroup: _memcg_initialize: job: alloc=648000MB mem.limit=648000MB memsw.limit=unlimited [2024-10-24T09:50:41.031] [1079.batch] task/cgroup: _memcg_initialize: step: alloc=648000MB mem.limit=648000MB memsw.limit=unlimited [2024-10-24T09:50:41.064] [1079.batch] debug levels are stderr='error', logfile='debug', syslog='quiet' [2024-10-24T09:50:41.064] [1079.batch] starting 1 tasks [2024-10-24T09:50:41.064] [1079.batch] task 0 (459058) started 2024-10-24T09:50:41 [2024-10-24T09:50:41.069] [1079.batch] _set_limit: RLIMIT_NOFILE : reducing req:1048576 to max:131072 [2024-10-24T09:51:23.066] debug: _rpc_terminate_job: uid = 64030 JobId=1079 [2024-10-24T09:51:23.067] debug: credential for job 1079 revoked [2024-10-24T09:51:23.067] [1079.batch] debug: Handling REQUEST_SIGNAL_CONTAINER [2024-10-24T09:51:23.067] [1079.batch] debug: _handle_signal_container for StepId=1079.batch uid=64030 signal=18 [2024-10-24T09:51:23.068] [1079.batch] Sent signal 18 to StepId=1079.batch [2024-10-24T09:51:23.068] [1079.batch] debug: Handling REQUEST_SIGNAL_CONTAINER [2024-10-24T09:51:23.068] [1079.batch] debug: _handle_signal_container for StepId=1079.batch uid=64030 signal=15 [2024-10-24T09:51:23.068] [1079.batch] error: *** JOB 1079 ON mihaigpu2 CANCELLED AT 2024-10-24T09:51:23 *** [2024-10-24T09:51:23.069] [1079.batch] Sent signal 15 to StepId=1079.batch [2024-10-24T09:51:23.069] [1079.batch] debug: Handling REQUEST_STATE [2024-10-24T09:51:23.071] [1079.batch] task 0 (459058) exited. Killed by signal 15. [2024-10-24T09:51:23.090] [1079.batch] debug: Handling REQUEST_STATE [2024-10-24T09:51:23.141] [1079.batch] debug: Handling REQUEST_STATE [2024-10-24T09:51:23.241] [1079.batch] debug: Handling REQUEST_STATE [2024-10-24T09:51:23.741] [1079.batch] debug: Handling REQUEST_STATE [2024-10-24T09:51:24.073] [1079.batch] debug: signaling condition [2024-10-24T09:51:24.073] [1079.batch] debug: jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded [2024-10-24T09:51:24.073] [1079.batch] debug: task/cgroup: fini: Tasks containment cgroup plugin unloaded [2024-10-24T09:51:24.073] [1079.batch] debug: get_exit_code task 0 killed by cmd [2024-10-24T09:51:24.073] [1079.batch] job 1079 completed with slurm_rc = 0, job_rc = 15 [2024-10-24T09:51:24.075] [1079.batch] debug: Message thread exited [2024-10-24T09:51:24.154] [1079.batch] done with job Anyone have any idea what could be the problem? Thank you, Mihai [View Less]

2 1

Randomly draining nodes
by Nacereddine Laddaoui 24 Oct '24

24 Oct '24

Hello everyone, I’ve recently encountered an issue where some nodes in our cluster enter a drain state randomly, typically after completing long-running jobs. Below is the output from the |sinfo| command showing the reason *“Prolog error”* : |root@controller-node:~# sinfo -R REASON USER TIMESTAMP NODELIST Prolog error slurm 2024-09-24T21:18:05 node[24,31] | When checking the |slurmd.log| files on the nodes, I noticed the following errors: |[2024-09-24T17:18:22.386] [217703.extern] … [View More]

6 9

loss of "unchangeable" node features
by Laura Hild 23 Oct '24

23 Oct '24

Has anyone else noticed, somewhere between versions 22.05.11 and 23.11.9, losing fixed Features defined for a node in slurm.conf, and instead now just having those controlled by a NodeFeaturesPlugin like node_features/knl_generic?

1 0

2025

2024

slurm-users ----- 2025 ----- May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users