- slurm-users - lists.schedmd.com

Randomly draining nodes
by Nacereddine Laddaoui 24 Oct '24

24 Oct '24

Hello everyone, I’ve recently encountered an issue where some nodes in our cluster enter a drain state randomly, typically after completing long-running jobs. Below is the output from the |sinfo| command showing the reason *“Prolog error”* : |root@controller-node:~# sinfo -R REASON USER TIMESTAMP NODELIST Prolog error slurm 2024-09-24T21:18:05 node[24,31] | When checking the |slurmd.log| files on the nodes, I noticed the following errors: |[2024-09-24T17:18:22.386] [217703.extern] error: _handle_add_extern_pid_internal: Job 217703 can't add pid 3311892 to jobacct_gather plugin in the extern_step. **(repeated 90 times)** [2024-09-24T17:18:22.917] [217703.extern] error: _handle_add_extern_pid_internal: Job 217703 can't add pid 3313158 to jobacct_gather plugin in the extern_step. ... [2024-09-24T21:17:45.162] launch task StepId=217703.0 request from UID:54059 GID:1600 HOST:<SLURMCTLD_IP> PORT:53514 [2024-09-24T21:18:05.166] error: Waiting for JobId=217703 REQUEST_LAUNCH_PROLOG notification failed, giving up after 20 sec [2024-09-24T21:18:05.166] error: slurm_send_node_msg: [(null)] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC_MSG) failed: Unexpected missing socket error [2024-09-24T21:18:05.166] error: _rpc_launch_tasks: unable to send return code to address:port=<SLURMCTLD_IP>:53514 msg_type=6001: No such file or directory | If you know how to solve these errors, please let me know. I would greatly appreciate any guidance or suggestions for further troubleshooting. Thank you in advance for your assistance. Best regards, -- Télécom Paris <https://www.telecom-paris.fr> *Nacereddine LADDAOUI* Ingénieur de Recherche et de Développement 19 place Marguerite Perey CS 20031 91123 Palaiseau Cedex Site web Télécom Paris <https://www.telecom-paris.fr>X Télécom Paris <https://twitter.com/TelecomParis_>Facebook Télécom Paris <https://www.facebook.com/TelecomParis>LinkedIn Télécom Paris <https://www.linkedin.com/school/telecom-paris/>Instagram Télécom Paris <https://www.instagram.com/telecom_paris/>Blog Télécom Paris <https://imtech.wp.imt.fr> Une école de l'IMT <https://www.imt.fr>

6 9

loss of "unchangeable" node features
by Laura Hild 23 Oct '24

23 Oct '24

Has anyone else noticed, somewhere between versions 22.05.11 and 23.11.9, losing fixed Features defined for a node in slurm.conf, and instead now just having those controlled by a NodeFeaturesPlugin like node_features/knl_generic?

1 0

Slurm version 24.05.4 is now available (CVE-2024-48936)
by Tim Wickberg 23 Oct '24

23 Oct '24

Slurm version 24.05.4 is now available and includes a fix for a recently discovered security issue with the new stepmgr subsystem. SchedMD customers were informed on October 9th and provided a patch on request; this process is documented in our security policy. [1] A mistake in authentication handling in stepmgr could permit an attacker to execute processes under other users' jobs. This is limited to jobs explicitly running with --stepmgr, or on systems that have globally enabled stepmgr through "SlurmctldParameters=enable_stepmgr" in their configuration. CVE-2024-48936. Downloads are available at https://www.schedmd.com/downloads.php . Release notes follow below. - Tim [1] https://www.schedmd.com/security-policy/ -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support > * Changes in Slurm 24.05.4 > ========================== > -- Fix generic int sort functions. > -- Fix user look up using possible unrealized uid in the dbd. > -- Fix FreeBSD compile issue with tls/none plugin. > -- slurmrestd - Fix regressions that allowed slurmrestd to be run as SlurmUser > when SlurmUser was not root. > -- mpi/pmix fix race conditions with het jobs at step start/end which could > make srun to hang. > -- Fix not showing some SelectTypeParameters in scontrol show config. > -- Avoid assert when dumping removed certain fields in JSON/YAML. > -- Improve how shards are scheduled with affinity in mind. > -- Fix MaxJobsAccruePU not being respected when MaxJobsAccruePA is set > in the same QOS. > -- Prevent backfill from planning jobs that use overlapping resources for the > same time slot if the job's time limit is less than bf_resolution. > -- Fix memory leak when requesting typed gres and --[cpus|mem]-per-gpu. > -- Prevent backfill from breaking out due to "system state changed" every 30 > seconds if reservations use REPLACE or REPLACE_DOWN flags. > -- slurmrestd - Make sure that scheduler_unset parameter defaults to true even > when the following flags are also set: show_duplicates, skip_steps, > disable_truncate_usage_time, run_away_jobs, whole_hetjob, > disable_whole_hetjob, disable_wait_for_result, usage_time_as_submit_time, > show_batch_script, and or show_job_environment. Additionaly, always make > sure show_duplicates and disable_truncate_usage_time default to true when > the following flags are also set: scheduler_unset, scheduled_on_submit, > scheduled_by_main, scheduled_by_backfill, and or job_started. This effects > the following endpoints: > 'GET /slurmdb/v0.0.40/jobs' > 'GET /slurmdb/v0.0.41/jobs' > -- Ignore --json and --yaml options for scontrol show config to prevent mixing > output types. > -- Fix not considering nodes in reservations with Maintenance or Overlap flags > when creating new reservations with nodecnt or when they replace down nodes. > -- Fix suspending/resuming steps running under a 23.02 slurmstepd process. > -- Fix options like sprio --me and squeue --me for users with a uid greater > than 2147483647. > -- fatal() if BlockSizes=0. This value is invalid and would otherwise cause the > slurmctld to crash. > -- sacctmgr - Fix issue where clearing out a preemption list using > preempt='' would cause the given qos to no longer be preempt-able until set > again. > -- Fix stepmgr creating job steps concurrently. > -- data_parser/v0.0.40 - Avoid dumping "Infinity" for NO_VAL tagged "number" > fields. > -- data_parser/v0.0.41 - Avoid dumping "Infinity" for NO_VAL tagged "number" > fields. > -- slurmctld - Fix a potential leak while updating a reservation. > -- slurmctld - Fix state save with reservation flags when a update fails. > -- Fix reservation update issues with parameters Accounts and Users, when > using +/- signs. > -- slurmrestd - Don't dump warning on empty wckeys in: > 'GET /slurmdb/v0.0.40/config' > 'GET /slurmdb/v0.0.41/config' > -- Fix slurmd possibly leaving zombie processes on start up in configless when > the initial attempt to fetch the config fails. > -- Fix crash when trying to drain a non-existing node (possibly deleted > before). > -- slurmctld - fix segfault when calculating limit decay for jobs with an > invalid association. > -- Fix IPMI energy gathering with multiple sensors. > -- data_parser/v0.0.39 - Remove xassert requiring errors and warnings to have a > source string. > -- slurmrestd - Prevent potential segfault when there is an error parsing an > array field which could lead to a double xfree. This applies to several > endpoints in data_parser v0.0.39, v0.0.40 and v0.0.41. > -- scancel - Fix a regression from 23.11.6 where using both the --ctld and > --sibling options would cancel the federated job on all clusters instead of > only the cluster(s) specified by --sibling. > -- accounting_storage/mysql - Fix bug when removing an association > specified with an empty partition. > -- Fix setting multiple partition state restore on a job correctly. > -- Fix difference in behavior when swapping partition order in job submission. > -- Fix security issue in stepmgr that could permit an attacker to execute > processes under other users' jobs. CVE-2024-48936.

1 0

HOST CPUs not used by jobs
by Bhaskar Chakraborty 21 Oct '24

21 Oct '24

I have a SLURM configuration of 2 hosts with 6 + 4 CPUs. I am submitting jobs with sbatch -n <CPU slots> <job script>. However, I see that even when I have exhausted all 10 CPU slots for the running jobs it's still allowing subsequent jobs to run ! The CPU slots availability is also show as full for the 2 hosts. No job is found pending. What could be problem? My Slurm.conf looks like (host names are changed to generic): ClusterName=MyClusterControlMachine=host1ControlAddr=<some address>SlurmUser=slurmsa#AuthType=auth/mungeStateSaveLocation=/var/spool/slurmdSlurmdSpoolDir=/var/spool/slurmdSlurmctldLogFile=/var/log/slurm/slurmctld.logSlurmdDebug=3SlurmctldDebug=6SlurmdLogFile=/var/log/slurm/slurmd.logAccountingStorageType=accounting_storage/slurmdbdAccountingStorageHost=host1#AccountingStoragePass=medslurmpass#AccountingStoragePass=/var/run/munge/munge.socket.2AccountingStorageUser=slurmsa#TaskPlugin=task/cgroupNodeName=host1 CPUs=6 SocketsPerBoard=3 CoresPerSocket=2 ThreadsPerCore=1 State=UNKNOWNNodeName=host2 CPUs=4 ThreadsPerCore=1 State=UNKNOWNPartitionName=debug Nodes=host1,host2 Default=YES MaxTime=INFINITE State=UPJobAcctGatherType=jobacct_gather/linuxJobAcctGatherFrequency=30 SelectType=select/cons_tresSelectTypeParameters=CR_CPUTaskPlugin=task/affinity Thanks in advance for any help! Regards,Bhaskar.

2 1

Customization of (error) messages after the job submission
by Sebastian Sitkiewicz 21 Oct '24

21 Oct '24

Dear SLUR Users and Administrators, I am interested in a way to customize the job submission exit statuses (mainly error codes) after the job has already been queued by the SLURM controller. We aim to provide more user-friendly messages and reminders in case of any errors or obstacles (also adjusted to our QoS/account system). For example, in the case of exceeding CPU minutes of given QoS (or account) and after the (successful) job submission, we would like to notify the user that his job has been queued (as it should) but won’t start until the CPU minutes limits are increased (and that he should contact the administrators to apply for more resources). Similarly, if the user queued a job that cannot be launched immediately because of exceeding the MaxJobs limit (per user), we would like to also give him some additional message after the srun/sbatch submission. We want to provide such information immediately after the job submission, without the need to check the status using `squeue` by the user. In the Job Launch Guide (https://slurm.schedmd.com/job_launch.html) there are distinguished following steps: 1. Call job_submit plugins to modify the request as appropriate 2. Validate that the options are valid for this user (e.g. valid partition name, valid limits, etc.) 3. Determine if this job is the highest priority runnable job, if so then really try to allocate resources for it now, otherwise only validate that it could run if no other jobs existed 4. Determine which nodes could be used for the job. If the feature specification uses an exclusive OR option, then multiple iterations of the selection process below will be required with disjoint sets of nodes 5. Call the select plugin to select the best resources for the request 6. The select plugin will consider network topology and the topology within a node (e.g. sockets, cores, and threads) to select the best resources for the job 7. If the job can not be initiated using available resources and preemption support is configured, the select plugin will also determine if the job can be initiated after preempting lower priority jobs. If so then initiate preemption as needed to start the job. From my understanding, to achieve our goal one would need to have access to source code or plugin related to point 2 (and some part of point 3). Unfortunately, the job_submit (lua) plugin from point 1 (and the cli_filter plugin as well) cannot be used because it only has access to the information on the parameters of the submitted job and the SLURM partitions (but not the QoS/account usage and their limits). Is there any way to extend the customization of job submission to include such features? Best regards, Sebastian -- dr inż. Sebastian Sitkiewicz Politechnika Wrocławska Wrocławskie Centrum Sieciowo-Superkomputerowe Dział Usług Obliczeniowych Wyb. Wyspiańskiego 27 50-370 Wrocław www.wcss.pl

1 0

Tracking costs - one single pool of credits, variable costs per partition
by John Snowdon 18 Oct '24

18 Oct '24

We are trying to design the charging and accounting system for our new institutional HPC facility and I'm having difficulty understanding exactly how we can use sacctmgr to achieve what we need. Until now, our previous HPC facilities have all operated as free delivery and we have not needed to track costs by user/group/project. Account codes have been purely optional. However, our new facility will be split into various resource types, with free partitions and paid/priority/reserved partitions across those resource types. All jobs will need to be submitted with an account code. For users submitting to 'free' partitions we don't need to track resource units against a balance, but the submitted account code would still be used for reporting purposes (i.e. "free resources accounted for % of all use by this project in August-September"). When submitting to a 'paid' partition, the account code needs to be checked to ensure it has a positive balance (or a balance that will not go past some negative threshold). Each of the 'paid' partitions may (will) have a different resource unit cost. A simple example: - Submit to a generic CPU paid partition -- 1 resource unit/token/credit/£/$ per allocated cpu, per hour of compute - Submit to a high-speed, non-blocking CPU paid partition -- 2 resource unit/token/credit/£/$ per allocated cpu, per hour of compute - Submit to a GPU paid partition -- 4 resource unit/token/credit/£/$ per allocated GPU card, per hour of compute We need to have *one* pool of resource units/tokens/credits per account - let's say 1000 credits, and a group of users may well decide to spend all of their credits on the generic CPU partition, all on the GPU partition, or some mixture of the two. So in the above examples, assuming one user (or group of users sharing the same account code) submit a 2 hour job to all three partitions, their one, single account code should be charged: - 2 units for the generic CPU partition - 4 units for the job on the low latency partition - 8 units for the gpu partition. - A total of 14 credits removed from their single account code Is this feasible to achieve without having to allocate credits to each of the partitions for an account, or creating a QOS variant for each and every combination of account and partition? John Snowdon Senior Research Infrastructure Engineer (HPC) Research Software Engineering Catalyst Building, Room 2.01 Newcastle University 3 Science Square Newcastle Helix Newcastle upon Tyne NE4 5TG https://rse.ncldata.dev/

2 1

How do you guys track which GPU is used by which job ?
by Sylvain MARET 18 Oct '24

18 Oct '24

Hey guys ! I'm looking to improve GPU monitoring on our cluster. I want to install this https://github.com/NVIDIA/dcgm-exporter and saw in the README that it can support tracking of job id : https://github.com/NVIDIA/dcgm-exporter?tab=readme-ov-file#enabling-hpc-job… However I haven't been able to see any examples on how to do it nor does slurm seem to expose this information by default. Does anyone do this here ? And if so do you have any examples I could try to follow ? If you have advise on best practices to monitor GPU I'd be happy to hear it out ! Regards, Sylvain Maret

5 6

Job information is not being added to accounting database on new setup
by Adrian Brady 17 Oct '24

17 Oct '24

Hi Everyone, I'm a new to slurm administration and looking for a bit of help! Just added Accounting to an existing cluster but job information is not being added to the Accounting Mariadb. When I submit a test job it gets scheduled fine and its visible with squeue, I get nothing returned from sacct! I have turned up the logging to debug5 on both slurmctld and slurmdbd logs and can't see any errors. I believe all the comms are ok between slurmctld and slurmdbd as when I enter the sacct command I can see the database is being queried but returning nothing, because nothing has been added to the tables. The cluster tables were created fine when I ran #sacctmgr add cluster ny5ktt $ sacct JobID JobName Partition Account AllocCPUS State ExitCode ------------ ---------- ---------- ---------- ---------- ---------- -------- # tail -f slurmdbd.log [2024-10-17T12:34:45.232] debug: REQUEST_PERSIST_INIT: CLUSTER:ny5ktt VERSION:9216 UID:10001 IP:10.202.233.117 CONN:10 [2024-10-17T12:34:45.232] debug2: accounting_storage/as_mysql: acct_storage_p_get_connection: acct_storage_p_get_connection: request new connection 1 [2024-10-17T12:34:45.233] debug2: Attempting to connect to localhost:3306 [2024-10-17T12:34:45.274] debug2: DBD_GET_JOBS_COND: called [2024-10-17T12:34:45.317] debug2: DBD_FINI: CLOSE:1 COMMIT:0 [2024-10-17T12:34:45.317] debug4: accounting_storage/as_mysql: acct_storage_p_commit: got 0 commits The Mariadb is running on it own node with slurmdbd and munged for authentication. I haven't setup any accounts, users, asssociations or enforcements yet. On my lab cluster, jobs were visible in the database without these being setup. I guess I must be missing something simple in the config that is stopping jobs being reported to slurmdbd. Master Node packages # rpm -qa |grep slurm slurm-slurmdbd-20.11.9-1.el8.x86_64 slurm-libs-20.11.9-1.el8.x86_64 slurm-20.11.9-1.el8.x86_64 slurm-slurmd-20.11.9-1.el8.x86_64 slurm-perlapi-20.11.9-1.el8.x86_64 slurm-doc-20.11.9-1.el8.x86_64 slurm-contribs-20.11.9-1.el8.x86_64 slurm-slurmctld-20.11.9-1.el8.x86_64 Database Node packages # rpm -qa |grep slurm slurm-slurmdbd-20.11.9-1.el8.x86_64 slurm-20.11.9-1.el8.x86_64 slurm-libs-20.11.9-1.el8.x86_64 slurm-devel-20.11.9-1.el8.x86_64 slurm.conf # # See the slurm.conf man page for more information. # ClusterName=ny5ktt ControlMachine=ny5-pr-kttslurm-01 ControlAddr=10.202.233.71 #BackupController= #BackupAddr= # AuthType=auth/munge #CheckpointType=checkpoint/none CryptoType=crypto/munge #DisableRootJobs=NO #EnforcePartLimits=NO #Epilog= #EpilogSlurmctld= #FirstJobId=1 #MaxJobId=999999 #GresTypes= #GroupUpdateForce=0 #GroupUpdateTime=600 #JobCheckpointDir=/var/slurm/checkpoint #JobCredentialPrivateKey= #JobCredentialPublicCertificate= #JobFileAppend=0 #JobRequeue=1 #JobSubmitPlugins= #KillOnBadExit=0 #LaunchType=launch/slurm #Licenses=foo*4,bar MailProg=/bin/true MaxJobCount=200000 #MaxStepCount=40000 #MaxTasksPerNode=128 MpiDefault=none #MpiParams=ports=#-# #PluginDir= #PlugStackConfig= #PrivateData=jobs ProctrackType=proctrack/cgroup #Prolog= #PrologFlags= #PrologSlurmctld= #PropagatePrioProcess=0 #PropagateResourceLimits= #PropagateResourceLimitsExcept= #RebootProgram= ReturnToService=1 #SallocDefaultCommand= SlurmctldPidFile=/var/run/slurm/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurm/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurm/d SlurmUser=slurm #SlurmdUser=root #SrunEpilog= #SrunProlog= StateSaveLocation=/var/spool/slurm/ctld SwitchType=switch/none #TaskEpilog= TaskPlugin=task/none #TaskPluginParam= #TaskProlog= #TopologyPlugin=topology/tree #TmpFS=/tmp #TrackWCKey=no #TreeWidth= #UnkillableStepProgram= #UsePAM=0 # # # TIMERS #BatchStartTimeout=10 #CompleteWait=0 #EpilogMsgTime=2000 #GetEnvTimeout=2 #HealthCheckInterval=0 #HealthCheckProgram= InactiveLimit=0 KillWait=30 #MessageTimeout=10 #ResvOverRun=0 #MinJobAge=300 #MinJobAge=43200 # CHG0057915 MinJobAge=14400 # CHG0057915 #MaxJobCount=50000 #MaxJobCount=100000 #OverTimeLimit=0 SlurmctldTimeout=120 SlurmdTimeout=300 #UnkillableStepTimeout=60 #VSizeFactor=0 Waittime=0 # # # SCHEDULING DefMemPerCPU=3000 #FastSchedule=1 #MaxMemPerCPU=0 #SchedulerTimeSlice=30 SchedulerType=sched/backfill SelectType=select/cons_tres #SelectTypeParameters=CR_Core #SelectTypeParameters=CR_CPU SelectTypeParameters=CR_CPU_Memory # ECR CHG0056915 10/14/2023 MaxArraySize=5001 # # # JOB PRIORITY #PriorityFlags= #PriorityType=priority/basic #PriorityDecayHalfLife= #PriorityCalcPeriod= #PriorityFavorSmall= #PriorityMaxAge= #PriorityUsageResetPeriod= #PriorityWeightAge= #PriorityWeightFairshare= #PriorityWeightJobSize= #PriorityWeightPartition= #PriorityWeightQOS= # # # LOGGING AND ACCOUNTING #AccountingStorageEnforce=0 #AccountingStorageEnforce=limits AccountingStorageHost=ny5-pr-kttslurmdb-01.ktt.schonfeld.com #AccountingStorageLoc= #AccountingStoragePass= #AccountingStoragePort= #AccountingStorageType=accounting_storage/none AccountingStorageType=accounting_storage/slurmdbd #AccountingStorageUser= AccountingStoreJobComment=YES #DebugFlags= #JobCompHost= #JobCompLoc= #JobCompPass= #JobCompPort= JobCompType=jobcomp/none #JobCompUser= #JobContainerType=job_container/none JobAcctGatherFrequency=60 JobAcctGatherType=jobacct_gather/none SlurmctldDebug=info SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurm/slurmd.log #SlurmdLogFile= #SlurmSchedLogFile= #SlurmSchedLogLevel= # # # POWER SAVE SUPPORT FOR IDLE NODES (optional) #SuspendProgram= #ResumeProgram= #SuspendTimeout= #ResumeTimeout= #ResumeRate= #SuspendExcNodes= #SuspendExcParts= #SuspendRate= #SuspendTime= # # # COMPUTE NODES ##using fqdn since the ctld domain is different. Can't use regex since it's not at the end ##save 17 and 18 as headnodes #NodeName=ny5-dv-kttres-17 Sockets=1 CoresPerSocket=18 ThreadsPerCore=2 Feature=HyperThread RealMemory=102400 #NodeName=ny5-dv-kttres-18 Sockets=1 CoresPerSocket=14 ThreadsPerCore=2 Feature=HyperThread RealMemory=102400 NodeName=ny5-dv-kttres-19 Sockets=1 CoresPerSocket=12 ThreadsPerCore=2 Feature=HyperThread RealMemory=102400 NodeName=ny5-dv-kttres-[20-21] Sockets=1 CoresPerSocket=18 ThreadsPerCore=2 Feature=HyperThread RealMemory=102400 NodeName=ny5-dv-kttres-[01-16] Sockets=2 CoresPerSocket=16 ThreadsPerCore=2 Feature=HyperThread RealMemory=233472 NodeName=ny5-dv-kttres-[22-35] Sockets=2 CoresPerSocket=32 ThreadsPerCore=2 Feature=HyperThread RealMemory=346884 PartitionName=ktt_slurm_light_1 Nodes=ny5-dv-kttres-[19-21] Default=NO MaxTime=INFINITE State=UP OverSubscribe=FORCE:2 PartitionName=ktt_slurm_medium_1 Nodes=ny5-dv-kttres-[01-08] Default=NO MaxTime=INFINITE State=UP OverSubscribe=FORCE:2 PartitionName=ktt_slurm_medium_2 Nodes=ny5-dv-kttres-[09-16] Default=NO MaxTime=INFINITE State=UP OverSubscribe=FORCE:2 PartitionName=ktt_slurm_medium_3 Nodes=ny5-dv-kttres-[22-28] Default=NO MaxTime=INFINITE State=UP OverSubscribe=FORCE:2 PartitionName=ktt_slurm_medium_4 Nodes=ny5-dv-kttres-[29-35] Default=NO MaxTime=INFINITE State=UP OverSubscribe=FORCE:2 PartitionName=ktt_slurm_large_1 Nodes=ny5-dv-kttres-[01-16] Default=YES MaxTime=INFINITE State=UP OverSubscribe=FORCE:2 PartitionName=ktt_slurm_large_2 Nodes=ny5-dv-kttres-[22-35] Default=NO MaxTime=INFINITE State=UP OverSubscribe=FORCE:2 Slurmdbd.conf AuthType=auth/munge DbdAddr=10.202.233.72 DbdHost=ny5-pr-kttslurmdb-01 DebugLevel=debug5 LogFile=/var/log/slurm/slurmdbd.log PidFile=/tmp/slurmdbd.pid StorageType=accounting_storage/mysql StorageHost=localhost #StorageHost=10.234.132.57 StorageUser=slurm SlurmUser=slurm StoragePass=xxxxxxx #StorageUser=slurm #StorageLoc=slurm_acct_db Database tables MariaDB [slurm_acct_db]> show tables; +--------------------------------+ | Tables_in_slurm_acct_db | +--------------------------------+ | acct_coord_table | | acct_table | | clus_res_table | | cluster_table | | convert_version_table | | federation_table | | ny5ktt_assoc_table | | ny5ktt_assoc_usage_day_table | | ny5ktt_assoc_usage_hour_table | | ny5ktt_assoc_usage_month_table | | ny5ktt_event_table | | ny5ktt_job_table | | ny5ktt_last_ran_table | | ny5ktt_resv_table | | ny5ktt_step_table | | ny5ktt_suspend_table | | ny5ktt_usage_day_table | | ny5ktt_usage_hour_table | | ny5ktt_usage_month_table | | ny5ktt_wckey_table | | ny5ktt_wckey_usage_day_table | | ny5ktt_wckey_usage_hour_table | | ny5ktt_wckey_usage_month_table | | qos_table | | res_table | | table_defs_table | | tres_table | | txn_table | | user_table | +--------------------------------+ Many Thanks Adrian Disclaimer Schonfeld Strategic Advisors (UK) LLP (“SSA UK”) is authorised and regulated by The Financial Conduct Authority. SSA UK is a limited liability partnership in England and Wales (No: OC420598) and its registered office is at 54 Jermyn Street, London, SW1Y 6LX. The contents of this message, including any attachments, are meant solely for the intended recipient and may be confidential, privileged, or otherwise protected from disclosure. If you receive this message in error, immediately alert the sender by reply e-mail, delete it and any attachments or copies from your systems, and do not read, disclose, distribute, or otherwise use the information contained herein. We do not waive any confidentiality or privilege if this message was misdirected. This e-mail does not constitute an offer to sell or a solicitation to buy any securities or an offer of any investment advisory services. If you reply to this email please note that we invest in securities and do not want to receive material, non-public information and you are instructed not to communicate any such information to us. We do not agree to keep confidential any information you provide nor restrict our trading activity, except as agreed pursuant to a written confidentiality agreement duly executed by us. We reserve the right to monitor and review the content of all messages sent to or from this e-mail address.

1 0

Issue with interactive jobs
by Nerjes, Onno 17 Oct '24

17 Oct '24

Dear all, we've set up SLURM 24.05.3 on our cluster and are experiencing an issue with interactive jobs. Before, we used 21.08 and pretty much the same settings, but without these issues. We've started with a fresh DB etc. The behavior of interactive jobs is very erratic. Sometimes they start absolutely fine, at other times they die silently in the background, while the user has to wait indefinitely. We have been unable to isolate certain users or nodes affected by this. On a given node, one user might be able to start an interactive job, while another user at the same time isn't able to. The day after, the situation might be the other way around. The exception are jobs that use a reservation. These start fine every time as far as we can tell. At the same time, the number of idle nodes does not seem to influence the behavior I described above. Failed allocation on the front end: [user1@login1 ~]$ salloc salloc: Pending job allocation 5052052 salloc: job 5052052 queued and waiting for resources The same job on the backend: 2024-10-14 11:41:57.680 slurmctld: _job_complete: JobId=5052052 done 2024-10-14 11:41:57.678 slurmctld: _job_complete: JobId=5052052 WEXITSTATUS 1 2024-10-14 11:41:57.678 slurmctld: Killing interactive JobId=5052052: Communication connection failure 2024-10-14 11:41:46.666 slurmctld: sched/backfill: _start_job: Started JobId=5052052 in devel on m02n01 2024-10-14 11:41:30.096 slurmctld: sched: _slurm_rpc_allocate_resources JobId=5052052 NodeList=(null) usec=6258 Raising the debug level has not brought additional information. We were hoping, that one of you might be able to provide some insight into what the next steps in troubleshooting might be. Best regards, Onno

1 0

Dependency jobs
by adam＠bramblecfd.com 17 Oct '24

17 Oct '24

Hi Slurm Users, I am trying to figure out if there is a way you can check if a running job has any jobs queued up after it that depend on the current running job, I know you can show job info and find what dependency a job is waiting for, But more after checking if there are jobs waiting on the current job to complete using the job ID, Many Thanks, Adam

3 2

AllowAccounts partition setting
by Marko Markoc 16 Oct '24

16 Oct '24

Hi All, At one point, when we were running slurm 22, we set up a few new partitions with `AllowAccounts` setting enabled. We limit access to these partitions only to a few accounts. We manually create these accounts and users using the `sacctmgr` command. At that time, everything was working. Only users in these accounts were able to run jobs in that partition. Few months ago we upgraded slurm to 23.02.7 but just a couple of days ago we noticed that anyone could submit jobs to these partitions. In addition to that, you can provide any value to the `--account` directive and slurm will accept and record it in the job accounting database. In our initial testing, you could only provide `--account` value that has been previously created. I'm still looking into this but I wanted to check if there was any change in version 23 that could have impacted this? I don't remember seeing anything in release notes. Thanks, Marko

6 8

Problem with nodes with 1 gpu
by Jörg Striewski 16 Oct '24

16 Oct '24

i cannot send jobs to nodes with one gpu, i don't find the bug in my configuration. can someone help me ? in slurm.conf GresTypes=gpu is set this are some nodes in slurm.conf NodeName=gpu-[001-003] CPUs=8 SocketsPerBoard=1 CoresPerSocket=4 RealMemory=31000 Gres=gpu:1080:1 NodeName=gpu-[010-019] CPUs=16 SocketsPerBoard=1 CoresPerSocket=8 RealMemory=64000 Gres=gpu:1080:2 the partition for this gpu nodes is # General GPU partitions PartitionName=GPU Nodes=gpu-[001-003,010-019] AllowAccounts=staff PreemptMode=REQUEUE PriorityTier=0 DefMemPerGPU=32000 DefCpuPerGPU=8 CpuBind=none TRESBillingWeights="GRES/gpu=1000" GraceTime=300 this are the entries for some nodes in gres.conf NodeName=gpu-[001-003] Name=gpu Type=1080 File=/dev/nvidia0 NodeName=gpu-[010-019] Name=gpu Type=1080 File=/dev/nvidia[0-1] when i send a job with sbatch to gpu-001 #SBATCH --job-name=hello #SBATCH --ntasks-per-node=1 #SBATCH --output=hello_%A.out #SBATCH --time=00:10:00 #SBATCH --mail-type=ALL #SBATCH --mail-user=striewski(a)ismll.de #SBATCH --partition=GPU #SBATCH --nodelist=gpu-001 #SBATCH --gres=gpu:1 [...] i get the error sbatch: error: Batch job submission failed: Requested node configuration is not available when i send the job to a node with 2 gpu's it runs with no error, just setting --nodelist=gpu-12 has someone a hint what i made wrong ? Mit freundlichen Grüßen / kind regards -- Jörg Striewski Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim Germany post address: Universitätsplatz 1, D-31141Hildesheim, Germany visitor address: Samelsonplatz 1, D-31141 Hildesheim,Germany Tel.(+49) 05121 / 883-40392 http://www.ismll.uni-hildesheim.de

2 2

unsuscribe
by LEROY Christine 208562 15 Oct '24

15 Oct '24

unsuscribe

1 0

How does --nodes=min[-max] determine number of nodes to allocate?
by Declan Valters 08 Oct '24

08 Oct '24

Hi, I am trying to understand how Slurm determines the actual number of nodes to allocate to a job specified with -nodes=min[-max] I've been trying to follow the source code through on Github - could anyone point me in the right area(s)/source files/functions to look at to understand the program logic behind this. Many thanks in advance, Declan The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th' ann an Oilthigh Dh?n ?ideann, cl?raichte an Alba, ?ireamh cl?raidh SC005336.

2 1

Jobs not getting scheduled, no priority calculation, but still in queue?
by Cutts, Tim 07 Oct '24

07 Oct '24

Something odd is going on on our cluster. User has a lot of pending jobs in a job array (a few thousand). squeue -u kmnx005 -r -t PD | head -5 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 3045324_875 core run_scp_ kmnx005 PD 0:00 1 (JobArrayTaskLimit) 3045324_876 core run_scp_ kmnx005 PD 0:00 1 (JobArrayTaskLimit) 3045324_877 core run_scp_ kmnx005 PD 0:00 1 (JobArrayTaskLimit) 3045324_878 core run_scp_ kmnx005 PD 0:00 1 (JobArrayTaskLimit) None are getting scheduled. But when I ask SLURM what that job’s priority is, it produces no output: $ sprio -j 3045324 JOBID PARTITION PRIORITY SITE AGE FAIRSHARE JOBSIZE PARTITION QOS TRES Any clues what’s going on here? -- Tim Cutts Scientific Computing Platform Lead AstraZeneca Find out more about R&D IT Data, Analytics & AI and how we can support you by visiting our Service Catalogue<https://azcollaboration.sharepoint.com/sites/CMU993> | ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com<https://www.astrazeneca.com>

2 3

SLURM GRES reservation not working properly on 24.05.1
by Minulakshmi S 07 Oct '24

07 Oct '24

Hello, *Issue 1:* I am using slurm version 24.05.1 , my slurmd has a single node where I connect multiple gres by enabling the overscribe feature. I am able to use the advance reservation of gres only using *gres** name* (tres=gres/gpu:*SYSTEM12*). i.e while in reservation period , if other users submits job with SYSTEM12 , then slurm places this job in queue *user1@host$ srun --gres=gpu:SYSTEM12:1 hostname* *srun: job 333 queued and waiting for resources * but when other users just submit a job without any system name , slurm jobs goes through on that gres immediately even though it is reserved. *user1@host$ srun --gres=gpu:1 hostname * *mylinux.wbi.com <http://mylinux.wbi.com/> * Also I can see GresUsed in busy mode using "*scontrol show node -d*" , this means the job is running on Gres/GPU and not on cpu etc. Same way , job submission based on Feature "rev1 in my case" is also going through even though it is reserved for other users in multiple partition slurm. *snippet of slurm.conf file* NodeName=cluster01 NodeAddr=cluster Port=6002CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2 Feature="rev1" Gres=gpu:SYSTEM12:1 RealMemory=64171 State=IDLE *Issue 2:* while execution , Slurm o/p's some extra prints in the srun output user1@host$ srun --gres=gpu:1 hostname srun: error: extract_net_cred: net_cred not provided srun: error: Malformed RPC of type RESPONSE_NODE_ALIAS_ADDRS(3017) received srun: error: slurm_unpack_received_msg: [mylinux.wbi.com]:41242] Header lengths are longer than data received *mylinux.wbi.com <http://mylinux.wbi.com/>* Regards, MS

1 1

Slurmctld process error 'double free or corruption' on RHEL 9 (Rocky Linux)
by William VINCENT 07 Oct '24

07 Oct '24

Hello I am writing to report an issue with the Slurmctld process on our RHEL 9 (Rocky Linux) . Twice in the past 5 days, the Slurmctld process has encountered an error that resulted in the service stopping. The error message displayed was "double free or corruption (out)". This error has caused significant disruption to our jobs, and we are concerned about its recurrence. We have tried troubleshooting the issue, but we have not been able to identify the root cause of the problem. We would appreciate any assistance or guidance you can provide to help us resolve this issue. Please let us know if you need any additional information or if there are any specific steps we should take to diagnose the problem further. Thank you for your attention to this matter. Best regards, _________________________ Jul 09 22:12:01 admin slurmctld[711010]: double free or corruption (fasttop) Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Main process exited, code=killed, status=6/ABRT Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Failed with result 'signal'. Jul 09 22:12:01 admin systemd[1]: slurmctld.service: Consumed 11min 26.451s CPU time. ..... Jul 14 10:15:01 admin slurmctld[1633720]: double free or corruption (out) Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Main process exited, code=killed, status=6/ABRT Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Failed with result 'signal'. Jul 14 10:15:02 admin systemd[1]: slurmctld.service: Consumed 7min 27.596s CPU time. _________________________ slurmctld -V slurm 22.05.9 ________________________ cat /etc/slurm/slurm.conf |grep -v '#' ClusterName=xxx SlurmctldHost=admin SlurmctldParameters=enable_configless SlurmUser=slurm AuthType=auth/munge CryptoType=crypto/munge SlurmctldPort=6817 StateSaveLocation=/var/spool/slurmctld SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmctldDebug=verbose DebugFlags=NO_CONF_HASH SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmdLogFile=/var/log/slurm/slurmd.log SlurmdDebug=verbose SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core,CR_LLN DefMemPerCPU=1024 MaxMemPerCPU=4096 GresTypes=gpu ProctrackType=proctrack/cgroup JobAcctGatherType=jobacct_gather/cgroup JobAcctGatherFrequency=15 JobCompType=jobcomp/none TaskPlugin=task/cgroup LaunchParameters=use_interactive_step AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=admin AccountingStoragePort=6819 AccountingStorageEnforce=associations AccountingStorageTRES=gres/gpu MailProg=/usr/bin/mailx EnforcePartLimits=YES MaxArraySize=200000 MaxJobCount=500000 MpiDefault=none ReturnToService=2 SwitchType=switch/none TmpFS=/tmpslurm/ UsePAM=1 InactiveLimit=0 KillWait=30 MessageTimeout=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 PriorityType=priority/multifactor PriorityFlags=FAIR_TREE,MAX_TRES PriorityDecayHalfLife=1-0 PriorityWeightFairshare=10000 NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=3500 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=2 Sockets=2 RealMemory=1700 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=1700 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=xxx NodeHostname=xxx CPUs=4 Sockets=4 RealMemory=3500 TmpDisk=1 CoresPerSocket=1 ThreadsPerCore=1 State=DRAIN NodeName=r9nc-24-[1-12] NodeHostname=r9nc-24-[1-12] Sockets=2 CoresPerSocket=12 ThreadsPerCore=1 CPUs=24 RealMemory=180000 State=UNKNOWN NodeName=r9nc-48-[1-4] NodeHostname=r9nc-48-[1-4] Sockets=2 CoresPerSocket=24 ThreadsPerCore=1 CPUs=48 RealMemory=480000 State=UNKNOWN NodeName=r9ng-1080-[1-7] NodeHostname=r9ng-1080-[1-7] Sockets=2 CoresPerSocket=10 ThreadsPerCore=1 CPUs=20 RealMemory=180000 State=UNKNOWN Gres=gpu:1080ti:4 NodeName=r9ng-1080-8 NodeHostname=r9ng-1080-8 Sockets=2 CoresPerSocket=10 ThreadsPerCore=1 CPUs=20 RealMemory=176687 State=UNKNOWN Gres=gpu:1080ti:1 PartitionName=24CPUNodes Nodes=r9nc-24-[1-12] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=7500 DefMemPerCPU=7500 TRESBillingWeights="CPU=1.0,Mem=0.125G" Default=YES PartitionName=48CPUNodes Nodes=r9nc-48-[1-4] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=10000 DefMemPerCPU=8000 TRESBillingWeights="CPU=1.0,Mem=0.125G" PartitionName=GPUNodes Nodes=r9ng-1080-[1-7] State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=9000 DefMemPerCPU=9000 PartitionName=GPUNodes1080-dev Nodes=r9ng-1080-8 State=UP MaxTime=UNLIMITED OverSubscribe=NO MaxMemPerCPU=9000 DefMemPerCPU=9000 Hidden=Yes _________________________ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST 24CPUNodes* up infinite 12 idle r9nc-24-[1-12] 48CPUNodes up infinite 2 idle r9nc-48-[1-2] GPUNodes up infinite 4 idle r9ng-1080-[4-7] GPUNodes1080-dev up infinite 1 idle r9ng-1080-8 -- William VINCENT Administrateur systèmes et réseaux

4 15

license server redundancy
by Yves Kondoszek 03 Oct '24

03 Oct '24

Hello. I'm configuring a cluster that will run jobs that use FlexLM licenses. We have what I believe is a quite standard configuration of license servers - though I haven't found anything in the documentation or in the mailing list archives - that often provide the same features across several servers; for example, the license token "simulation" will be available at let's say 1717@server1, 1717@server2 and 4000@server3 . FlexLM clients support those through a path-style environment variable, typically LM_LICENSE_FILE=1717@server1:1717@server2:4000@server3 . Now, I have set up Slurm for remote dynamic licenses and update them live through "LastConsumed" from lmstat requests, so that Slurm is aware of some users checking them out externally (out of Slurm control). But my problem is that the documentation clearly states that: "When submitting jobs to remote licenses, the name and server must be used." So I have to specifically choose a server: $ sbatch -L simulation@server2 script.sh instead of telling Slurm to just use whatever license is available on any of the servers. Does any of you have any experience with redundant FlexLM licenses through several servers? Is there a way Slurm could support this?

1 0

GPU Accounting
by Emyr James 03 Oct '24

03 Oct '24

We have a node with 8 H100 GPUs that are split into MIG instances. We are using cgroups. This seems to work fine. Users can do something like sbatch --gres="gpu:1g.10gb:1"... and the job starts on the node with the gpus and cuda visible devices and the pytorch debug shows that the cgroup only gives them the gpu they asked for. In the accounting database, jobs in the job table always have the "gres_used" column be empty. I'd expect to see "gpu:1g.10gb:1" appearing for the job above. I have this set in slurm.conf AccountingStorageTRES=gres/gpu How can I see what gres was requested with the job ? At the moment I only see something like this in AllocTres billing=1,cpu=1,gres/gpu=1,mem=8G,node=1 and can't see any way to see what the specific MIG gpu asked for was. This is related to the email from Richard Lefebvre dated 7th June 2023 entitled "Billing/accounting for MIGs is not working". As far as I can see this got no replies. We are running slurm version 23.11.6. Regards, Emyr James Head of Scientific IT CRG - Centre for Genomic Regulation

2 1

SLUG Presentations Now Online!
by Victoria Hobson 01 Oct '24

01 Oct '24

Available presentations from this year's SLUG event are now online. They can be found at https://www.schedmd.com/publications/ We thank all those who presented and attended for a great event! -- Victoria Hobson SchedMD LLC Vice President of Marketing

1 0

Job Step State
by Emyr James 01 Oct '24

01 Oct '24

Dear all, I am working on a script to take completed job accounting data from the slurm accounting database and insert the equivalent data into a clickhouse table for fast reporting I can see that all the information is included in the cluster_job_table and cluster_job_step_table which seem to be joined on job_db_inx To get the cpu usage and peak memory usage etc. I can see that I need to parse the tres columns in the job steps. I couldn't find any column called MaxRSS in the database even though the sacct command prints this. I then found some data in tres_table and assume that sacct is using this. Please correct me if I'm wrong and if sacct is getting information from somwhere other than the accounting database? for the state column I get this... select state, count(*) as num from crg_step_table group by state order by num desc limit 10; +-------+--------+ | state | num | +-------+--------+ | 3 | 590635 | | 5 | 28345 | | 4 | 4401 | | 11 | 962 | | 1 | 8 | +-------+--------+ When I use sacct I see statuses seach as COMPLETED, OUT_OF_MEMORY etc. so there must be a mapping somewhere between these state ids and that text. Can someone prvide that mapping or point me to where it's defined in the database or in the code ? Many thanks, Emyr James Head of Scientific IT CRG - Centre for Genomic Regulation

2 2

Hardcoded CGroups v2 Slice
by Khalid Al-Hawaj 01 Oct '24

01 Oct '24

Hello, I am in the process of setting up SLURM to be used in a profiling cluster. The purpose of SLURM is to allow users to submit jobs to be profiled. The latency is a very important aspect of profiling the applications correctly. I was able to leverage cgroupsv2.0 to isolate user.slice from the cores that would be used by SLURM jobs. The issue is that slurmstepd shares the resources with system.slice; I was digging through the code, and I saw that the creation of the scope is here: https://github.com/SchedMD/slurm/blob/master/src/plugins/cgroup/v2/cgroup_v… And I noticed that the slice is hardcoded in the following line: https://github.com/SchedMD/slurm/blob/master/src/plugins/cgroup/v2/cgroup_v… So, my question, now, is about why is the slice hardcoded? What was the reason behind such a decision? I would have thought that the slice chosen would be set through cgroups.conf, instead. I would like to switch the slice for slurmstepd to a slice other than system.slice; by doing so, I would be able to isolate cores better by making sure that services' processes are isolated from the cores used for SLURM jobs. I can definitely change the defined value in the code and recompile. Are there anything to consider before doing so? Thanks, Khalid

1 0

Updated one compute node to Ubuntu 24.04 LTS, now it does not receive jobs
by Cristóbal Navarro 29 Sep '24

29 Sep '24

Dear community, I am having a strange issue I have been unable to find the cause. Last week I did a full update on the cluster, which is composed of the master node, and two compute nodes (nodeGPU01 -> DGXA100 and nodeGPU02 -> custom GPU server). After the update, I got - master node ended up with Ubuntu 24.04, - nodeGPU01 with latest DGX OS (still Ubuntu 22.04) - nodeGPU02 with Ubuntu 24.04 LTS. - Launching jobs from master choosing the partitions of nodeGPU01 works perfectly. - Launching jobs from master choosing the partition of nodeGPU02 stopped working (hangs). The nodeGPU02 (Ubuntu 24) is no longer processing jobs successfully, while the other nodeGPU01 works perfectly even when the master has Ubuntu 24. Any help is welcome, I have tried many things and had no success in finding the cause of this. Please let me know if you need more information. Many thanks in advance. This is the initial `slurmd` log of the problematic node (nodeGPU02), notice the messages in yellow ➜ ~ sudo systemctl status slurmd.service ● slurmd.service - Slurm node daemon Loaded: loaded (/etc/systemd/system/slurmd.service; enabled; preset: enabled) Active: active (running) since Sat 2024-09-28 14:00:22 -03; 4s ago Main PID: 4821 (slurmd) Tasks: 1 Memory: 17.0M (peak: 29.7M) CPU: 174ms CGroup: /system.slice/slurmd.service └─4821 /usr/sbin/slurmd -D -s Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug: MPI: Loading all types Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug: mpi/pmix_v5: init: PMIx plugin loaded Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug: mpi/pmix_v5: init: PMIx plugin loaded Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug2: No mpi.conf file (/etc/slurm/mpi.conf) Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: slurmd started on Sat, 28 Sep 2024 14:00:25 -0300 Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug: _step_connect: connect() failed for /var/spool/slurmd/slurmd/nodeGPU02_57436.0: Connection refused Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug2: health_check success rc:0 output: Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: CPUs=128 Boards=1 Sockets=2 Cores=64 Threads=1 Memory=773744 TmpDisk=899181 Uptime=2829 CPUSpecList=(null) FeaturesAvail=(nu> Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug: _step_connect: connect() failed for /var/spool/slurmd/slurmd/nodeGPU02_57436.0: Connection refused Sep 28 14:00:25 nodeGPU02 slurmd[4821]: slurmd: debug: _handle_node_reg_resp: slurmctld sent back 11 TRES This is the verbose output of the srun command (notice yellow messages). ➜ ~ srun -vvvp rtx hostname srun: defined options srun: -------------------- -------------------- srun: partition : rtx srun: verbose : 3 srun: -------------------- -------------------- srun: end of defined options srun: debug: propagating RLIMIT_CPU=18446744073709551615 srun: debug: propagating RLIMIT_FSIZE=18446744073709551615 srun: debug: propagating RLIMIT_DATA=18446744073709551615 srun: debug: propagating RLIMIT_STACK=8388608 srun: debug: propagating RLIMIT_CORE=0 srun: debug: propagating RLIMIT_RSS=18446744073709551615 srun: debug: propagating RLIMIT_NPROC=3090276 srun: debug: propagating RLIMIT_NOFILE=1024 srun: debug: propagating RLIMIT_MEMLOCK=18446744073709551615 srun: debug: propagating RLIMIT_AS=18446744073709551615 srun: debug: propagating SLURM_PRIO_PROCESS=0 srun: debug: propagating UMASK=0002 srun: debug: Entering slurm_allocation_msg_thr_create() srun: debug: port from net_stream_listen is 34081 srun: debug: Entering _msg_thr_internal srun: Waiting for resource configuration srun: Nodes nodeGPU02 are ready for job srun: jobid 57463: nodes(1):`nodeGPU02', cpu counts: 1(x1) srun: debug2: creating job with 1 tasks srun: debug2: cpu:1 is not a gres: srun: debug: requesting job 57463, user 99, nodes 1 including ((null)) srun: debug: cpus 1, tasks 1, name hostname, relative 65534 srun: CpuBindType=(null type) srun: debug: Entering slurm_step_launch srun: debug: mpi/pmix_v4: pmixp_abort_agent_start: (null) [0]: pmixp_agent.c:382: Abort agent port: 41393 srun: debug: mpi/pmix_v4: mpi_p_client_prelaunch: (null) [0]: mpi_pmix.c:285: setup process mapping in srun srun: debug: Entering _msg_thr_create() srun: debug: mpi/pmix_v4: _pmix_abort_thread: (null) [0]: pmixp_agent.c:353: Start abort thread srun: debug: initialized stdio listening socket, port 33223 srun: debug: Started IO server thread (140079189182144) srun: debug: Entering _launch_tasks srun: launching StepId=57463.0 on host nodeGPU02, 1 tasks: 0 srun: debug2: Called _file_readable srun: debug2: Called _file_writable srun: route/default: init: route default plugin loaded srun: debug2: Called _file_writable srun: topology/none: init: topology NONE plugin loaded srun: debug2: Tree head got back 0 looking for 1 srun: debug: slurm_recv_timeout at 0 of 4, timeout srun: error: slurm_receive_msgs: [[nodeGPU02]:6818] failed: Socket timed out on send/recv operation srun: debug2: Tree head got back 1 srun: debug: launch returned msg_rc=1001 err=5004 type=9001 srun: debug2: marking task 0 done on failed node 0 srun: error: Task launch for StepId=57463.0 failed on node nodeGPU02: Socket timed out on send/recv operation srun: error: Application launch failed: Socket timed out on send/recv operation srun: Job step aborted srun: debug2: false, shutdown srun: debug2: false, shutdown srun: debug2: Called _file_readable srun: debug2: Called _file_writable srun: debug2: Called _file_writable srun: debug2: false, shutdown srun: debug: IO thread exiting srun: debug: mpi/pmix_v4: _conn_readable: (null) [0]: pmixp_agent.c:105: false, shutdown srun: debug: mpi/pmix_v4: _pmix_abort_thread: (null) [0]: pmixp_agent.c:355: Abort thread exit srun: debug2: slurm_allocation_msg_thr_destroy: clearing up message thread srun: debug2: false, shutdown srun: debug: Leaving _msg_thr_internal srun: debug2: spank: spank_pyxis.so: exit = 0 This is the `tail -f` log of slurmctld when launching a simple `srun hostname` [2024-09-28T14:08:10.264] ==================== [2024-09-28T14:08:10.264] JobId=57463 nhosts:1 ncpus:1 node_req:1 nodes=nodeGPU02 [2024-09-28T14:08:10.264] Node[0]: [2024-09-28T14:08:10.264] Mem(MB):65536:0 Sockets:2 Cores:64 CPUs:1:0 [2024-09-28T14:08:10.264] Socket[0] Core[0] is allocated [2024-09-28T14:08:10.264] -------------------- [2024-09-28T14:08:10.264] cpu_array_value[0]:1 reps:1 [2024-09-28T14:08:10.264] ==================== [2024-09-28T14:08:10.264] gres/gpu: state for nodeGPU02 [2024-09-28T14:08:10.264] gres_cnt found:3 configured:3 avail:3 alloc:0 [2024-09-28T14:08:10.264] gres_bit_alloc: of 3 [2024-09-28T14:08:10.264] gres_used:(null) [2024-09-28T14:08:10.264] topo[0]:(null)(0) [2024-09-28T14:08:10.264] topo_core_bitmap[0]:0-63 of 128 [2024-09-28T14:08:10.264] topo_gres_bitmap[0]:0 of 3 [2024-09-28T14:08:10.264] topo_gres_cnt_alloc[0]:0 [2024-09-28T14:08:10.264] topo_gres_cnt_avail[0]:1 [2024-09-28T14:08:10.264] topo[1]:(null)(0) [2024-09-28T14:08:10.264] topo_core_bitmap[1]:0-63 of 128 [2024-09-28T14:08:10.264] topo_gres_bitmap[1]:1 of 3 [2024-09-28T14:08:10.264] topo_gres_cnt_alloc[1]:0 [2024-09-28T14:08:10.264] topo_gres_cnt_avail[1]:1 [2024-09-28T14:08:10.264] topo[2]:(null)(0) [2024-09-28T14:08:10.264] topo_core_bitmap[2]:0-63 of 128 [2024-09-28T14:08:10.264] topo_gres_bitmap[2]:2 of 3 [2024-09-28T14:08:10.264] topo_gres_cnt_alloc[2]:0 [2024-09-28T14:08:10.264] topo_gres_cnt_avail[2]:1 [2024-09-28T14:08:10.265] sched: _slurm_rpc_allocate_resources JobId=57463 NodeList=nodeGPU02 usec=1339 [2024-09-28T14:08:10.368] ==================== [2024-09-28T14:08:10.368] JobId=57463 StepId=0 [2024-09-28T14:08:10.368] JobNode[0] Socket[0] Core[0] is allocated [2024-09-28T14:08:10.368] ==================== [2024-09-28T14:08:30.409] _job_complete: JobId=57463 WTERMSIG 12 [2024-09-28T14:08:30.410] gres/gpu: state for nodeGPU02 [2024-09-28T14:08:30.410] gres_cnt found:3 configured:3 avail:3 alloc:0 [2024-09-28T14:08:30.410] gres_bit_alloc: of 3 [2024-09-28T14:08:30.410] gres_used:(null) [2024-09-28T14:08:30.410] topo[0]:(null)(0) [2024-09-28T14:08:30.410] topo_core_bitmap[0]:0-63 of 128 [2024-09-28T14:08:30.410] topo_gres_bitmap[0]:0 of 3 [2024-09-28T14:08:30.410] topo_gres_cnt_alloc[0]:0 [2024-09-28T14:08:30.410] topo_gres_cnt_avail[0]:1 [2024-09-28T14:08:30.410] topo[1]:(null)(0) [2024-09-28T14:08:30.410] topo_core_bitmap[1]:0-63 of 128 [2024-09-28T14:08:30.410] topo_gres_bitmap[1]:1 of 3 [2024-09-28T14:08:30.410] topo_gres_cnt_alloc[1]:0 [2024-09-28T14:08:30.410] topo_gres_cnt_avail[1]:1 [2024-09-28T14:08:30.410] topo[2]:(null)(0) [2024-09-28T14:08:30.410] topo_core_bitmap[2]:0-63 of 128 [2024-09-28T14:08:30.410] topo_gres_bitmap[2]:2 of 3 [2024-09-28T14:08:30.410] topo_gres_cnt_alloc[2]:0 [2024-09-28T14:08:30.410] topo_gres_cnt_avail[2]:1 [2024-09-28T14:08:30.410] _job_complete: JobId=57463 done [2024-09-28T14:08:58.687] gres/gpu: state for nodeGPU01 [2024-09-28T14:08:58.687] gres_cnt found:8 configured:8 avail:8 alloc:0 [2024-09-28T14:08:58.687] gres_bit_alloc: of 8 [2024-09-28T14:08:58.687] gres_used:(null) [2024-09-28T14:08:58.687] topo[0]:A100(808464705) [2024-09-28T14:08:58.687] topo_core_bitmap[0]:48-63 of 128 [2024-09-28T14:08:58.687] topo_gres_bitmap[0]:0 of 8 [2024-09-28T14:08:58.687] topo_gres_cnt_alloc[0]:0 [2024-09-28T14:08:58.687] topo_gres_cnt_avail[0]:1 [2024-09-28T14:08:58.687] topo[1]:A100(808464705) [2024-09-28T14:08:58.687] topo_core_bitmap[1]:48-63 of 128 [2024-09-28T14:08:58.687] topo_gres_bitmap[1]:1 of 8 [2024-09-28T14:08:58.687] topo_gres_cnt_alloc[1]:0 [2024-09-28T14:08:58.687] topo_gres_cnt_avail[1]:1 [2024-09-28T14:08:58.687] topo[2]:A100(808464705) [2024-09-28T14:08:58.687] topo_core_bitmap[2]:16-31 of 128 [2024-09-28T14:08:58.687] topo_gres_bitmap[2]:2 of 8 [2024-09-28T14:08:58.687] topo_gres_cnt_alloc[2]:0 [2024-09-28T14:08:58.687] topo_gres_cnt_avail[2]:1 [2024-09-28T14:08:58.687] topo[3]:A100(808464705) [2024-09-28T14:08:58.687] topo_core_bitmap[3]:16-31 of 128 [2024-09-28T14:08:58.688] topo_gres_bitmap[3]:3 of 8 [2024-09-28T14:08:58.688] topo_gres_cnt_alloc[3]:0 [2024-09-28T14:08:58.688] topo_gres_cnt_avail[3]:1 [2024-09-28T14:08:58.688] topo[4]:A100(808464705) [2024-09-28T14:08:58.688] topo_core_bitmap[4]:112-127 of 128 [2024-09-28T14:08:58.688] topo_gres_bitmap[4]:4 of 8 [2024-09-28T14:08:58.688] topo_gres_cnt_alloc[4]:0 [2024-09-28T14:08:58.688] topo_gres_cnt_avail[4]:1 [2024-09-28T14:08:58.688] topo[5]:A100(808464705) [2024-09-28T14:08:58.688] topo_core_bitmap[5]:112-127 of 128 [2024-09-28T14:08:58.688] topo_gres_bitmap[5]:5 of 8 [2024-09-28T14:08:58.688] topo_gres_cnt_alloc[5]:0 [2024-09-28T14:08:58.688] topo_gres_cnt_avail[5]:1 [2024-09-28T14:08:58.688] topo[6]:A100(808464705) [2024-09-28T14:08:58.688] topo_core_bitmap[6]:80-95 of 128 [2024-09-28T14:08:58.688] topo_gres_bitmap[6]:6 of 8 [2024-09-28T14:08:58.688] topo_gres_cnt_alloc[6]:0 [2024-09-28T14:08:58.688] topo_gres_cnt_avail[6]:1 [2024-09-28T14:08:58.688] topo[7]:A100(808464705) [2024-09-28T14:08:58.688] topo_core_bitmap[7]:80-95 of 128 [2024-09-28T14:08:58.688] topo_gres_bitmap[7]:7 of 8 [2024-09-28T14:08:58.688] topo_gres_cnt_alloc[7]:0 [2024-09-28T14:08:58.688] topo_gres_cnt_avail[7]:1 [2024-09-28T14:08:58.688] type[0]:A100(808464705) [2024-09-28T14:08:58.688] type_cnt_alloc[0]:0 [2024-09-28T14:08:58.688] type_cnt_avail[0]:8 [2024-09-28T14:08:58.690] gres/gpu: state for nodeGPU02 [2024-09-28T14:08:58.690] gres_cnt found:3 configured:3 avail:3 alloc:0 [2024-09-28T14:08:58.690] gres_bit_alloc: of 3 [2024-09-28T14:08:58.690] gres_used:(null) [2024-09-28T14:08:58.690] topo[0]:(null)(0) [2024-09-28T14:08:58.690] topo_core_bitmap[0]:0-63 of 128 [2024-09-28T14:08:58.690] topo_gres_bitmap[0]:0 of 3 [2024-09-28T14:08:58.690] topo_gres_cnt_alloc[0]:0 [2024-09-28T14:08:58.690] topo_gres_cnt_avail[0]:1 [2024-09-28T14:08:58.690] topo[1]:(null)(0) [2024-09-28T14:08:58.690] topo_core_bitmap[1]:0-63 of 128 [2024-09-28T14:08:58.690] topo_gres_bitmap[1]:1 of 3 [2024-09-28T14:08:58.690] topo_gres_cnt_alloc[1]:0 [2024-09-28T14:08:58.690] topo_gres_cnt_avail[1]:1 [2024-09-28T14:08:58.690] topo[2]:(null)(0) [2024-09-28T14:08:58.690] topo_core_bitmap[2]:0-63 of 128 [2024-09-28T14:08:58.690] topo_gres_bitmap[2]:2 of 3 [2024-09-28T14:08:58.690] topo_gres_cnt_alloc[2]:0 [2024-09-28T14:08:58.690] topo_gres_cnt_avail[2]:1 [2024-09-28T14:09:49.763] Resending TERMINATE_JOB request JobId=57463 Nodelist=nodeGPU02 This is the `tail -f` log of slurmd when launching the job from master, notice the messages in yellow [2024-09-28T14:08:10.270] debug2: Processing RPC: REQUEST_LAUNCH_PROLOG [2024-09-28T14:08:10.321] debug2: prep/script: _run_subpath_command: prolog success rc:0 output: [2024-09-28T14:08:10.323] debug2: Finish processing RPC: REQUEST_LAUNCH_PROLOG [2024-09-28T14:08:10.377] debug: Checking credential with 720 bytes of sig data [2024-09-28T14:08:10.377] debug2: Start processing RPC: REQUEST_LAUNCH_TASKS [2024-09-28T14:08:10.377] debug2: Processing RPC: REQUEST_LAUNCH_TASKS [2024-09-28T14:08:10.377] launch task StepId=57463.0 request from UID:10082 GID:10088 HOST:10.10.0.1 PORT:36478 [2024-09-28T14:08:10.377] CPU_BIND: JobNode[0] CPU[0] Step alloc [2024-09-28T14:08:10.377] CPU_BIND: ==================== [2024-09-28T14:08:10.377] CPU_BIND: Memory extracted from credential for StepId=57463.0 job_mem_limit=65536 step_mem_limit=65536 [2024-09-28T14:08:10.377] debug: Waiting for job 57463's prolog to complete [2024-09-28T14:08:10.377] debug: Finished wait for job 57463's prolog to complete [2024-09-28T14:08:10.378] error: _send_slurmstepd_init failed [2024-09-28T14:08:10.384] debug2: debug level read from slurmd is 'debug2'. [2024-09-28T14:08:10.385] debug2: _read_slurmd_conf_lite: slurmd sent 11 TRES. [2024-09-28T14:08:10.385] debug2: Received CPU frequency information for 128 CPUs [2024-09-28T14:08:10.385] select/cons_tres: common_init: select/cons_tres loaded [2024-09-28T14:08:10.385] debug: switch/none: init: switch NONE plugin loaded [2024-09-28T14:08:10.385] [57463.0] debug: auth/munge: init: loaded [2024-09-28T14:08:10.385] [57463.0] debug: Reading cgroup.conf file /etc/slurm/cgroup.conf [2024-09-28T14:08:10.395] [57463.0] debug: cgroup/v2: init: Cgroup v2 plugin loaded [2024-09-28T14:08:10.396] [57463.0] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded [2024-09-28T14:08:10.396] [57463.0] debug: acct_gather_energy/none: init: AcctGatherEnergy NONE plugin loaded [2024-09-28T14:08:10.396] [57463.0] debug: acct_gather_profile/none: init: AcctGatherProfile NONE plugin loaded [2024-09-28T14:08:10.396] [57463.0] debug: acct_gather_interconnect/none: init: AcctGatherInterconnect NONE plugin loaded [2024-09-28T14:08:10.396] [57463.0] debug: acct_gather_filesystem/none: init: AcctGatherFilesystem NONE plugin loaded [2024-09-28T14:08:10.396] [57463.0] debug2: Reading acct_gather.conf file /etc/slurm/acct_gather.conf [2024-09-28T14:08:10.396] [57463.0] debug2: hwloc_topology_init [2024-09-28T14:08:10.399] [57463.0] debug2: xcpuinfo_hwloc_topo_load: xml file (/var/spool/slurmd/slurmd/hwloc_topo_whole.xml) found [2024-09-28T14:08:10.400] [57463.0] debug: CPUs:128 Boards:1 Sockets:2 CoresPerSocket:64 ThreadsPerCore:1 [2024-09-28T14:08:10.401] [57463.0] debug: task/cgroup: init: core enforcement enabled [2024-09-28T14:08:10.401] [57463.0] debug: task/cgroup: task_cgroup_memory_init: task/cgroup/memory: TotCfgRealMem:773744M allowed:100%(enforced), swap:0%(enforced), max:100%(773744M) max+swap:0%(773744M) min:30M kmem:100%(773744M permissive) min:30M [2024-09-28T14:08:10.401] [57463.0] debug: task/cgroup: init: memory enforcement enabled [2024-09-28T14:08:10.401] [57463.0] debug: task/cgroup: init: device enforcement enabled [2024-09-28T14:08:10.401] [57463.0] debug: task/cgroup: init: Tasks containment cgroup plugin loaded [2024-09-28T14:08:10.401] [57463.0] debug: jobacct_gather/linux: init: Job accounting gather LINUX plugin loaded [2024-09-28T14:08:10.401] [57463.0] cred/munge: init: Munge credential signature plugin loaded [2024-09-28T14:08:10.401] [57463.0] debug: job_container/none: init: job_container none plugin loaded [2024-09-28T14:08:10.401] [57463.0] debug: gres/gpu: init: loaded [2024-09-28T14:08:10.401] [57463.0] debug: gpu/generic: init: init: GPU Generic plugin loaded [2024-09-28T14:08:30.415] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-09-28T14:08:30.415] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-09-28T14:08:30.415] debug: _rpc_terminate_job: uid = 777 JobId=57463 [2024-09-28T14:08:30.415] debug: credential for job 57463 revoked [2024-09-28T14:08:30.415] debug: sent SUCCESS, waiting for step to start [2024-09-28T14:08:30.415] debug: Blocked waiting for JobId=57463, all steps [2024-09-28T14:08:58.688] debug2: Start processing RPC: REQUEST_NODE_REGISTRATION_STATUS [2024-09-28T14:08:58.689] debug2: Processing RPC: REQUEST_NODE_REGISTRATION_STATUS [2024-09-28T14:08:58.689] debug: _step_connect: connect() failed for /var/spool/slurmd/slurmd/nodeGPU02_57436.0: Connection refused [2024-09-28T14:08:58.692] debug: _handle_node_reg_resp: slurmctld sent back 11 TRES. [2024-09-28T14:08:58.692] debug2: Finish processing RPC: REQUEST_NODE_REGISTRATION_STATUS -- Cristóbal A. Navarro

1 1

errors compiling Slurm 18 on RHEL 9: [Makefile:577: scancel] Error 1 & It's not recommended to have unversioned Obsoletes
by Robert Kudyba 27 Sep '24

27 Sep '24

We're in the process of upgrading but first we're moving to RHEL 9. My attempt to compile using rpmbuild -v -ta --define "_lto_cflags %{nil}" slurm-18.08.9.tar.bz2 (H/T to Brian for this flag <https://groups.google.com/g/slurm-users/c/W8YfGIn1rDI/m/4bsSAoqZAAAJ>). I've stumped Google and the Slurm mailing list with the scancel error so hoping someone here knows of a work around. /bin/ld: opt.o:/root/rpmbuild/BUILD/slurm-18.08.9/src/scancel/../../src/scancel/scancel.h:78: multiple definition of `opt'; scancel.o:/root/rpmbuild/BUILD/slurm-18.08.9/src/scancel/../../src/scancel/scancel.h:78: first defined here collect2: error: ld returned 1 exit status make[3]: *** [Makefile:577: scancel] Error 1 make[3]: Leaving directory '/root/rpmbuild/BUILD/slurm-18.08.9/src/scancel' make[2]: *** [Makefile:563: all-recursive] Error 1 make[2]: Leaving directory '/root/rpmbuild/BUILD/slurm-18.08.9/src' make[1]: *** [Makefile:690: all-recursive] Error 1 make[1]: Leaving directory '/root/rpmbuild/BUILD/slurm-18.08.9' make: *** [Makefile:589: all] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.jhiGyR (%build) RPM build errors: Macro expanded in comment on line 22: %_prefix path install path for commands, libraries, etc. line 70: It's not recommended to have unversioned Obsoletes: Obsoletes: slurm-lua slurm-munge slurm-plugins Macro expanded in comment on line 158: %define _unpackaged_files_terminate_build 0 line 224: It's not recommended to have unversioned Obsoletes: Obsoletes: slurm-sql line 256: It's not recommended to have unversioned Obsoletes: Obsoletes: slurm-sjobexit slurm-sjstat slurm-seff line 275: It's not recommended to have unversioned Obsoletes: Obsoletes: pam_slurm Bad exit status from /var/tmp/rpm-tmp.jhiGyR (%build) #!/bin/sh RPM_SOURCE_DIR="/root" RPM_BUILD_DIR="/root/rpmbuild/BUILD" RPM_OPT_FLAGS="-O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection" RPM_LD_FLAGS="-Wl,-z,relro -Wl,--as-needed "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 " RPM_ARCH="x86_64" RPM_OS="linux" RPM_BUILD_NCPUS="48" export RPM_SOURCE_DIR RPM_BUILD_DIR RPM_OPT_FLAGS RPM_LD_FLAGS RPM_ARCH RPM_OS RPM_BUILD_NCPUS RPM_LD_FLAGS RPM_DOC_DIR="/usr/share/doc" export RPM_DOC_DIR RPM_PACKAGE_NAME="slurm" RPM_PACKAGE_VERSION="18.08.9" RPM_PACKAGE_RELEASE="1.el9" export RPM_PACKAGE_NAME RPM_PACKAGE_VERSION RPM_PACKAGE_RELEASE LANG=C export LANG unset CDPATH DISPLAY ||: RPM_BUILD_ROOT="/root/rpmbuild/BUILDROOT/slurm-18.08.9-1.el9.x86_64" export RPM_BUILD_ROOT PKG_CONFIG_PATH="${PKG_CONFIG_PATH}:/usr/lib64/pkgconfig:/usr/share/pkgconfig" export PKG_CONFIG_PATH CONFIG_SITE=${CONFIG_SITE:-NONE} export CONFIG_SITE set -x umask 022 cd "/root/rpmbuild/BUILD" cd 'slurm-18.08.9' CFLAGS="${CFLAGS:--O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection}" ; export CFLAGS ; CXXFLAGS="${CXXFLAGS:--O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection}" ; export CXXFLAGS ; FFLAGS="${FFLAGS:--O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules}" ; export FFLAGS ; FCFLAGS="${FCFLAGS:--O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules}" ; export FCFLAGS ; LDFLAGS="${LDFLAGS:--Wl,-z,relro -Wl,--as-needed "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 }" ; export LDFLAGS ; LT_SYS_LIBRARY_PATH="${LT_SYS_LIBRARY_PATH:-/usr/lib64:}" ; export LT_SYS_LIBRARY_PATH ; CC="${CC:-gcc}" ; export CC ; CXX="${CXX:-g++}" ; export CXX; [ ""x != x ] && for file in $(find . -type f -name configure -print); do /usr/bin/sed -r --in-place=.backup 's/^char $\*f$  = /__attribute__ ((used)) char (*f) () = /g' $file; diff -u $file.backup $file && mv $file.backup $file /usr/bin/sed -r --in-place=.backup 's/^char $\*f$ ;/__attribute__ ((used)) char (*f) ();/g' $file; diff -u $file.backup $file && mv $file.backup $file /usr/bin/sed -r --in-place=.backup 's/^char \$2 ;/__attribute__ ((used)) char \$2 ();/g' $file; diff -u $file.backup $file && mv $file.backup $file /usr/bin/sed --in-place=.backup '1{$!N;$!N};$!N;s/int x = 1;\nint y = 0;\nint z;\nint nan;/volatile int x = 1; volatile int y = 0; volatile int z, nan;/;P;D' $file; diff -u $file.backup $file && mv $file.backup $file /usr/bin/sed --in-place=.backup 's#^lt_cv_sys_global_symbol_to_cdecl=.*#lt_cv_sys_global_symbol_to_cdecl="sed -n -e '"'"'s/^T .* \$.*\$$/extern int \\1();/p'"'"' -e '"'"'s/^$symcode* .* \$.*\$$/extern char \\1;/p'"'"'"#' $file; diff -u $file.backup $file && mv $file.backup $file done; [ "1" = 1 ] && for i in $(find $(dirname ./configure) -name config.guess -o -name config.sub) ; do [ -f /usr/lib/rpm/redhat/$(basename $i) ] && /usr/bin/rm -f $i && /usr/bin/cp -fv /usr/lib/rpm/redhat/$(basename $i) $i ; done ; [ "1" = 1 ] && [ x != "x"-Wl,-z,lazy"" ] && for i in $(find . -name ltmain.sh) ; do /usr/bin/sed -i.backup -e 's~compiler_flags=$~compiler_flags=""-Wl,-z,lazy""~' $i done ; ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu \ --program-prefix= \ --disable-dependency-tracking \ \ --prefix=/usr \ --exec-prefix=/usr \ --bindir=/usr/bin \ --sbindir=/usr/sbin \ --sysconfdir=/etc/slurm \ --datadir=/usr/share \ --includedir=/usr/include \ --libdir=/usr/lib64 \ --libexecdir=/usr/libexec \ --localstatedir=/var \ --sharedstatedir=/var/lib \ --mandir=/usr/share/man \ --infodir=/usr/share/info \ \ make -j48 RPM_EC=$? for pid in $(jobs -p); do kill -9 ${pid} || continue; done exit ${RPM_EC}

2 2

Re: SLUG'24 presentation slides?
by Kilian Cavalotti 27 Sep '24

27 Sep '24

Awesome, thanks Victoria! Cheers, -- Kilian On Thu, Sep 26, 2024 at 11:17 AM Victoria Hobson <victoria(a)schedmd.com> wrote: > Hi Kilian, > > We're getting these posted now and an email will go out when they are > available! > > Thanks, > > > Victoria Hobson > > *Vice President of Marketing * > > 909.609.8889 > > www.schedmd.com > > > On Mon, Sep 23, 2024 at 10:49 AM Kilian Cavalotti via slurm-users < > slurm-users(a)lists.schedmd.com> wrote: > >> Hi SchedMD, >> >> I'm sure they will eventually, but do you know when the slides of the >> SLUG'24 presentation will be available online at >> https://slurm.schedmd.com/publications.html, like previous editions'? >> >> Thanks! >> -- >> Kilian >> >> -- >> slurm-users mailing list -- slurm-users(a)lists.schedmd.com >> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com >> > -- Kilian

1 0

A note on updating Slurm from 23.02 to 24.05 & multi-cluster
by Ward Poelmans 26 Sep '24

26 Sep '24

Hi all, We hit a snag when updating our clusters from Slurm 23.02 to 24.05. After updating the slurmdbd, our multi cluster setup was broken until everything was updated to 24.05. We had not anticipated this. SchedMD says that fixing it would be a very complex operation. Hence, this warning to everybody on planning to update: make sure to quickly updating everything once you've updated the slurmdbd daemon. Reference: https://support.schedmd.com/show_bug.cgi?id=20931 Ward

3 3

Jobs pending with reason "priority" but nodes are idle
by Long, Daniel S. 25 Sep '24

25 Sep '24

Hi, On our cluster we have some jobs that are queued even though there are available nodes to run on. The listed reason is "priority" but that doesn't really make sense to me. Slurm isn't picking another job to run on those nodes; it's just not running anything at all. We do have a quite heterogeneous cluster, but as far as I can tell the queued jobs aren't requesting anything that would preclude them from running on the idle nodes. They are array jobs, if that makes a difference. Thanks for any help you all can provide.

3 6

Max TRES per user and node
by Guillaume COCHARD 25 Sep '24

25 Sep '24

Hello, We are looking for a method to limit the TRES used by each user on a per-node basis. For example, we would like to limit the total memory allocation of jobs from a user to 200G per node. There is MaxTRESperNode (https://slurm.schedmd.com/sacctmgr.html#OPT_MaxTRESPerNode), but unfortunately, this is a per-job limit, not per user. Ideally, we would like to apply this limit on partitions and/or QoS. Does anyone know if this is possible and how to achieve it? Thank you,

4 9

SLURM Telegraf Plugin
by Pablo Collado Soto 25 Sep '24

25 Sep '24

Hi all, I recently wrote an SLURM input plugin [0] for Telegraf [1]. I just wanted to let the community know so that you can use it if you'd find that useful. Maybe its existence can also be included in the documentation somewhere? Anyway, thanks a ton for your time, Pablo Collado Soto References: 0: https://github.com/influxdata/telegraf/tree/master/plugins/inputs/slurm 1: https://www.influxdata.com/time-series-platform/telegraf/ + -------------------------------------- + | Never let your sense of morals prevent | | you from doing what is right. | | -- Salvor Hardin, "Foundation" | + -------------------------------------- +

2 1

Setting up fairshare accounting
by tluchko 24 Sep '24

24 Sep '24

Hello, We have a new cluster and I'm trying to setup fairshare accounting. I'm trying to track CPU, MEM and GPU. It seems that billing for individual jobs is correct, but billing isn't being accumulated (TRESRunMin is always 0). In my slurm.conf, I think the relevant lines are AccountingStorageType=accounting_storage/slurmdbd AccountingStorageTRES=gres/gpu PriorityFlags=MAX_TRES PartitionName=gpu Nodes=node[1-7] MaxCPUsPerNode=384 MaxTime=7-0:00:00 State=UP TRESBillingWeights="CPU=1.0,MEM=0.125G,GRES/gpu=9.6" PartitionName=cpu Nodes=node[1-7] MaxCPUsPerNode=182 MaxTime=7-0:00:00 State=UP TRESBillingWeights="CPU=1.0,MEM=0.125G,GRES/gpu=9.6" I currently have one recently finished job and one running job. sacct gives $ sacct --format=JobID,JobName,ReqTRES%50,AllocTRES%50,TRESUsageInAve%50,TRESUsageInMax%50 JobID JobName ReqTRES AllocTRES TRESUsageInAve TRESUsageInMax ------------ ---------- -------------------------------------------------- -------------------------------------------------- -------------------------------------------------- -------------------------------------------------- 154 interacti+ billing=9,cpu=1,gres/gpu=1,mem=1G,node=1 billing=9,cpu=2,gres/gpu=1,mem=2G,node=1 154.interac+ interacti+ cpu=2,gres/gpu=1,mem=2G,node=1 cpu=00:00:00,energy=0,fs/disk=2480503,mem=3M,page+ cpu=00:00:00,energy=0,fs/disk=2480503,mem=3M,page+ 155 interacti+ billing=9,cpu=1,gres/gpu=1,mem=1G,node=1 billing=9,cpu=2,gres/gpu=1,mem=2G,node=1155.interac+ interacti+ cpu=2,gres/gpu=1,mem=2G,node=1 billing=9 seems correct to me, since I have 1 GPU allocated, which has the largest score of 9.6. However, sshare doesn't show anything in TRESRunMins sshare --format=Account,User,RawShares,FairShare,RawUsage,EffectvUsage,TRESRunMins%110 Account User RawShares FairShare RawUsage EffectvUsage TRESRunMins -------------------- ---------- ---------- ---------- ----------- ------------- -------------------------------------------------------------------------------------------------------------- root 21589714 1.000000 cpu=0,mem=0,energy=0,node=0,billing=0,fs/disk=0,vmem=0,pages=0,gres/gpu=0,gres/gpumem=0,gres/gpuutil=0 abrol_group 2000 0 0.000000 cpu=0,mem=0,energy=0,node=0,billing=0,fs/disk=0,vmem=0,pages=0,gres/gpu=0,gres/gpumem=0,gres/gpuutil=0 luchko_group 2000 21589714 1.000000 cpu=0,mem=0,energy=0,node=0,billing=0,fs/disk=0,vmem=0,pages=0,gres/gpu=0,gres/gpumem=0,gres/gpuutil=0 luchko_group tluchko 1 0.333333 21589714 1.000000 cpu=0,mem=0,energy=0,node=0,billing=0,fs/disk=0,vmem=0,pages=0,gres/gpu=0,gres/gpumem=0,gres/gpuutil=0 Why is TRESRunMin all 0 but RawUsage is not for tluchko? I have checked and slurmdbd is running. Thank you, Tyler Sent with [Proton Mail](https://proton.me/) secure email.

1 2

SLUG'24 presentation slides?
by Kilian Cavalotti 23 Sep '24

23 Sep '24

Hi SchedMD, I'm sure they will eventually, but do you know when the slides of the SLUG'24 presentation will be available online at https://slurm.schedmd.com/publications.html, like previous editions'? Thanks! -- Kilian

1 0

what updates NODEADDR
by Jakub Szarlat 21 Sep '24

21 Sep '24

Hi I'm using dynamic nodes with "slurmd -Z" with slurm 23.11.1. Firstly I find that when you do "scontrol show node" it shows the NODEADDR as ip rather than the NODENAME. Because I'm playing around with running this in containers on docker swarm I find this ip can be wrong. I can force it with scontrol update however after a while something updates it to something else again. Does anybody know if this is done by slurmd or slurmctld or something else? How can I stop this from happening? How can I get the node to register with the hostname rather than ip? cheers, Jakub

2 1

SLURM GRES reservation not working properly on 24.05.1
by Minulakshmi S 20 Sep '24

20 Sep '24

Hello, *Issue 1:* I am using slurm version 24.05.1 , my slurmd has a single node where I connect multiple gres by enabling the overscribe feature. I am able to use the advance reservation of gres only using *gres** name* (tres=gres/gpu:*SYSTEM12*). i.e while in reservation period , if other users submits job with SYSTEM12 , then slurm places this job in queue *user1@host$ srun --gres=gpu:SYSTEM12:1 hostname* *srun: job 333 queued and waiting for resources * but when other users just submit a job without any system name , slurm jobs goes through on that gres immediately even though it is reserved. *user1@host$ srun --gres=gpu:1 hostname * *mylinux.wbi.com <http://mylinux.wbi.com> * Also I can see GresUsed in busy mode using "*scontrol show node -d*" , this means the job is running on Gres/GPU and not on cpu etc. Same way , job submission based on Feature "rev1 in my case" is also going through even though it is reserved for other users in multiple partition slurm. *snippet of slurm.conf file* NodeName=cluster01 NodeAddr=cluster Port=6002CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2 Feature="rev1" Gres=gpu:SYSTEM12:1 RealMemory=64171 State=IDLE *Issue 2:* while execution , Slurm o/p's some extra prints in the srun output user1@host$ srun --gres=gpu:1 hostname srun: error: extract_net_cred: net_cred not provided srun: error: Malformed RPC of type RESPONSE_NODE_ALIAS_ADDRS(3017) received srun: error: slurm_unpack_received_msg: [[inv1715771615.nxdi.us-aus01.nxp.com]:41242] Header lengths are longer than data received *mylinux.wbi.com <http://mylinux.wbi.com>* Regards, MS

1 0

Can't schedule on cloud node: State=IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING
by Xaver Stiensmeier 20 Sep '24

20 Sep '24

Dear slurm-user list, I have a cloud node that is powered up and down on demand. Rarely it can happen that slurm's resumeTimeout is reached and the node is therefore powered down. We have set ReturnToService=2 in order to avoid the node being marked down, because the instance behind that node is created on demand and therefore after a failure nothing stops the system to start the node again as it is a different instance. I thought this would be enough, but apparently the node is still marked with "NOT_RESPONDING" which leads to slurm not trying to schedule on it. After a while NOT_RESPONDING is removed, but I would like to move it directly from within my fail script if possible so that the node can return to service immediately and not be blocked by "NOT_RESPONDING". Best regards, Xaver

2 3

SlurmDBD errors
by Sajesh Singh 19 Sep '24

19 Sep '24

OS: CentOS 8.5 Slurm: 22.05 Recently upgraded to 22.05. Upgrade was successful, but after a while I started to see the following messages in the slurmdbd.log file: error: We have more time than is possible (9344745+7524000+0)(16868745) > 12362400 for cluster CLUSTERNAME(3434) from 2024-09-18T13:00:00 - 2024-09-18T14:00:00 tres 1 (this may happen if oversubscription of resources is allowed without Gang) We do have partitions with overlapping nodes, but do not have "Suspend,Gang" set as the global PreemptMode mode. It is currently set to requeue. I have also check sacct and there are no runaway jobs listed. Oversubscription is not enabled on any of the queues as well. Do I need to modify my slurm config to address or is this an error condition caused by the upgrade? Thank you, SS

2 2

Change a job from --exclusive to --exclusive=user
by Gerhard Strangar 18 Sep '24

18 Sep '24

Hello, is it possible to change a pending job from --exclusive to --exclusive=user? I tried scontrol update jobid=... oversubscribe=user, but it seems to only accept yes or no. Gerhard

1 0

Feature request: Max Jobs Per Minute
by Ransom, Geoffrey M. 16 Sep '24

16 Sep '24

Hello We have another batch of new users and some more batches of large array jobs with very short runtimes due to errors in the jobs or just by design. Trying to deal with these issues, Setting ArrayTaskThrottle and user education, I had a thought that it would be very nice to have a limit on how many jobs can start in a given minute for users, so if they posted a 200000 array job with 15 second tasks then the scheduler wouldn't launch more than a 100 or 200 per minute and be less likely to bog down, but if they had longer runtimes (1 hour +) it would take a few extra minutes to start using all the resources they are allowed to but not add much overall delay to the whole set of jobs. I thought about adding something to our CLI filter, but usually these jobs are asking for a runtime of 3-4 hours even though they run for <30 seconds so the submit options don't indicate the problem jobs ahead of time. We currently limit our users to %80 of the available resources which is way more than slurm needs to bog down with fast turnover jobs, but we have users who complain that they can't use that other 20% when the cluster is not busy so putting in lower default restrictions is not currently an option. Has this already been discussed and isn't feasible for technical reasons? (Not finding anything like this yet searching the archives) I think slurm used have a feature request severity on their bug submission site. Is there a severity level they prefer to have suggested requests like this? Thanks

2 2

Detailed locations for SLUG'24
by Bjørn-Helge Mevik 10 Sep '24

10 Sep '24

Dear all SLUG attendees! The information about which buildings/addresses the SLUG reception and presentations are to be held is not very visible on the https://slug24.splashthat.com. There is a map there with all locations (https://www.google.com/maps/d/u/0/edit?mid=1bcGaTiW0TNB5noQsjQ3ulctzKuqlGrQ…), but I've gotten questions about it, so: The reception on Wednesday will be held in the top floor of Oslo Science Park (Forskningsparken). Address: Gaustadalléen 21. There will be someone in the reception who can point you in the right direction. The presentations will be held in auditorium 3 in Helga Engs Hus ("Helga Eng's House"). Address: Sem Sælands vei 7. Lunch will be in the canteen in the same building. The closest subway station to both these buildings is Blindern Subway Station (Blindern T-banestasjon). Looking forward to see you there! -- Regards, Bjørn-Helge Mevik, dr. scient, Department for Research Computing, University of Oslo

1 1

Issue with cgroup v2 when IgnoreSystemd=yes
by Honoré Bergeron 09 Sep '24

09 Sep '24

Hi, This is a follow-up from https://groups.google.com/g/slurm-users/c/JI3UkbCtj3U, but as I could not find any progress, I am opening a new thread. When setting IgnoreSystemd=yes in cgroup.conf, I have the error: error: common_file_write_content: unable to open '/sys/fs/cgroup/system.slice/cgroup.subtree_control' for writing: No such file or directory error: Cannot enable cpuset in /sys/fs/cgroup/system.slice/cgroup.subtree_control: No such file or directory error: common_file_write_content: unable to open '/sys/fs/cgroup/system.slice/cgroup.subtree_control' for writing: No such file or directory error: Cannot enable memory in /sys/fs/cgroup/system.slice/cgroup.subtree_control: No such file or directory error: common_file_write_content: unable to open '/sys/fs/cgroup/system.slice/cgroup.subtree_control' for writing: No such file or directory error: Cannot enable cpu in /sys/fs/cgroup/system.slice/cgroup.subtree_control: No such file or directory error: Could not create scope directory /sys/fs/cgroup/system.slice/slurmstepd.scope: No such file or directory error: Couldn't load specified plugin name for cgroup/v2: Plugin init() callback failed error: cannot create cgroup context for cgroup/v2 error: Unable to initialize cgroup plugin error: slurmd initialization failed I wrote a patch that solves this issue: --- cgroup_v2.c.orig 2024-09-02 13:18:21.376312875 +0200 +++ cgroup_v2.c 2024-09-02 13:22:00.516986953 +0200 @@ -43,6 +43,7 @@ #include <sys/inotify.h> #include <poll.h> #include <unistd.h> +#include <libgen.h> #include "slurm/slurm.h" #include "slurm/slurm_errno.h" @@ -743,11 +744,33 @@ return SLURM_SUCCESS; } +static int _mkdir(const char *path, mode_t mode) +{ + int rc; + char *dir, *pdir; + + dir = strdup(path); + if (dir == NULL) { + return ENOMEM; + } + pdir = dirname(dir); + if (strcmp(pdir, path) != 0) { + rc = _mkdir(pdir, mode); + if (rc && (errno != EEXIST)) { + free(dir); + return rc; + } + } + rc = mkdir(path, mode); + free(dir); + return rc; +} + static int _init_new_scope(char *scope_path) { int rc; - rc = mkdir(scope_path, 0755); + rc = _mkdir(scope_path, 0755); if (rc && (errno != EEXIST)) { error("Could not create scope directory %s: %m", scope_path); return SLURM_ERROR; This patch concerns the file src/plugins/cgroup/v2/cgroup_v2.c. Am I missing something ? Cheers, Honoré.

1 0

Configuration for nodes with different TmpFs locations and TmpDisk sizes
by Jake Longo 06 Sep '24

06 Sep '24

Hi, We have a number of machines in our compute cluster that have larger disks available for local data. I would like to add them to the same partition as the rest of the nodes but assign them a larger TmpDisk value which would allow users to request a larger tmp and land on those machines. The main hurdle is that (for reasons beyond my control) the larger local disks are on a special mount point /largertmp whereas the rest of the compute cluster uses the vanilla /tmp. I can't see an obvious way to make this work as the TmpFs value appears to be global only and attempting to set TmpDisk to a value larger than TmpFs for those nodes will put the machine into an invalid state. I couldn't see any similar support tickets or anything in the mail archive but I wouldn't have thought it would be that unusual to do this. Thanks in advance! Jake

2 1

salloc not starting shell despite LaunchParameters=use_interactive_step
by Loris Bennett 06 Sep '24

06 Sep '24

Hi, With $ salloc --version slurm 23.11.10 and $ grep LaunchParameters /etc/slurm/slurm.conf LaunchParameters=use_interactive_step the following $ salloc --partition=interactive --ntasks=1 --time=00:03:00 --mem=1000 --qos=standard salloc: Granted job allocation 18928869 salloc: Nodes c001 are ready for job creates a job $ squeue --me JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 18928779 interacti interact loris R 1:05 1 c001 but causes the terminal to block. From a second terminal I can log into the compute node: $ ssh c001 [13:39:36] loris@c001 (1000) ~ Is that the expected behaviour or should salloc return a shell directly on the compute node (like srun --pty /bin/bash -l used to do)? Cheers, Loris -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin

5 8

Nodelist syntax and semantics
by Jackson, Gary L. 05 Sep '24

05 Sep '24

Is there a description of the “nodelist” syntax and semantics somewhere other than the source code? By “nodelist” I mean expressions like “name[000,099-100]” and how this one, for example, expands to “name000, name099, name100”. -- Gary

2 1

Configuration for nodes with different TmpFs locations and TmpDisk sizes
by Jake Longo 05 Sep '24

05 Sep '24

Hi all, We have a number of machines in our compute cluster that have larger disks available for local data. I would like to add them to the same partition as the rest of the nodes but assign them a larger TmpDisk value which would allow users to request a larger tmp and land on those machines. The main hurdle is that (for reasons beyond my control) the larger local disks are on a special mount point /largertmp whereas the rest of the compute cluster uses the vanilla /tmp. I can't see an obvious way to make this work as the TmpFs value appears to be global only and attempting to set TmpDisk to a value larger than TmpFs for those nodes will put the machine into an invalid state. I couldn't see any similar support tickets or anything in the mail archive but I wouldn't have thought it would be that unusual to do this. Thanks in advance! Jake

2 1

Make a job pending in the plugin
by benjamin.jin＠bos-semi.com 05 Sep '24

05 Sep '24

Hello all, I am tyring to build a custom plugin to force some jobs to be pended. In the official document, `ESLURM*` errors are only valid for `job_submit_lua`. I tried to send `ESLURM_JOB_PENDING`, but it only rejects the job submission. Does anyone know how to pend a job in job_submit plugin? Thanks.

2 1

Bug? sbatch not respecting MaxMemPerNode setting
by Angel de Vicente 05 Sep '24

05 Sep '24

Hello, we found an issue with Slurm 24.05.1 and the MaxMemPerNode setting. Slurm is installed in a single workstation, and thus, the number of nodes is just 1. The relevant sections in slurm.conf read: ,---- | EnforcePartLimits=ALL | PartitionName=short Nodes=..... State=UP Default=YES MaxTime=2-00:00:00 MaxCPUsPerNode=76 MaxMemPerNode=231000 OverSubscribe=FORCE:1 `---- Now, if I submit a job requesting 76 CPUs and each one needing 4000M (for a total of 304000M), Slurm does indeed respect the MaxMemPerNode setting and the job is not submitted in the following cases ("-N 1" is not really necessary, as there is only one node): ,---- | $ sbatch -N 1 -n 1 -c 76 -p short --mem-per-cpu=4000M test.batch | sbatch: error: Batch job submission failed: Memory required by task is not available | | $ sbatch -N 1 -n 76 -c 1 -p short --mem-per-cpu=4000M test.batch | sbatch: error: Batch job submission failed: Memory required by task is not available | | $ sbatch -n 1 -c 76 -p short --mem-per-cpu=4000M test.batch | sbatch: error: Batch job submission failed: Memory required by task is not available `---- But with this submission Slurm is happy: ,---- | $ sbatch -n 76 -c 1 -p short --mem-per-cpu=4000M test.batch | Submitted batch job 133982 `---- and the slurmjobcomp.log file does indeed tell me that the memory went above MaxMemPerNode: ,---- | JobId=133982 UserId=......(10487) GroupId=domain users(2000) Name=test JobState=CANCELLED Partition=short TimeLimit=45 StartTime=2024-09-04T09:11:17 EndTime=2024-09-04T09:11:24 NodeList=...... NodeCnt=1 ProcCnt=76 WorkDir=/tmp/. ReservationName= Tres=cpu=76,mem=304000M,node=1,billing=76 Account=ddgroup QOS=domino WcKey= Cluster=...... SubmitTime=2024-09-04T09:11:17 EligibleTime=2024-09-04T09:11:17 DerivedExitCode=0:0 ExitCode=0:0 `---- What is the best way to report issues like this to the Slurm developers? I thought of adding it to https://support.schedmd.com/ but it is not clear to me if that page is only meant for Slurm users with a Support Contract? Cheers, -- Ángel de Vicente Research Software Engineer (Supercomputing and BigData) Instituto de Astrofísica de Canarias (https://www.iac.es/en)

2 3

Slurm Remote Task Launch
by Bhaskar Chakraborty 04 Sep '24

04 Sep '24

Hello, I have a slurm job which needs to launch multiple tasks across the allocated hosts for the job. My criteria is that most of the tasks need to be launched from within the main task launchedby slurm in the launch compute node. So, if the allocated hosts are h1, h2 & h3 with h1 being the main launcher node then theinitial task, say launchTask, launched in h1 will need to launch RemoteTask1, RemoteTask2 in h2 & h3 at somepoint during execution. Can I use srun from inside launchTask to do so? If yes, what would be the syntax / args?If no, then what alternative I have other than using rsh/ssh which mayn't be available in the cluster. Thanks in advance! Regards,Bhaskar.

1 0

Best practices for tracking jobs started across multiple clusters for accounting purposes.
by Di Bernardini, Fabio 02 Sep '24

02 Sep '24

I need to account for jobs composed of multiple jobs launched on multiple federated (and non-federated) clusters, which therefore have different job IDs. What are the best practices to prevent users from bypassing this tracking? NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale: 10.329,14 EUR i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico

3 4

QOS MaxTRESPU node=X intepretation
by David Magda 30 Aug '24

30 Aug '24

Hello, Have a question on how to interpret a Node=X MaxTRESPU value for a QOS: If (e.g.) X=4, and each node has (say) 64 CPUs or cores: if a particular job needs 32 cores, then would two jobs count as the equivalent of one node (2*32=64)? And if X=4 for the QOS, would that mean that eight of these jobs would be able to run ((4*64)/32=8)? Or does this TRES limit only get activated -N/--nodes is used? Don't see anything obvious: https://slurm.schedmd.com/resource_limits.html https://slurm.schedmd.com/sacctmgr.html https://slurm.schedmd.com/tres.html Thanks for any info. Regards, David

1 0

playing with --nodes=<size_string>
by Matteo Guglielmi 30 Aug '24

30 Aug '24

Hello, I have a cluster with four Intel nodes (node[01-04], Feature=intel) and four Amd nodes (node[05-08], Feature=amd). # job file #SBATCH --ntasks=3 #SBATCH --nodes=2,4 #SBATCH --constraint="[intel|amd]" env | grep SLURM # slurm.conf PartitionName=DEFAULT MinNodes=1 MaxNodes=UNLIMITED # log SLURM_JOB_USER=software SLURM_TASKS_PER_NODE=1(x3) SLURM_JOB_UID=1002 SLURM_TASK_PID=49987 SLURM_LOCALID=0 SLURM_SUBMIT_DIR=/home/software SLURMD_NODENAME=node01 SLURM_JOB_START_TIME=1724932865 SLURM_CLUSTER_NAME=cluster SLURM_JOB_END_TIME=1724933465 SLURM_CPUS_ON_NODE=1 SLURM_JOB_CPUS_PER_NODE=1(x3) SLURM_GTIDS=0 SLURM_JOB_PARTITION=nodes SLURM_JOB_NUM_NODES=3 SLURM_JOBID=26 SLURM_JOB_QOS=lprio SLURM_PROCID=0 SLURM_NTASKS=3 SLURM_TOPOLOGY_ADDR=node01 SLURM_TOPOLOGY_ADDR_PATTERN=node SLURM_MEM_PER_CPU=0 SLURM_NODELIST=node[01-03] SLURM_JOB_ACCOUNT=dalco SLURM_PRIO_PROCESS=0 SLURM_NPROCS=3 SLURM_NNODES=3 SLURM_SUBMIT_HOST=master SLURM_JOB_ID=26 SLURM_NODEID=0 SLURM_CONF=/etc/slurm/slurm.conf SLURM_JOB_NAME=mpijob SLURM_JOB_GID=1002 SLURM_JOB_NODELIST=node[01-03] <<<=== why three nodes? Shouldn't this still be two nodes? Thank you.

2 5

Print Slurm Stats on Login
by Paul Edmon 29 Aug '24

29 Aug '24

We are working to make our users more aware of their usage. One of the ideas we came up with was to having some basic usage stats printed at login (usage over past day, fairshare, job efficiency, etc). Does anyone have any scripts or methods that they use to do this? Before baking my own I was curious what other sites do and if they would be willing to share their scripts and methodology. -Paul Edmon-

10 20

Multiple Counts Question
by Matteo Guglielmi 29 Aug '24

29 Aug '24

Hello, Does anyone know why this is possible in slurm: --constraint="[rack1*2&rack2*4]" and this is not: --constraint="[rack1*2|rack2*4]" ? Thank you.

1 0

REST API - get_user_environment
by jpuerto＠psc.edu 29 Aug '24

29 Aug '24

In previous versions (v0.0.36) of the REST API the job submission endpoint had a field titled "get_user_environment"; however, it doesn't appear to exist in v0.0.40. Is there an equivalent parameter that should be used? What is the suggested approach for mimicking this behavior in v0.0.40? Best regards, Juan

5 13

ResumeAfterTime - Lacking Info
by Sid Young 28 Aug '24

28 Aug '24

G'Day all, Can anyone shed light on the parameter "Resume AfterTime" returned from the command "scontrol show node XXX" Can it be used to automatically resume a "Down"ed node? Sid

1 0

Invitation for Feedback on the Slurm-Lab Project
by Patrick Pun 28 Aug '24

28 Aug '24

Hi Slurm User Group, I’m excited to share a project I’ve been working on: Slurm-Lab. It’s a containerised, easy-to-deploy environment designed for learning, testing, and experimenting with Slurm. This project has allowed me and some of my friends to explore various Slurm features directly on our laptops, and I hope it can be just as useful to some of you. Highlights: • Quick Deployment: Setting up a Slurm environment is a breeze—just clone the project and pull the image to get started. • User-Friendly Interface: Interact with your sandbox environment seamlessly through JupyterHub in your browser. • Educational Focus: Explore different Slurm features with provided examples and interactive notebooks, perfect for both new and experienced users. Why I Need Your Feedback: As members of the Slurm community, your insights and experience are incredibly valuable. I’m seeking your feedback on how Slurm-Lab can better serve our community. Whether it’s feature suggestions, bug reports, or general comments. Please take a look at the project and share your thoughts. You can contribute by opening issues or submitting merge requests directly on GitLab. GitLab Repo: https://gitlab.com/CSniper/slurm-lab GitHub Mirror: https://github.com/csniper-patrick/slurm-lab Thank you for your time and for helping make Slurm-Lab a valuable resource for the entire community! Cheers, Patrick

1 0

Spread a multistep job across clusters
by Di Bernardini, Fabio 27 Aug '24

27 Aug '24

Hi everyone, for accounting reasons, I need to create only one job across two or more federated clusters with two or more srun steps. I'm trying with hetjobs but it's not clear to me from the documentation (https://slurm.schedmd.com/heterogeneous_jobs.html) if this is possible and how to do it. I'm trying with this script, but the steps are executed on only the first cluster. Can you tell me if there is a mistake in the hetjob or if it has to be done in another way? #!/bin/bash #SBATCH hetjob #SBATCH --clusters=cluster2 srun -v --het-group=0 hostname #SBATCH hetjob #SBATCH --clusters=cluster3 srun -v --het-group=1 hostname NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale: 10.329,14 EUR i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico

3 2

Slurm versions 24.05.3 and 23.11.10 are now available
by Marshall Garey 27 Aug '24

27 Aug '24

We are pleased to announce the availability of Slurm versions 24.05.3 and 23.11.10. Version 24.05.3 fixes a potential database problem when deleting a qos. This bug only existed in 24.05. Both versions have fixes for jobs potentially being stuck when using cloud nodes when some nodes are powered down, a regression in 23.11.9 and 24.05.2 that caused sattach to crash, and some other minor issues. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.05.3 > ========================== > -- data_parser/v0.0.40 - Added field descriptions > -- slurmrestd - Avoid creating new slurmdbd connection per request to > '* /slurm/slurmctld/*/*' endpoints. > -- Fix compilation issue with switch/hpe_slingshot plugin. > -- Fix gres per task allocation with threads-per-core. > -- data_parser/v0.0.41 - Added field descriptions > -- slurmrestd - Change back generated OpenAPI schema for > `DELETE /slurm/v0.0.40/jobs/` to RequestBody instead of using parameters > for request. slurmrestd will continue accept endpoint requests via > RequestBody or HTTP query. > -- topology/tree - Fix issues with switch distance optimization. > -- Fix potential segfault of secondary slurmctld when falling back to the > primary when running with a JobComp plugin. > -- Enable --json/--yaml=v0.0.39 options on client commands to dump data using > data_parser/v0.0.39 instead or outputting nothing. > -- switch/hpe_slingshot - Fix issue that could result in a 0 length state file. > -- Fix unnecessary message protocol downgrade for unregistered nodes. > -- Fix unnecessarily packing alias addrs when terminating jobs with a mix of > non-cloud/dynamic nodes and powered down cloud/dynamic nodes. > -- accounting_storage/mysql - Fix issue when deleting a qos that could remove > too many commas from the qos and/or delta_qos fields of the assoc table. > -- slurmctld - Fix memory leak when using RestrictedCoresPerGPU. > -- Fix allowing access to reservations without MaxStartDelay set. > -- Fix regression introduced in 24.05.0rc1 breaking srun --send-libs parsing. > -- Fix slurmd vsize memory leak when using job submission/allocation commands > that implicitly or explicitly use --get-user-env. > -- slurmd - Fix node going into invalid state when using CPUSpecList and > setting CPUs to the # of cores on a multithreaded node > -- Fix reboot asap nodes being considered in backfill after a restart. > -- Fix --clusters/-M queries for clusters outside of a federation when > fed_display is configured. > -- Fix scontrol allowing updating job with bad cpus-per-task value. > -- sattach - Fix regression from 24.05.2 security fix leading to crash. > -- mpi/pmix - Fix assertion when built under --enable-debug. > * Changes in Slurm 23.11.10 > =========================== > -- switch/hpe_slingshot - Fix issue that could result in a 0 length state file. > -- Fix unnecessary message protocol downgrade for unregistered nodes. > -- Fix unnecessarily packing alias addrs when terminating jobs with a mix of > non-cloud/dynamic nodes and powered down cloud/dynamic nodes. > -- Fix allowing access to reservations without MaxStartDelay set. > -- Fix scontrol allowing updating job with bad cpus-per-task value. > -- sattach - Fix regression from 23.11.9 security fix leading to crash.

1 0

Slurm management of Lenovo SD665 V3 dual-server trays?
by Ole Holm Nielsen 26 Aug '24

26 Aug '24

We're experimenting with ways to manage our new racks of Lenovo SD665 V3 dual-server trays with Direct Water Cooling (further information is in our Wiki page https://wiki.fysik.dtu.dk/ITwiki/Lenovo_SD665_V3/ ) Management problems arise because 2 servers share a tray with common power and water cooling. This wouldn't be so bad if it weren't for Lenovo's NVIDIA/Mellanox SharedIO Infiniband adapters, where the left-hand node's IB adapter is a client of the right-hand node's adapter. So we can't reboot or power down the right-hand node without killing any MPI jobs that happen to be using the left-hand node. My question is if other Slurm sites owning Lenovo dual-server trays with SharedIO Infiniband adapters have developed some clever ways of handling such node pairs a single entity somehow? Is there anything we should configure on the Slurm side to make such nodes easier to manage? Thanks for sharing any insights, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

2 5

How to select a container runtime system?
by wdennis＠nec-labs.com 23 Aug '24

23 Aug '24

We are getting a few calls to support container workloads on our Slurm cluster; I want to support these user's usecases, so am looking into it now. The problem for me is, I'm not super-familiar with container runtimes excepting (regular rootful) Docker... I see that any Slurm-compatible runtime must be "rootless", so that takes out the integration with regular Docker. How does one choose between the many options (from what I gather, Apptainer/Singularity, Podman, and rootless Docker seem to be the most widely-used.) We also have to support GPUs in these containers, but I assume that any of these can support that. Can anyone tell me what the most popular Slurm-compatible container runtime may be, and plusses/minuses of the three I mentioned above? Thanks!

2 1

Slurmdbd purge and reported downtime
by Davide DelVento 23 Aug '24

23 Aug '24

I am confused by the reported amount of Down and PLND Down by sreport. According to it, our cluster would have had a significant amount of downtime, which I know didn't happen (or, according to the documentation "time that slurmctld was not responding", see https://slurm.schedmd.com/sreport.html) Could it be my purge settings causing this problem? How can I check (maybe in some logs, maybe in the future) if actually slurmctld was not responding? The expected long-term numbers should be less than the ones reported for last month when we had an issue with a few nodes.... Thanks! [davide@login ~]$ grep Purge /opt/slurm/slurmdbd.conf #JobPurge=12 #StepPurge=1 PurgeEventAfter=1month PurgeJobAfter=12month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/22 -------------------------------------------------------------------------------- Cluster Utilization 2022-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Down Idle Planned Reported --------- -------------- --------------- --------------- -------------- ---------------- --------- ---------------- cluster cpu 19.50% 12.07% 3.92% 64.36% 0.15% 100.03% cluster mem 16.13% 13.17% 4.56% 66.13% 0.00% 99.99% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/23 -------------------------------------------------------------------------------- Cluster Utilization 2023-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Down Idle Planned Reported --------- -------------- --------------- --------------- -------------- --------------- --------- ---------------- cluster cpu 28.74% 18.80% 6.44% 45.77% 0.24% 100.02% cluster mem 22.52% 20.54% 7.38% 49.55% 0.00% 99.98% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/24 -------------------------------------------------------------------------------- Cluster Utilization 2024-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Down Idle Planned Reported --------- -------------- -------------- --------------- -------------- --------------- -------- --------------- cluster cpu 29.92% 24.88% 17.73% 27.45% 0.02% 100.00% cluster mem 20.07% 28.60% 19.57% 31.76% 0.00% 100.00% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=8/8/24 -------------------------------------------------------------------------------- Cluster Utilization 2024-08-08T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Dow Idle Planned Reported --------- -------------- ------------- ------------ -------- -------------- -------- -------------- cluster cpu 15.96% 2.53% 0.00% 81.51% 0.00% 100.00% cluster mem 9.18% 2.22% 0.00% 88.60% 0.00% 100.00% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=7/7/24 -------------------------------------------------------------------------------- Cluster Utilization 2024-07-07T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Dow Idle Planned Reported --------- -------------- -------------- ------------- -------- -------------- -------- -------------- cluster cpu 27.07% 2.57% 0.00% 70.34% 0.02% 100.00% cluster mem 17.35% 2.26% 0.00% 80.40% 0.00% 100.00%

2 2

trying to figure out how to troubleshoot cloud node resume/suspend
by Alex Chekholko 23 Aug '24

23 Aug '24

Hi all, I have a cloud cluster running in GCP that seems to have gotten stuck in a state where the slurmctld will not start/stop compute nodes, it just sits there with thousands of jobs in the queue and only a few compute nodes up and running (out of thousands). I can try to kick it by setting node states to drain/resume/power_up/power_down, whatever, nothing seems to happen. In slurmctld.log I get stuff like this: 2024-08-22 16:44:31,761 INFO: created 1 instances: nodes=hpcmarjola-computenodeset-384 but then in resume.log there is no corresponding entry One hypothesis I have is that it's taking like 30+mins to spin up a new node, but I know GCP doesn't take that long to create the VM, so how can I troubleshoot more what the slurm resume script is doing? e.g. I saw in the log: [2024-08-22T16:39:57.151] powering up node hpcmarjola-computenodeset-7 [2024-08-22T16:59:10.371] Node hpcmarjola-computenodeset-7 now responding I tried setting SlurmctldDebug = verbose #instead of info, so it prints power / resume status messages to the log And I see things like this: [2024-08-23T16:40:29.006] node hpcmarjola-computenodeset-1068 not resumed by ResumeTimeout(600) - marking down and power_save [2024-08-23T16:40:29.006] requeue job JobId=184535_503(185038) due to failure of node hpcmarjola-computenodeset-1068 [2024-08-23T16:40:29.006] Requeuing JobId=184535_503(185038) [2024-08-23T16:40:29.007] POWER: power_save: waking nodes hpcmarjola-computenodeset-[2154-2156] [2024-08-23T16:40:29.007] POWER: power_save: handle failed nodes hpcmarjola-computenodeset-[1065-1068] I'll try bumping this value but that seems crazy high: ResumeTimeout=2400 slurm 23.11.7 I read through these pages: https://slurm.schedmd.com/power_save.html my config files (generated by gcp hpc toolkit scripts): topology.conf: SwitchName=nodeset-root Switches=computenodeset SwitchName=computenodeset Nodes=hpcmarjola-computenodeset-[0-4999] cloud.conf: PrivateData=cloud LaunchParameters=enable_nss_slurm,use_interactive_step SlurmctldParameters=cloud_dns,enable_configless,idle_on_node_suspend SchedulerParameters=bf_continue,salloc_wait_nodes,ignore_prefer_validation SuspendProgram=/slurm/scripts/suspend.py ResumeProgram=/slurm/scripts/resume.py ResumeFailProgram=/slurm/scripts/suspend.py ResumeRate=2 ResumeTimeout=600 SuspendRate=2 SuspendTimeout=600 TreeWidth=16 TopologyPlugin=topology/tree NodeSet=x-login Feature=x-login PartitionName=x-login Nodes=x-login State=UP DefMemPerCPU=1 Hidden=YES RootOnly=YES NodeName=DEFAULT State=UNKNOWN RealMemory=126832 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=1 CPUs=8 NodeName=hpcmarjola-computenodeset-[0-4999] State=CLOUD NodeSet=computenodeset Nodes=hpcmarjola-computenodeset-[0-4999] PartitionName=compute Nodes=computenodeset State=UP DefMemPerCPU=15854 SuspendTime=600 Oversubscribe=Exclusive PowerDownOnIdle=NO Default=YES ResumeTimeout=600 SuspendTimeout=600 SuspendExcParts=x-login slurm.conf: # slurm.conf # https://slurm.schedmd.com/slurm.conf.html # https://slurm.schedmd.com/configurator.html ProctrackType=proctrack/cgroup SlurmctldPidFile=/var/run/slurm/slurmctld.pid SlurmdPidFile=/var/run/slurm/slurmd.pid TaskPlugin=task/affinity,task/cgroup MaxNodeCount=64000 # # # SCHEDULING SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory # # # LOGGING AND ACCOUNTING AccountingStoreFlags=job_comment JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/cgroup SlurmctldDebug=info SlurmdDebug=info DebugFlags=Power # # # TIMERS MessageTimeout=60 ################################################################################ # vvvvv WARNING: DO NOT MODIFY SECTION BELOW vvvvv # ################################################################################ SlurmctldHost=hpcmarjola-controller(internal-cloud-dns-redacted) AuthType=auth/munge AuthInfo=cred_expire=120 AuthAltTypes=auth/jwt CredType=cred/munge MpiDefault=none ReturnToService=2 SlurmctldPort=6820-6830 SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm StateSaveLocation=/var/spool/slurm # # # LOGGING AND ACCOUNTING AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=hpcmarjola-controller ClusterName=hpcmarjola SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdLogFile=/var/log/slurm/slurmd-%n.log # # # GENERATED CLOUD CONFIGURATIONS include cloud.conf ################################################################################ # ^^^^^ WARNING: DO NOT MODIFY SECTION ABOVE ^^^^^ # ################################################################################ MaxArraySize = 200001 MaxJobCount = 200001

1 0

Manually initiating a DB purge
by Sid Young 22 Aug '24

22 Aug '24

G'Day all, I have 3 years worth of job records in the slurm DB and we do not have any need to actually track anything at this stage, I would like to keep 12months worth of jobs so I need to purge 2 years worth at some point.. Is there a command to issue via scontrol to kick off a SlurmPurge using the values in the slurmdbd.conf file (currently these are 1000days, so I can see a few dozen records disappear before I shorten it to 365days in incremental steps). Sid

2 4

active reservation lost when increasing the duration
by Danny Marc Rotscher 22 Aug '24

22 Aug '24

Dear All, we are running Slurm 23.11.8 and I loose an active reservation and the jobs running in that reservation, when I tried to increase the duration of that reservation. I only want to tell you, that you may not run into the same problem. Kind regards, Danny Rotscher

1 0

Slurm hanging behavior
by Richard Yang 20 Aug '24

20 Aug '24

Hello Slurm community, We are using slurm as the system to deploy training jobs on a large gpu cluster, but encounter a strange behavior. As new comers, we wonder if this is a known behavior. Below is some more info: * We are running a relatively older version 22.0.5 * At relatively higher load, we encountered hanging. It is particularly puzzling in the following sense: assume we have nodelist1 with 6 hosts and nodelist2 with 7 hosts. We run simple ‘hostname’. Deploying on nodelist1 alone or nodelusr2 alone will be fine, but with all 13 hosts, the debug messages show that the execution hang after showing that the last task done. It then hangs for exactly 180 seconds. Does anyone know the potential issue? We sure be happy to post more config details or debug messages. Thank you so much! Richard

1 0

Access to --constraint=<arg> in Lua cli_filter?
by Kevin Buckley 20 Aug '24

20 Aug '24

If I supply a --constraint=<arg> option to an sbatch/salloc/srun, does the arg appear inside any object that a Lua CLI Filter could access? I've tried this basic check if is_unset(options['constraint']) then slurm_errorf('constraint is unset ') end and seen that that object is, indeed, unset. Kevin

2 2

Unable to run sequential jobs simultaneously on the same node
by Arko Roy 20 Aug '24

20 Aug '24

I want to run 50 sequential jobs (essentially 50 copies of the same code with different input parameters) on a particular node. However, as soon as one of the jobs gets executed, the other 49 jobs get killed immediately with exit code 9. The jobs are not interacting and are strictly parallel. However, if the 50 jobs run on 50 different nodes, it runs successfully. Can anyone please help with possible fixes? I see a discussion almost along the similar lines in https://groups.google.com/g/slurm-users/c/I1T6GWcLjt4 But could not get the final solution. -- Arko Roy Assistant Professor School of Physical Sciences Indian Institute of Technology Mandi Kamand, Mandi Himachal Pradesh - 175 005, India Email: arko(a)iitmandi.ac.in Web: https://faculty.iitmandi.ac.in/~arko/

5 7

sreport syntax for TRES/GPU usage
by Robert Kudyba 16 Aug '24

16 Aug '24

In a 25 node heterogeneous cluster with 4 different types of GPUs, to get granular to see which GPUs were used most over a time period we have to set AccountingStorageTRES to something like: AccountingStorageTRES=gres/gpu,gres/gpu:rtx8000,gres/gpu:v100s,gres/gpu:a40,gres/gpu:a100 Unfortunately it's currently at: AccountingStorageTRES=gres/gpu At least all nodes have the same GPU within each node. What are some good options to sreport to get details on usage over a year, e.g., percentage of CPU vs GPU, which partitions/accounts used the most GPUs, etc. From this example: sreport -tminper -t Percent cluster utilization --tres="cpu,gres/gpu" start=2023-07-01 -------------------------------------------------------------------------------- Cluster Utilization 2023-07-01T00:00:00 - 2024-08-15T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Dow Idle Reserved Reported --------- -------------- ----------- ---------- -------- ----------- ---------- ----------- cluster cpu 43.81% 2.87% 0.00% 48.35% 4.97% 99.86% cluster gres/gpu 50.36% 3.59% 0.00% 46.05% 0.00% 100.38% Is that showing that 50% of all jobs were run with GPUs? How do we read the Idle column? Why does Reported show > 100% for gres?

1 0

Upgrade compute node to 24.05.2
by Sid Young 15 Aug '24

15 Aug '24

G'Day all, I've been upgrading cmy cluster from 20.11.0 in small steps to get to 24.05.2. Currently 1 have all nodes on 23.02.8, the controller on 24.05.2 and a single test node on 24.05.2. All are Centos 7.9 (upgrade to Oracle Linux 8.10 is Phase 2 of the upgrades). When I check the slurmd status on the test node I get: [root@hpc-dev-01 24.05.2]# systemctl status slurmd ● slurmd.service - Slurm node daemon Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2024-08-15 10:45:15 AEST; 24s ago Main PID: 46391 (slurmd) Tasks: 1 Memory: 1.2M CGroup: /system.slice/slurmd.service └─46391 /usr/sbin/slurmd --systemd Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: Considering each NUMA node as a socket Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: Node reconfigured socket/core boundaries SocketsPerBoard=4:8(hw) CoresPerSocket=16:8(hw) Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: Considering each NUMA node as a socket Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: slurmd version 24.05.2 started Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: *plugin_load_from_file: Incompatible Slurm plugin /usr/lib64/slurm/mpi_none.so version (23.02.8)* Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: error: Couldn't load specified plugin name for mpi/none: Incompatible plugin version Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: error: MPI: Cannot create context for mpi/none Aug 15 10:45:15 hpc-dev-01 systemd[1]: Started Slurm node daemon. Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: slurmd started on Thu, 15 Aug 2024 10:45:15 +1000 Aug 15 10:45:15 hpc-dev-01 slurmd[46391]: slurmd: CPUs=64 Boards=1 Sockets=8 Cores=8 Threads=1 Memory=257778 TmpDisk=15998 Uptime=2898769 CPUSpecL...ve=(null) Hint: Some lines were ellipsized, use -l to show in full. [root@hpc-dev-01 24.05.2]# We don't use MPI (life science workloads)... should I remove the file? If it is version 23.02.8 then doesn't 24.05.2 have that plugin built in? There are no references to mpi i the slurm.conf file Sid

2 1

Seeking Commercial SLURM Subscription Provider
by John Joseph 13 Aug '24

13 Aug '24

Dear All, Good morning. We successfully implemented a 4-node SLURM cluster with shared storage using GlusterFS and were able to run COMSOL programs on it. After this learning experience, we've determined that it would be beneficial to switch to a commercial SLURM subscription for better support. We are currently seeking a solution provider who can offer support based on a commercial subscription. I would like to reach out to the group for recommendations or advice on how we can avail these services commercially. Thank you.Joseph John

2 1

Annoying canonical question about converting SLURM_JOB_NODELIST to a host list for mpirun
by Jeffrey Layton 12 Aug '24

12 Aug '24

Good afternoon, I know this question has been asked a million times, but what is the canonical way to convert the list of nodes for a job that is container in a Slurm variable, I use SLURM_JOB_NODELIST, to a host list appropriate for mpirun in OpenMPI (perhaps MPICH as well)? Before anyone says, compile OpenMPI with Slurm, I can't change the Slurm installation. I have a script that does the conversion on a single node, but when I try a cluster that does not include the single node, I get an error: scontrol: error: host list is empty The line in the script corresponding to this is, list=$(scontrol show hostname $SLURM_NODELIST) I've tried using the env variable SLURM_JOB_NODELIST and I get the same error message. Thanks! Jeff

3 8

FairShare if there's only one account?
by Drucker, Daniel 10 Aug '24

10 Aug '24

Simple question: Does FairShare still work if every user is under one account? E.g.: $ sacctmgr show assoc format=Account,User Account User ---------- ---------- root root root mic mic asmith mic bsmith mic csmith mic djones mic ejones mic frubble Will it divide time up fairly between the users? I have: PriorityType=priority/multifactor PriorityFavorSmall=YES PriorityWeightAge=50000 PriorityWeightFairshare=100000 PriorityWeightJobSize=0 PriorityWeightQOS=0 In 21.08.8. -- Daniel M. Drucker, Ph.D. Director of IT, MGB Imaging at Belmont McLean Hospital, a Harvard Medical School Affiliate The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

5 31

Jobs distribution over CPUs
by Rafał Lalik 09 Aug '24

09 Aug '24

Hi, I have a very simple computing farm on a single PC with AMD Ryzen 7950X (2x16 cores). I have configured my slurm to use up to 25 CPUs: NodeName=palmer CPUs=25 RealMemory=40000 State=UNKNOWN # Boards=1 SocketsPerBoard=1 CoresPerSocket=16 ThreadsPerCore=2 PartitionName=main Nodes=ALL Default=YES MaxTime=INFINITE State=UP But using htop I see that with all 25 jobs running I use max 16 cores. It seems that like * 6 jobs are using 100% of CPU * 20 jobs use 50% CPU each I use slurm 24.05.2, but I am pretty sure in the past when using I think one of 22.x version, the jobs distribution was like 25 CPUs for 25 jobs with 100% each. Is there anything I can do in the configuration of my farm to have better utilisation of CPU cores? Right now like half of them is not really used. Regards, Rafał

1 0

The issue in the distribution of job
by Sundaram Kumaran 09 Aug '24

09 Aug '24

Dear All, May I have your suggestion in my issue facing, While the job is launched using "salloc -N4--mem 4000 -p active" I find the job is running in the one compute node and the other 3 machines are free, I don`t find the job is distributed evenly, May I have your suggestion, I do squeue /scontrol to find the job distribution and it displays the 4 machines but when I check on the respective machines I don`t find the job running only one machine takes the whole node, Is there any issue in my conf file or what needs to be done, May I have your suggestion pls. FYI, salloc -N4 --mem 4000 -p active [cid:image001.png@01DAE8B4.215FBA50] While using TOP, I find only Debussy is used heavily, I don`t find my job is evenly distributed, May I have your guidance pls. [cid:image002.jpg@01DAEA98.2573DE10] Regards, KumaranS This e-mail and any attachments are only for the use of the intended recipient and may contain material that is confidential, privileged and/or protected by the Official Secrets Act. If you are not the intended recipient, please delete it or notify the sender immediately. Please do not copy or use it for any purpose or disclose the contents to any other person.

2 1

Optimizing CPU socket affinities and NVLink
by Matthew R. Baney 08 Aug '24

08 Aug '24

Hello, I've recently adopted setting AutoDetect=nvml in our GPU nodes' gres.conf files to automatically populate Cores and Links for GPUs, which has been working well. I'm now wondering if I can prioritize having single GPU jobs scheduled on NVLink pairs (these are PCIe A6000s) where one of the GPUs in the pair is already running a single GPU job, assuming the CPU socket with affinity has enough cores to handle the job. We have some users wanting to run single GPU jobs and others wanting to run dual GPU jobs, both on the same nodes, so we would prefer not to configure each NVLink pair as a single GRES, for better job throughput. As is, I've observed that for a node with at least 4 GPUs and 2 sockets (one NVLink pair per socket), Slurm will prioritize evening out core allocation between the sockets. Once the second single GPU job is submitted, one GPU in each NVLink pair is taken up and a subsequent dual GPU job can still run, but doesn't have access to an NVLink pair. We've also got a few nodes where single GPUs have failed, resulting in some NVLink'd pairs and usually a single non-NVLink'd GPU (3 or 7 total GPUs). It'd be ideal if single GPU jobs also got prioritized for scheduling on the non-NVLink'd GPU in this case. Is this possible? All the best, Matthew -- Matthew Baney Assistant Director of Computational Systems mbaney(a)umd.edu | (301) 405-6756 University of Maryland Institute for Advanced Computer Studies 3154 Brendan Iribe Center 8125 Paint Branch Dr. College Park, MD 20742

1 0

Integrate podman to run containers
by amin.raeiszadeh＠gmail.com 08 Aug '24

08 Aug '24

Hello, I have a test cluster consist of two nodes, one as controller and the other as compute node. I followed all the steps from slurm documentation and I want to run jobs as containers but I get the following error when running podman run hello-world on controller node: time="2024-08-06T12:02:54+02:00" level=warning msg="freezer not supported: openat2 /sys/fs/cgroup/system.slice/slurmstepd.scope/job_332/step_0/user/arlvm6.ara.332.0.0/cgroup.freeze: no such file or directory" srun: error: arlvm6: task 0: Exited with exit code 1 time="2024-08-06T12:02:54+02:00" level=warning msg="lstat /sys/fs/cgroup/system.slice/slurmstepd.scope/job_332/step_0/user/arlvm6.ara.332.0.0: no such file or directory" time="2024-08-06T12:02:54+02:00" level=error msg="runc run failed: unable to start container process: unable to apply cgroup configuration: rootless needs no limits + no cgrouppath when no permission is granted for cgroups: mkdir /sys/fs/cgroup/system.slice/slurmstepd.scope/job_332/step_0/user/arlvm6.ara.332.0.0: permission denied" As I tracked on the compute node this path exists /sys/fs/cgroup/system.slice/slurmstepd.scope/ but it looks that could not create the job_332/step_0/user/arlvm6.ara.332.0.0 . The cgroup.conf: CgroupPlugin=cgroup/v2 ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes It is the oci.conf: EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)=" RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)=" RunTimeQuery="runc --rootless=true --root=/run/user/1223609544/ state %n.%u.%j.%s.%t" RunTimeKill="runc --rootless=true --root=/run/user/1223609544/ kill -a %n.%u.%j.%s.%t SIGKILL" RunTimeDelete="runc --rootless=true --root=/run/user/1223609544/ delete --force %n.%u.%j.%s.%t" RunTimeRun="runc --rootless=true --root=/run/user/1223609544/ run %n.%u.%j.%s.%t -b %b" As you see I changed the kill command a bit because without SIGKILL param it could not kill the containers. I test again the oci run time on both controller and compute nodes and I think might be helpful to mention two points: the delete command will not work because if you kill the container then there is no resource to be deleted at least in my tests. there is no pause and resume in oci.conf but I test them and got the same error for freezer support and cgroup permissions.

1 0

Find out submit host of past job?
by Steffen Grunewald 07 Aug '24

07 Aug '24

Hello everyone, I've grepped the manual pages and crawled the 'net, but couldn't find any answer to the following problem: I can see that the ctld keeps a record of it below /var/spool/slurm - as long as the job is running or waiting (and shown by "squeue") - and that this stores the environment that contains SLURM_SUBMIT_HOST - but this information seems to be lost when the job finishes. Is there a way to find out what the value of SLURM_SUBMIT_HOST was? I'd be interested in a few more env variables, but this one should be sufficient for a start... Is "sacct" just lacking a job field, or is this info indeed dropped and not stored in the DB? Thanks, Steffen -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~

6 9

"error: slurm_auth_get_host: Lookup failed: Unknown host" in slurmctld.log
by Chris Taylor 06 Aug '24

06 Aug '24

I keep getting this logged on my Slurm control host: [2024-08-06T20:31:40.196] error: slurm_auth_get_host: Lookup failed: Unknown host I don't see an identifiable pattern and I'm not sure how to troubleshoot. The jobs being submitted at or around that time seem fine and nobody's complained about their jobs not running, so I can't track down a bad or nonexistent DNS record. Chris

1 0

With slurm, how to allocate a whole node for a single multi-threaded process?
by Henrique Almeida 06 Aug '24

06 Aug '24

Hello, everyone, with slurm, how to allocate a whole node for a single multi-threaded process? https://stackoverflow.com/questions/78818547/with-slurm-how-to-allocate-a-w… -- Henrique Dante de Almeida hdante(a)gmail.com

8 16

LRMS error: (-1) Job missing from SLURM."
by Felix 06 Aug '24

06 Aug '24

Hello at site RO-14-ITIM, after a power failure I get the following problem 2024-08-06 15:53:04 Finished - job id: c9INDmclYv5ngvuSSqSAreymYz3jwmOETUEmV71LDmABFKDm7KNpMn, unix user: 1900:1900, name: "org.nordugrid.ARC-CE-result-ops", owner: "/dc=eu/dc=egi/c=hr/o=robots/o=srce/cn=robot:argo-egi@cro-ngi.hr", lrms: SLURM, queue: debug, lrmsid: 274399, failure: "LRMS error: (-1) Job missing from SLURM." 2024-08-06 15:53:04 Finished - job id: tjJNDmclYv5ngvuSSqSAreymYz3jwmOETUEmd71LDmABFKDmePf7To, unix user: 1900:1900, name: "org.nordugrid.ARC-CE-result-ops", owner: "/dc=eu/dc=egi/c=hr/o=robots/o=srce/cn=robot:argo-egi@cro-ngi.hr", lrms: SLURM, queue: debug, lrmsid: 274400, failure: "LRMS error: (-1) Job missing from SLURM." 2024-08-06 15:53:04 Finished - job id: kiJNDmclYv5ngvuSSqSAreymYz3jwmOETUEml71LDmABFKDmCmwifm, unix user: 1900:1900, name: "org.nordugrid.ARC-CE-result-ops", owner: "/dc=eu/dc=egi/c=hr/o=robots/o=srce/cn=robot:argo-egi@cro-ngi.hr", lrms: SLURM, queue: debug, lrmsid: 274398, failure: "LRMS error: (-1) Job missing from SLURM." The jobs can not be seen in sinfo or squeue And indication on how where to look up the problem? Thank you Felix -- Dr. Eng. Farcas Felix National Institute of Research and Development of Isotopic and Molecular Technology, IT - Department - Cluj-Napoca, Romania Mobile: +40742195323

2 1

when running `salloc --gres=gpu:1` should I see all gpus in nvidia-smi ?
by Oren 05 Aug '24

05 Aug '24

Hello, When I am running this command: `salloc --nodelist=gpu03 -p A4500_Features --gres=gpu:1` and then automatically ssh to the job, what should I see when I run nvidia-smi? All the GPUs in the host or just a single one? Thanks

2 2

_refresh_assoc_mgr_qos_list: no new list given back keeping cached one
by joao.damas＠syngenta.com 05 Aug '24

05 Aug '24

Hi all, We are doing a simple setup for a Slurm cluster (version 23.11.6). We follow the documentation and we are trying a setup still without accounting or slurmdbd. The slurm.conf is really simple: ``` ClusterName=Develop SlurmctldHost=head # Slurm configuration AuthType=auth/munge CryptoType=crypto/munge SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdLogFile=/var/log/slurm/slurmd.log SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm StateSaveLocation=/var/spool/slurmctld # Nodes NodeName=worker1 CoresPerSocket=2 Sockets=1 ThreadsPerCore=1 NodeName=worker2 CoresPerSocket=2 Sockets=1 ThreadsPerCore=1 # Partitions PartitionName=develop Default=YES MaxTime=UNLIMITED Nodes="worker1,worker2" ``` When running a simple `srun sleep 10`, all works well and the log file shows: [2024-05-15T12:34:12.741] sched: _slurm_rpc_allocate_resources JobId=1 NodeList=worker1 usec=549 [2024-05-15T12:34:22.775] _job_complete: JobId=1 WEXITSTATUS 0 [2024-05-15T12:34:22.775] _job_complete: JobId=1 done But when creating a scrip with the same sleep command, and submiting using `sbatch test.sh`, the log shows: [2024-05-15T12:35:39.916] _slurm_rpc_submit_batch_job: JobId=2 InitPrio=1 usec=368 [2024-05-15T12:35:40.000] error: _refresh_assoc_mgr_qos_list: no new list given back keeping cached one. [2024-05-15T12:35:40.000] sched: JobId=2 has invalid account [2024-05-15T12:35:40.145] sched/backfill: _start_job: Started JobId=2 in develop on worker1 [2024-05-15T12:35:50.172] _job_complete: JobId=2 WEXITSTATUS 0 [2024-05-15T12:35:50.172] _job_complete: JobId=2 done We have the same account with the UID and GID, as said in the documentation. Looking at the function that seems to spit out that error (https://github.com/SchedMD/slurm/blob/e9f28ede27795f525e62f998cb2d40931d884…), it appears like there should be some accounting setup? We do not have slurmdbd setup and the documentation states we should test basic functionality before implementing that daemon. Any tips? Thanks in advance. João

4 3

problem with squeue --json with version 24.05.1
by Markus Köberl 05 Aug '24

05 Aug '24

$ squeue --version slurm 24.05.1 $ squeue --json malloc(): invalid size (unsorted) Aborted forcing an older data_parser version works: $ squeue --json=v0.0.40 regards Markus Köberl -- Markus Koeberl Graz University of Technology Signal Processing and Speech Communication Laboratory E-mail: markus.koeberl(a)tugraz.at

4 5

Upgrade node while jobs running
by Sid Young 03 Aug '24

03 Aug '24

G'day all, I've been waiting for node to become idle before upgrading them however some jobs take a long time. If I try to remove all the packages I assume that kills the slurmstep program and with it the job. Sid

5 5

slumrestd 24.05.1: crashes when GET on /slurm/v0.0.41/nodes : unsorted double linked list corrupted
by Josef Dvořáček 01 Aug '24

01 Aug '24

Isn't this failure familiar to anyone? When I ask API endpoint "localhost:6820/slurm/v0.0.41/jobs", slurmrestd segrafults with unsorted double linked list corrupted. Anyone using this API endpoint without segfaulting? I do the get using curl: curl --header X-SLURM-USER-NAME:root --header X-SLURM-USER-TOKEN:$SLURM_JWT -G localhost:6820/slurm/v0.0.41/jobs In comparison, curl --header X-SLURM-USER-NAME:root --header X-SLURM-USER-TOKEN:$SLURM_JWT -G localhost:6820/slurm/v0.0.41/nodes Works well. josef čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug: _on_url: [[localhost]:52909] url path: /slurm/v0.0.41/jobs query: (null) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: operations_router: [[localhost]:52909] GET /slurm/v0.0.41/jobs čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: slurmrestd: operations_router: [[localhost]:52909] GET /slurm/v0.0.41/jobs čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[localhost]:52909] attempting user_name root token authentication pass through čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[localhost]:52909] attempting user_name root token authentication pass through čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=1 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi.json"](0, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=1 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi.yaml"](1, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=1 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi"](2, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: skip non-matching subdirectories: registered=2 requested=3 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["openapi","v3"](3, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match shares to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed shares čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","shares"](4, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match reconfigure to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed reconfigure čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","reconfigure"](5, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match diag to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed diag čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","diag"](6, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match ping to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed ping čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","ping"](7, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match licenses to jobs: FAILURE čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed licenses čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","licenses"](8, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: method skip for ["slurm","v0.0.41","job","submit"](9, GET != POST) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","job","submit"](9, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: method skip for ["slurm","v0.0.41","job","allocate"](10, GET != POST) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match failed for ["slurm","v0.0.41","job","allocate"](10, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match slurm to slurm: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match v0.0.41 to v0.0.41: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path: string attempt match jobs to jobs: SUCCESS čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _match_path_from_data: match successful for ["slurm","v0.0.41","jobs"](11, GET) to ["slurm","v0.0.41","jobs"](0x7F9C64001CB0) čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: operations_router: [[localhost]:52909] found callback handler: (0x0) callback_tag=0 path=/slurm/v0.0.41/jobs parser=data_parser/v0.0.41 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: _resolve_mime: [[localhost]:52909] did not provide a known content type header. Assuming URL encoded. čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug5: _parse_http_accept_entry: found */* with q=1.000000 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: _resolve_mime: [[localhost]:52909] accepts */* with q=1.000000 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: _resolve_mime: [[localhost]:52909] found accepts */*=application/json with q=1.000000 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug3: _resolve_mime: [[localhost]:52909] mime read: application/x-www-form-urlencoded write: application/json čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug3: _call_handler: [[localhost]:52909] BEGIN: calling ctxt handler: 0x7F9C9D294A36[0] for path: /slurm/v0.0.41/jobs čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug: wrap_openapi_ctxt_callback: [[localhost]:52909] GET using data_parser/v0.0.41 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x1 from 0x408376 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x408376 from 0x1 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug: accounting_storage/slurmdbd: _connect_dbd_conn: Sent PersistInit msg čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x1 from 0x408376 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: debug4: xsignal: Swap signal PIPE[13] to 0x408376 from 0x1 čec 24 14:37:55 slurmserver2.koios.lan slurmrestd[1502900]: malloc(): unsorted double linked list corrupted čec 24 14:37:55 slurmserver2.koios.lan systemd[1]: Started Process Core Dump (PID 1502951/UID 0).

2 3

Slurm versions 24.05.2, 23.11.9, and 23.02.8 are now available (security fix for switch plugins)
by Tim Wickberg 31 Jul '24

31 Jul '24

Slurm versions 24.05.2, 23.11.9, and 23.02.8 are now available and include a fix for a recently discovered security issue with the switch plugins. SchedMD customers were informed on July 17th and provided a patch on request; this process is documented in our security policy. [1] For the switch/hpe_slingshot and switch/nvidia_imex plugins, a user could override the isolation between Slingshot VNIs or IMEX channels. If you do not have one of these switch plugins configured, then you are not impacted by this issue. It is unclear what, if any, information could be accessed with access to an unauthorized channel. This disclosure is being made out of an abundance of caution. If you do have one of these plugins enabled, the slurmctld must be restarted before the slurmd daemons to avoid disruption. Downloads are available at https://www.schedmd.com/downloads.php . Release notes follow below. - Tim [1] https://www.schedmd.com/security-policy/ -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support > * Changes in Slurm 24.05.2 > ========================== > -- Fix energy gathering rpc counter underflow in _rpc_acct_gather_energy when > more than 10 threads try to get energy at the same time. This prevented > the possibility to get energy from slurmd by any step until slurmd was > restarted, so losing energy accounting metrics in the node. > -- accounting_storage/mysql - Fix issue where new user with wckey did not > have a default wckey sent to the slurmctld. > -- slurmrestd - Prevent slurmrestd segfault when handling the following > endpoints when none of the optional parameters are specified: > 'DELETE /slurm/v0.0.40/jobs' > 'DELETE /slurm/v0.0.41/jobs' > 'GET /slurm/v0.0.40/shares' > 'GET /slurm/v0.0.41/shares' > 'GET /slurmdb/v0.0.40/instance' > 'GET /slurmdb/v0.0.41/instance' > 'GET /slurmdb/v0.0.40/instances' > 'GET /slurmdb/v0.0.41/instances' > 'POST /slurm/v0.0.40/job/{job_id}' > 'POST /slurm/v0.0.41/job/{job_id}' > -- Fix IPMI energy gathering when no IPMIPowerSensors are specified in > acct_gather.conf. This situation resulted in an accounted energy of 0 > for job steps. > -- Fix a minor memory leak in slurmctld when updating a job dependency. > -- scontrol,squeue - Fix regression that caused incorrect values for > multisocket nodes at '.jobs[].job_resources.nodes.allocation' for > 'scontrol show jobs --(json|yaml)' and 'squeue --(json|yaml)'. > -- slurmrestd - Fix regression that caused incorrect values for > multisocket nodes at '.jobs[].job_resources.nodes.allocation' to be dumped > with endpoints: > 'GET /slurm/v0.0.41/job/{job_id}' > 'GET /slurm/v0.0.41/jobs' > -- jobcomp/filetxt - Fix truncation of job record lines > 1024 characters. > -- Fixed regression that prevented compilation on FreeBSD hosts. > -- switch/hpe_slingshot - Drain node on failure to delete CXI services. > -- Fix a performance regression from 23.11.0 in cpu frequency handling when no > CpuFreqDef is defined. > -- Fix one-task-per-sharing not working across multiple nodes. > -- Fix inconsistent number of cpus when creating a reservation using the > TRESPerNode option. > -- data_parser/v0.0.40+ - Fix job state parsing which could break filtering. > -- Prevent cpus-per-task to be modified in jobs where a -c value has been > explicitly specified and the requested memory constraints implicitly > increase the number of CPUs to allocate. > -- slurmrestd - Fix regression where args '-s v0.0.39,dbv0.0.39' and > '-d v0.0.39' would result in 'GET /openapi/v3' not registering as a valid > possible query resulting in 404 errors. > -- slurmrestd - Fix memory leak for dbv0.0.39 jobs query which occurred if the > query parameters specified account, association, cluster, constraints, > format, groups, job_name, partition, qos, reason, reservation, state, users, > or wckey. This affects the following endpoints: > 'GET /slurmdb/v0.0.39/jobs' > -- slurmrestd - In the case the slurmdbd does not respond to a persistent > connection init message, prevent the closed fd from being used, and instead > emit an error or warning depending on if the connection was required. > -- Fix 24.05.0 regression that caused the slurmdbd not to send back an error > message if there is an error initializing a persistent connection. > -- Reduce latency of forwarded x11 packets. > -- Add "curr_dependency" (representing the current dependency of the job) > and "orig_dependency" (representing the original requested dependency of > the job) fields to the job record in job_submit.lua (for job update) and > jobcomp.lua. > -- Fix potential segfault of slurmctld configured with > SlurmctldParameters=enable_rpc_queue from happening on reconfigure. > -- Fix potential segfault of slurmctld on its shutdown when rate limitting > is enabled. > -- slurmrestd - Fix missing job environment for SLURM_JOB_NAME, > SLURM_OPEN_MODE, SLURM_JOB_DEPENDENCY, SLURM_PROFILE, SLURM_ACCTG_FREQ, > SLURM_NETWORK and SLURM_CPU_FREQ_REQ to match sbatch. > -- Add missing bash-completions dependency to slurm-smd-client debian package. > -- Fix bash-completions installation in debian pacakges. > -- Fix GRES environment variable indices being incorrect when only using a > subset of all GPUs on a node and the --gres-flags=allow-task-sharing option > -- Add missing mariadb/mysql client package dependency to debian package. > -- Fail the debian package build early if mysql cannot be found. > -- Prevent scontrol from segfaulting when requesting scontrol show reservation > --json or --yaml if there is an error retrieving reservations from the > slurmctld. > -- switch/hpe_slingshot - Fix security issue around managing VNI access. > -- switch/nvidia_imex - Fix security issue managing IMEX channel access. > -- switch/nvidia_imex - Allow for compatibility with job_container/tmpfs. > * Changes in Slurm 23.11.9 > ========================== > -- Fix many commands possibly reporting an "Unexpected Message Received" when > in reality the connection timed out. > -- Fix heterogeneous job components not being signaled with scancel --ctld and > 'DELETE slurm/v0.0.40/jobs' if the job ids are not explicitly given, > the heterogeneous job components match the given filters, and the > heterogeneous job leader does not match the given filters. > -- Fix regression from 23.02 impeding job licenses from being cleared. > -- Move error to log_flag which made _get_joules_task error to be logged to the > user when too many rpcs were queued in slurmd for gathering energy. > -- slurmrestd - Prevent a slurmrestd segfault when modifying an association > without specifying max TRES limits in the request if those TRES > limits are currently defined in the association. This affects the following > fields of endpoint 'POST /slurmdb/v0.0.38/associations/': > 'associations/max/tres/per/job' > 'associations/max/tres/per/node' > 'associations/max/tres/total' > 'associations/max/tres/minutes/per/job' > 'associations/max/tres/minutes/total' > -- Fix power_save operation after recovering from a failed reconfigure. > -- scrun - Delay shutdown until after start requested. This caused scrun > to never start or shutdown and hung forever when using --tty. > -- Fix backup slurmctld potentially not running the agent when taking over as > the primary controller. > -- Fix primary controller not running the agent when a reconfigure of the > slurmctld fails. > -- jobcomp/{elasticsearch,kafka} - Avoid sending fields with invalid date/time. > -- Fix energy gathering rpc counter underflow in _rpc_acct_gather_energy when > more than 10 threads try to get energy at the same time. This prevented > the possibility to get energy from slurmd by any step until slurmd was > restarted, so losing energy accounting metrics in the node. > -- slurmrestd - Fix memory leak for dbv0.0.39 jobs query which occurred if the > query parameters specified account, association, cluster, constraints, > format, groups, job_name, partition, qos, reason, reservation, state, users, > or wckey. This affects the following endpoints: > 'GET /slurmdb/v0.0.39/jobs' > -- switch/hpe_slingshot - Fix security issue around managing VNI access. > * Changes in Slurm 23.02.8 > ========================== > -- Fix rare deadlock when a dynamic node registers at the same time that a > once per minute background task occurs. > -- Fix assertion in developer mode on a failed message unpack. > -- switch/hpe_slingshot - Fix security issue around managing VNI access.

1 0

slurmd error: port already in use, resulting in slaves not being able to communicate with master slurmctld
by Shooktija S N 30 Jul '24

30 Jul '24

Hi, I'm trying to set up a Slurm (version 22.05.8) cluster consisting of 3 nodes with these hostnames and local IP addresses: server1 - 10.36.17.152 server2 - 10.36.17.166 server3 - 10.36.17.132 I had scrambled together a minimum working example using these resources: https://github.com/SergioMEV/slurm-for-dummies https://blog.devops.dev/slurm-complete-guide-a-to-z-concepts-setup-and-trou… For a while everything looked fine and I was able to run the command I usually use to see if everything is fine: srun --label --nodes=3 hostname Which used to show the expected output of the hostnames of all 3 computers, namely: server1, server2, and server3. However - after having made no changes to the configs - the command no longer works if I specify the number of nodes as anything more than 1, this behaviour is consistent on all 3 computers, the output of 'sinfo' is also included below: root@server1:~# srun --nodes=1 hostnameserver1root@server1:~# root@server1:~# srun --nodes=3 hostnamesrun: Required node not available (down, drained or reserved)srun: job 312 queued and waiting for resources^Csrun: Job allocation 312 has been revokedsrun: Force Terminated JobId=312root@server1:~# root@server1:~# ssh server2 "srun --nodes=1 hostname"server1root@server1:~# root@server1:~# ssh server2 "srun --nodes=3 hostname"srun: Required node not available (down, drained or reserved)srun: job 314 queued and waiting for resources^Croot@server1:~# root@server1:~# root@server1:~# sinfoPARTITION AVAIL TIMELIMIT NODES STATE NODELISTmainPartition* up infinite 2 down* server[2-3]mainPartition* up infinite 1 idle server1root@server1:~# Turns out, slurmctld on the master node (hostname: server1) and slurmd on the slave nodes (hostnames: server2 & server3) are throwing some errors probably related to networking: A few lines before and after the first occurence of the error in slurmctld.log on the master node - it's the only type of error I have noticed in the logs (pastebin to the entire log <https://pastebin.com/GBSWXZJR>): root@server1:/var/log# grep -B 20 -A 5 -m1 -i "error" slurmctld.log[2024-07-26T13:13:49.579] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 1 partitions[2024-07-26T13:13:49.580] debug: power_save module disabled, SuspendTime < 0[2024-07-26T13:13:49.580] Running as primary controller[2024-07-26T13:13:49.580] debug: No backup controllers, not launching heartbeat.[2024-07-26T13:13:49.580] debug: priority/basic: init: Priority BASIC plugin loaded[2024-07-26T13:13:49.580] No parameter for mcs plugin, default values set[2024-07-26T13:13:49.580] mcs: MCSParameters = (null). ondemand set.[2024-07-26T13:13:49.580] debug: mcs/none: init: mcs none plugin loaded[2024-07-26T13:13:49.580] debug2: slurmctld listening on 0.0.0.0:6817[2024-07-26T13:13:52.662] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded[2024-07-26T13:13:52.662] debug2: Processing RPC: MESSAGE_NODE_REGISTRATION_STATUS from UID=0[2024-07-26T13:13:52.662] debug: gres/gpu: init: loaded[2024-07-26T13:13:52.662] debug: validate_node_specs: node server1 registered with 0 jobs[2024-07-26T13:13:52.662] debug2: _slurm_rpc_node_registration complete for server1 usec=229[2024-07-26T13:13:53.586] debug: Spawning registration agent for server[2-3] 2 hosts[2024-07-26T13:13:53.586] SchedulerParameters=default_queue_depth=100,max_rpc_cnt=0,max_sched_time=2,partition_job_depth=0,sched_max_job_start=0,sched_min_interval=2[2024-07-26T13:13:53.586] debug: sched: Running job scheduler for default depth.[2024-07-26T13:13:53.586] debug2: Spawning RPC agent for msg_type REQUEST_NODE_REGISTRATION_STATUS[2024-07-26T13:13:53.587] debug2: Tree head got back 0 looking for 2[2024-07-26T13:13:53.588] debug2: _slurm_connect: failed to connect to 10.36.17.166:6818: Connection refused[2024-07-26T13:13:53.588] debug2: Error connecting slurm stream socket at 10.36.17.166:6818: Connection refused[2024-07-26T13:13:53.588] debug2: _slurm_connect: failed to connect to 10.36.17.132:6818: Connection refused[2024-07-26T13:13:53.588] debug2: Error connecting slurm stream socket at 10.36.17.132:6818: Connection refused[2024-07-26T13:13:54.588] debug2: _slurm_connect: failed to connect to 10.36.17.166:6818: Connection refused[2024-07-26T13:13:54.588] debug2: Error connecting slurm stream socket at 10.36.17.166:6818: Connection refused[2024-07-26T13:13:54.589] debug2: _slurm_connect: failed to connect to 10.36.17.132:6818: Connection refused The connections to 10.36.17.166:6818 and 10.36.17.132:6818 are refused. Those are ports specified by the 'SlurmdPort' key in slurm.conf There are similar errors in the slurmd.log files on both the slave nodes as well: slurmd.log on server2, the error is only at the end of the file (pastebin to the entire log <https://pastebin.com/TwSMiAp7>): root@server2:/var/log# tail -5 slurmd.log [2024-07-26T13:13:53.018] debug: mpi/pmix_v4: init: PMIx plugin loaded[2024-07-26T13:13:53.018] debug: mpi/pmix_v4: init: PMIx plugin loaded[2024-07-26T13:13:53.018] debug2: No mpi.conf file (/etc/slurm/mpi.conf)[2024-07-26T13:13:53.018] error: Error binding slurm stream socket: Address already in use[2024-07-26T13:13:53.018] error: Unable to bind listen port (6818): Address already in use slurmd.log on server3 (pastebin to the entire log <https://pastebin.com/K55cAGLb>): root@server3:/var/log# tail -5 slurmd.log [2024-07-26T13:13:53.383] debug: mpi/pmix_v4: init: PMIx plugin loaded[2024-07-26T13:13:53.383] debug: mpi/pmix_v4: init: PMIx plugin loaded[2024-07-26T13:13:53.383] debug2: No mpi.conf file (/etc/slurm/mpi.conf)[2024-07-26T13:13:53.384] error: Error binding slurm stream socket: Address already in use[2024-07-26T13:13:53.384] error: Unable to bind listen port (6818): Address already in use I use this script to restart slurm whenever I change any of the configs, could the order in which these operations are being done cause the problems I'm facing: #! /bin/bashscp /etc/slurm/slurm.conf /etc/slurm/gres.conf server2:/etc/slurm/ && echo copied slurm.conf and gres.conf to server2;scp /etc/slurm/slurm.conf /etc/slurm/gres.conf server3:/etc/slurm/ && echo copied slurm.conf and gres.conf to server3;echoecho restarting slurmctld and slurmd on server1(scontrol shutdown ; sleep 3 ; rm -f /var/log/slurmd.log /var/log/slurmctld.log ; slurmctld -d ; sleep 3 ; slurmd) && echo doneecho restarting slurmd on server2(ssh server2 "rm -f /var/log/slurmd.log /var/log/slurmctld.log ; slurmd") && echo doneecho restarting slurmd on server3(ssh server3 "rm -f /var/log/slurmd.log /var/log/slurmctld.log ; slurmd") && echo done Config files: slurm.conf without the comments: root@server1:/etc/slurm# grep -v "#" slurm.conf ClusterName=DlabClusterSlurmctldHost=server1GresTypes=gpuProctrackType=proctrack/linuxprocReturnToService=1SlurmctldPidFile=/var/run/slurmctld.pidSlurmctldPort=6817SlurmdPidFile=/var/run/slurmd.pidSlurmdPort=6818SlurmdSpoolDir=/var/spool/slurmdSlurmUser=rootStateSaveLocation=/var/spool/slurmctldTaskPlugin=task/affinity,task/cgroupInactiveLimit=0KillWait=30MinJobAge=300SlurmctldTimeout=120SlurmdTimeout=300Waittime=0SchedulerType=sched/backfillSelectType=select/cons_tresJobCompType=jobcomp/noneJobAcctGatherFrequency=30SlurmctldDebug=debug2SlurmctldLogFile=/var/log/slurmctld.logSlurmdDebug=debug2SlurmdLogFile=/var/log/slurmd.logNodeName=server[1-3] RealMemory=128636 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:1PartitionName=mainPartition Nodes=ALL Default=YES MaxTime=INFINITE State=UP gres.conf: root@server1:/etc/slurm# cat gres.confNodeName=server1 Name=gpu File=/dev/nvidia0NodeName=server2 Name=gpu File=/dev/nvidia0NodeName=server3 Name=gpu File=/dev/nvidia0 These config files are the same on all 3 computers. As a complete beginner to Linux and Slurm administration, I have been struggling to understand even the most basic documentation, and I have been unable to find answers online. Any assistance would be greatly appreciated. Thanks!

1 1

slurmctld hourly: Unexpected missing socket error
by Jason Ellul 29 Jul '24

29 Jul '24

Hi all, I am hoping someone can help with our problem. Every hour after restarting slurmctld the controller becomes unresponsive to commands for 1 sec, reporting errors such as: [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934767]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934760]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934875]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934906]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[939016]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error It occurs consistently at around the hour mark, but generally not at other times, unless we run a reconfigure or restart the controller. We don’t see any issues in the slurmdbd.log and the errors are also always msg type RESPONSE. We have tried building a new server on different infrastructure, but the problem has persisted. Yesterday we even tried updating slurm to v24.05.1 in the hope that may provide a fix. During our troubleshooting we have: Set: * SchedulerParameters = max_rpc_cnt=400,sched_min_interval=50000,sched_max_job_start=300,batch_sched_delay=20,bf_resolution=600,bf_min_prio_reserve=2000,bf_min_age_reserve=600 * SlurmctldPort = 6808-6817 But although the stats in sdiag have improved we still see the errors. On our monitoring software we also see a drop in network and disk activity during this 1 second, always at approx. 1 hour after restarting the controller. Many Thanks in advance Jason Jason Ellul Head - Research Computing Facility Office of Cancer Research Peter MacCallum Cancer Centre

2 4

Final Call for SLUG Standard Registration
by Victoria Hobson 29 Jul '24

29 Jul '24

Slurm User Group (SLUG) 2024 is set for September 12-13 at the University of Oslo in Oslo, Norway. Registration information, abstracts, and travel recommendations can be found here:https://slug24.splashthat.com/ The last day to register with standard pricing ($900) is this Friday, August 2nd. After this, final registration will run until August 30th at a price of $1100. SLUG is the best way to interact with the Slurm community and to interact with the SchedMD Support & Training staff. Don't forget to register. We can't wait to see you in Oslo! -- Victoria Hobson SchedMD LLC Vice President of Marketing

1 0

Convergence of Kube and Slurm?
by Dan Healy 29 Jul '24

29 Jul '24

Bright Cluster Manager has some verbiage on their marketing site that they can manage a cluster running both Kubernetes and Slurm. Maybe I misunderstood it. But nevertheless, I am encountering groups more frequently that want to run a stack of containers that need private container networking. What’s the current state of using the same HPC cluster for both Slurm and Kube? Note: I’m aware that I can run Kube on a single node, but we need more resources. So ultimately we need a way to have Slurm and Kube exist in the same cluster, both sharing the full amount of resources and both being fully aware of resource usage. Thanks, Daniel Healy

5 4

Slurm fails before nvidia-smi command
by Aziz Ogutlu 29 Jul '24

29 Jul '24

Hi there all, We have Dell server with 2 x Nvidia H100 and running slurm on it. After restart server if we do not write nvidia-smi command slurm fails. When we run nvidia-smi && systemctl restart slurmd && systemctl restart slurmctld , slurm queue begins. Do you have any idea about this error and what can we do for this issue? -- Best regards, Aziz Öğütlü Eduline Bilişim Sanayi ve Ticaret Ltd. Şti. www.eduline.com.tr Merkez Mah. Ayazma Cad. No:37 Papirus Plaza Kat:6 Ofis No:118 Kağıthane - İstanbul - Türkiye 34406 Tel : +90 212 324 60 61 Cep: +90 541 350 40 72

4 4

Slurm sacct ResvCPURAW invalid field in version 24.12.5
by Mike Mikailov 29 Jul '24

29 Jul '24

Dear All, Does anyone know what is the equivalent field for ResvCPURAW of sacct command of Slurm in its version 23.11.5. I get: sacct: error: Invalid field requested “ResvCPURAW”. Your help at your earliest is very much appreciated. Best, -Mike Sent from my iPhone

2 1

Background tasks in Slurm scripts?
by Steffen Grunewald 26 Jul '24

26 Jul '24

Good morning, yesterday I came across a Slurm (sbatch) script that, after doing some stuff in the foreground, runs another executable in the background - and doesn't "wait" for it to finish - literally the last line of the script is executable & (and that executable is supposed to take several 10 seconds or more to finish) How would Slurm handle this? Will the end of the script immediately trigger the job epilog, and what would happen to the leftover task? This certainly is discussed somewhere in the manual pages and other documentation but up to now I failed to find that place... Thanks, Steffen -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~

3 3

API - Specify GPUs
by jpuerto＠psc.edu 26 Jul '24

26 Jul '24

Good Afternoon All, It does not seem that the REST API allows for folks to configure their jobs to utilize GPUs, using the traditional methods. IE, there does not appear to be an equivalent between the --gpus (or --gres) flag on sbatch/srun and the REST API's job submission endpoint. Can anyone point me towards what should be used if the version we are on does not support tres specifications? Best regards, Juan Puerto

2 1

Error binding slurm stream socket: Address already in use, and GPU GRES verification
by Shooktija S N 23 Jul '24

23 Jul '24

Hi, I am trying to set up Slurm with GPUs as GRES on a 3 node configuration (hostnames: server1, server2, server3). For a while everything looked fine and I was able to run srun --label --nodes=3 hostname which is what I use to test if Slurm is working correctly, and then it randomly stops. Turns out slurmctld is not working and it throws the following error (the two lines are consecutive in the log file): root@server1:/var/log# grep -i error slurmctld.log [2024-07-22T14:47:32.302] error: Error binding slurm stream socket: Address already in use[2024-07-22T14:47:32.302] fatal: slurm_init_msg_engine_port error Address already in use This error is being thrown after having made no changes to the config files, in fact the cluster wasn't used at all for a few weeks before this error was thrown. This is the simple script I use to restart Slurm: root@server1:~# cat slurmRestart.sh #! /bin/bashscp /etc/slurm/slurm.conf server2:/etc/slurm/ && echo copied slurm.conf to server2;scp /etc/slurm/slurm.conf server3:/etc/slurm/ && echo copied slurm.conf to server3;rm /var/log/slurmd.log /var/log/slurmctld.log ; systemctl restart slurmd slurmctld ; echo restarting slurm on server1;(ssh server2 "rm /var/log/slurmd.log /var/log/slurmctld.log ; systemctl restart slurmd slurmctld") && echo restarting slurm on server2;(ssh server3 "rm /var/log/slurmd.log /var/log/slurmctld.log ; systemctl restart slurmd slurmctld") && echo restarting slurm on server3; Could the error be due to the slurmd and/or slurmctld not being started in the right order? The other question I have is regarding the configuration of a GPU as a GRES - how do I verify that it has been configured correctly? I was told to use srun nvidia-smi with and without having enabled GPU use, but whether or not I enable GPU usage has no effect on the output of the command: root@server1:~# srun --nodes=1 nvidia-smi --query-gpu=uuid --format=csvuuidGPU-55f127a8-dbf4-fd12-3cad-c0d5f2dcb005root@server1:~# root@server1:~# srun --nodes=1 --gpus-per-node=1 nvidia-smi --query-gpu=uuid --format=csvuuidGPU-55f127a8-dbf4-fd12-3cad-c0d5f2dcb005 I am sceptical if about whether the GPU has properly been configured, is this the best way to check if it has? *The error:* I first noticed this happening when I tried to run the command I usually use to see if everything is fine, the srun command runs only one node, and the only way to stop it if I specify the number of nodes as 3 is to press Ctrl+C: root@server1:~# srun --label --nodes=1 hostname0: server1root@server1:~# ssh server2 "srun --label --nodes=1 hostname"0: server1root@server1:~# ssh server3 "srun --label --nodes=1 hostname"0: server1root@server1:~# srun --label --nodes=3 hostnamesrun: Required node not available (down, drained or reserved)srun: job 265 queued and waiting for resources^Csrun: Job allocation 265 has been revokedsrun: Force Terminated JobId=265root@server1:~# ssh server2 "srun --label --nodes=3 hostname"srun: Required node not available (down, drained or reserved)srun: job 266 queued and waiting for resources^Croot@server1:~# ssh server3 "srun --label --nodes=3 hostname"srun: Required node not available (down, drained or reserved)srun: job 267 queued and waiting for resourcesroot@server1:~# *The logs:* 1) The last 30 lines of */var/log/slurmctld.log* at the debug5 level in server #1 (pastebin to the entire log <https://pastebin.com/fw4C4xtr>): root@server1:/var/log# tail -30 slurmctld.log [2024-07-22T14:47:32.301] debug: Updating partition uid access list[2024-07-22T14:47:32.301] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/resv_state` as buf_t[2024-07-22T14:47:32.301] debug3: Version string in resv_state header is PROTOCOL_VERSION[2024-07-22T14:47:32.301] Recovered state of 0 reservations[2024-07-22T14:47:32.301] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/trigger_state` as buf_t[2024-07-22T14:47:32.301] State of 0 triggers recovered[2024-07-22T14:47:32.301] read_slurm_conf: backup_controller not specified[2024-07-22T14:47:32.301] select/cons_tres: select_p_reconfigure: select/cons_tres: reconfigure[2024-07-22T14:47:32.301] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 1 partitions[2024-07-22T14:47:32.301] debug: power_save module disabled, SuspendTime < 0[2024-07-22T14:47:32.301] Running as primary controller[2024-07-22T14:47:32.301] debug: No backup controllers, not launching heartbeat.[2024-07-22T14:47:32.301] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/priority_basic.so[2024-07-22T14:47:32.301] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Priority BASIC plugin type:priority/basic version:0x160508[2024-07-22T14:47:32.301] debug: priority/basic: init: Priority BASIC plugin loaded[2024-07-22T14:47:32.301] debug3: Success.[2024-07-22T14:47:32.301] No parameter for mcs plugin, default values set[2024-07-22T14:47:32.301] mcs: MCSParameters = (null). ondemand set.[2024-07-22T14:47:32.301] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/mcs_none.so[2024-07-22T14:47:32.301] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:mcs none plugin type:mcs/none version:0x160508[2024-07-22T14:47:32.301] debug: mcs/none: init: mcs none plugin loaded[2024-07-22T14:47:32.301] debug3: Success.[2024-07-22T14:47:32.302] debug3: _slurmctld_rpc_mgr pid = 3159324[2024-07-22T14:47:32.302] debug3: _slurmctld_background pid = 3159324[2024-07-22T14:47:32.302] error: Error binding slurm stream socket: Address already in use[2024-07-22T14:47:32.302] fatal: slurm_init_msg_engine_port error Address already in use[2024-07-22T14:47:32.304] slurmscriptd: debug3: Called _handle_close[2024-07-22T14:47:32.304] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.304] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.304] slurmscriptd: debug: _slurmscriptd_mainloop: finished 2) Entirety of *slurmctld.log on server #2*: root@server2:/var/log# cat slurmctld.log [2024-07-22T14:47:32.614] debug: slurmctld log levels: stderr=debug5 logfile=debug5 syslog=quiet[2024-07-22T14:47:32.614] debug: Log file re-opened[2024-07-22T14:47:32.615] slurmscriptd: debug: slurmscriptd: Got ack from slurmctld, initialization successful[2024-07-22T14:47:32.615] slurmscriptd: debug: _slurmscriptd_mainloop: started[2024-07-22T14:47:32.616] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.616] debug: slurmctld: slurmscriptd fork()'d and initialized.[2024-07-22T14:47:32.616] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.616] debug: _slurmctld_listener_thread: started listening to slurmscriptd[2024-07-22T14:47:32.616] debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.616] debug3: Called _msg_readable[2024-07-22T14:47:32.616] slurmctld version 22.05.8 started on cluster dlabcluster[2024-07-22T14:47:32.616] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/cred_munge.so[2024-07-22T14:47:32.616] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x160508[2024-07-22T14:47:32.616] cred/munge: init: Munge credential signature plugin loaded[2024-07-22T14:47:32.616] debug3: Success.[2024-07-22T14:47:32.616] error: This host (server2/server2) not a valid controller[2024-07-22T14:47:32.617] slurmscriptd: debug3: Called _handle_close[2024-07-22T14:47:32.617] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.617] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.617] slurmscriptd: debug: _slurmscriptd_mainloop: finished 3) Entirety of *slurmctld.log on server #3*: root@server3:/var/log# cat slurmctld.log [2024-07-22T14:47:32.927] debug: slurmctld log levels: stderr=debug5 logfile=debug5 syslog=quiet[2024-07-22T14:47:32.927] debug: Log file re-opened[2024-07-22T14:47:32.928] slurmscriptd: debug: slurmscriptd: Got ack from slurmctld, initialization successful[2024-07-22T14:47:32.928] slurmscriptd: debug: _slurmscriptd_mainloop: started[2024-07-22T14:47:32.928] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.928] debug: slurmctld: slurmscriptd fork()'d and initialized.[2024-07-22T14:47:32.928] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.928] slurmctld version 22.05.8 started on cluster dlabcluster[2024-07-22T14:47:32.929] debug: _slurmctld_listener_thread: started listening to slurmscriptd[2024-07-22T14:47:32.929] debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.929] debug3: Called _msg_readable[2024-07-22T14:47:32.929] debug3: Trying to load plugin /usr/lib/x86_64-linux-gnu/slurm-wlm/cred_munge.so[2024-07-22T14:47:32.929] debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x160508[2024-07-22T14:47:32.929] cred/munge: init: Munge credential signature plugin loaded[2024-07-22T14:47:32.929] debug3: Success.[2024-07-22T14:47:32.929] error: This host (server3/server3) not a valid controller[2024-07-22T14:47:32.930] slurmscriptd: debug3: Called _handle_close[2024-07-22T14:47:32.930] slurmscriptd: debug4: eio: handling events for 1 objects[2024-07-22T14:47:32.930] slurmscriptd: debug3: Called _msg_readable[2024-07-22T14:47:32.930] slurmscriptd: debug: _slurmscriptd_mainloop: finished *The config files (shared by all 3 computers):* 1) */etc/slurm/slurm.conf* without the comments: root@server1:/etc/slurm# grep -v "#" slurm.conf ClusterName=DlabClusterSlurmctldHost=server1GresTypes=gpuProctrackType=proctrack/linuxprocReturnToService=1SlurmctldPidFile=/var/run/slurmctld.pidSlurmctldPort=6817SlurmdPidFile=/var/run/slurmd.pidSlurmdPort=6818SlurmdSpoolDir=/var/spool/slurmdSlurmUser=rootStateSaveLocation=/var/spool/slurmctldTaskPlugin=task/affinity,task/cgroupInactiveLimit=0KillWait=30MinJobAge=300SlurmctldTimeout=120SlurmdTimeout=300Waittime=0SchedulerType=sched/backfillSelectType=select/cons_tresJobCompType=jobcomp/noneJobAcctGatherFrequency=30SlurmctldDebug=debug5SlurmctldLogFile=/var/log/slurmctld.logSlurmdDebug=debug5SlurmdLogFile=/var/log/slurmd.logNodeName=server[1-3] RealMemory=128636 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:1PartitionName=mainPartition Nodes=ALL Default=YES MaxTime=INFINITE State=UP 2) */etc/slurm/gres.conf*: root@server1:/etc/slurm# cat gres.conf NodeName=server1 Name=gpu File=/dev/nvidia0NodeName=server2 Name=gpu File=/dev/nvidia0NodeName=server3 Name=gpu File=/dev/nvidia0 These files are the same on all 3 computers: root@server1:/etc/slurm# diff slurm.conf <(ssh server2 "cat /etc/slurm/slurm.conf")root@server1:/etc/slurm# diff slurm.conf <(ssh server3 "cat /etc/slurm/slurm.conf")root@server1:/etc/slurm# diff gres.conf <(ssh server2 "cat /etc/slurm/gres.conf")root@server1:/etc/slurm# diff gres.conf <(ssh server3 "cat /etc/slurm/gres.conf")root@server1:/etc/slurm# Thank you, Shookti

1 0

Cgroup
by stth 22 Jul '24

22 Jul '24

Hello, I am configuring cgroups on my server for the first time. I've created a cgroup.conf file in the Slurm directory with the following values: ConstrainCores=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes AllowedRAMSpace=90 AllowedSwapSpace=10 I feel like this configuration might be incomplete. Can anyone provide assistance? I haven't found any specific guidance online. Thanks!

2 1

Custom Plugin Integration
by Bhaskar Chakraborty 20 Jul '24

20 Jul '24

Hello, We wish to have a schedulingintegration with Slurm. Our own application has a backend system which willdecide the placement of jobs across hosts & CPU cores. The backend takesits own time to come back with a placement (which may take a few seconds) & we expect slurm to update it regularly about any change in the current state ofavailable resources. For this we believe we have 3 options broadly: - We use the const_tres Select plugin & modify it to let it query our backend system for job placements. - We write our own Select plugin avoiding any other Select plugin. - We use existing select plugin & also register our own plugin. Idea is that our plugin will cater to 'our' jobs (specific partition, say) while all other jobs would be taken up by the default plugin. Problem with a> is that this leads to modification ofexisting plugin code & calling (our) library code from inside Select pluginlib. With b> the issue is unless we have the full Slurm cluster toourselves this isn't viable. Any insight how to proceed with this? Where shouldour select plugin, assuming we need to make one, fits in the slurm integration. We are not sure whether c> is allowed in Slurm. We went through existing Select plugins Linear & cons_tres.However, not able to figure out how to use them or write something on similarlines to suit our purpose. Any help in this regard is appreciated. Apologies if this question (or any other very similar) is already answered,please point to the relevant thread then. Thanks in advance for any pointers. Regards, Bhaskar.

3 7

CLOUD nodes with unknown IP addresses
by Martin Lee 19 Jul '24

19 Jul '24

I've read the following in the slurm power saving docs: https://slurm.schedmd.com/power_save.html > *cloud_dns* > By default, Slurm expects that the network addresses for cloud nodes won't be known until creation of the node and that Slurm will be notified of the node's address upon registration. Since Slurm communications rely on the node configuration found in the slurm.conf, Slurm will tell the client command, after waiting for all nodes to boot, each node's IP address. However, in environments where the nodes are in DNS, this step can be avoided by configuring this option. > I am creating the nodes on demand and don't know the IP ahead of the instance start, so cloud_dns is not set. I'm confused specifically by "Slurm will be notified of the node's address upon registration." Who/what is expected to do this? If it is expected to be performed by the ResumeProgram, does it need to be done before slurmd starts on the node? Is it OK if the node does it after slurmd has started with something like: scontrol update nodename=$(hostname -s) nodeaddr=$(hostname -I) nodehostname=$(hostname) scontrol reconfigure Thank you, Martin

2 2

Slurm Users cannot open gui login nodes with X11 without logging into the Compute node.
by Alper AYKUT 18 Jul '24

18 Jul '24

Hello 1 Login Server I have a total of 3 calculation nodes. Especially in non-GUI jobs, I can run scripts successfully with Slurm sbatch. However, when it is with Gui, I run it with X11. However, my problem for now is this: When I run sbatch gui.sh, it runs smoothly and automatically selects the calculation node and gui is active on the login server. But the node it selects must be logged in with the same user for x11 to work. How can I run the application with gui on the login node without logging in on the calculation node? The calculation node gives the following error when I run the script without logging in. How can I solve this problem. srun: error: Ignoring --x11 option for a job step within an existing job. Set x11 options at job allocation time. srun: error: tutelservernode01: task 0: Exited with exit code 255 My test Script ; #!/bin/bash #SBATCH --partition=debug #Kullanılacak partition (compute, debug, vb.) #SBATCH --job-name=xclock # İşin adı #SBATCH --output=xclock-%j.log # İşin çıktı dosyası #SBATCH --error=xclock-%j.err # İşin hata çıktı dosyası # Slurm'un hangi node'dan işlemi çalıştıracağını seçmesine izin verin # --nodelist parametresi kullanılmazsa, Slurm otomatik olarak bir node seçecektir. # xclock komutunu başlatmak için DISPLAY ortam değişkenini ayarlayın #export DISPLAY=localhost:10.0 # MobaXterm veya başka bir X sunucusunun adresine göre ayarlayın export DISPLAY=$DISPLAY # xclock komutunu sbatch içinde başlatın srun --x11 xclock thankyou for support.

1 0

Nodes TRES double what is requested
by jack.mellor＠mbda.co.uk 12 Jul '24

12 Jul '24

Hi, We are running slurm 23.02.6. Our nodes have hyperthreading disabled and we have slurm.conf set to CPU=32 for each node (each node has 2 processes with 16 cores). When we allocated a job, such as salloc -n 32, it will allocate a whole node but using sinfo shows double the allocation in the TRES=64. It also shows in sinfo that the node has 4294967264 idle CPUs. Not sure if its a known bug, or an issue with our config? I have tried various things, like setting the sockets/boards in slurm.conf. Thanks Jack

5 4

SLURM noob administrator question
by Cutts, Tim 11 Jul '24

11 Jul '24

Still learning about SLURM, so please forgive me if I ask a naïve question I like to use Anders Halager’s gnodes command to visualise the state of our nodes. I’ve noticed lately that we fairly often see things like this (apologies for line wrap): +- core - 46 cores & 186GB -----------------------------------------+-------------------------------------------------------------------+-------------------------------------------------------------------+ | seskscpn301 0G ___OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO | seskscpn309 172G ..........................................!!!! | seskscpn317 0G OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO | Now, you can see in this that nodes 301 and 317 are more or less fully loaded. This is great. But 309 is in an interesting state. Four overloaded cores, and all other cores unused, and plenty of RAM available. And yet SLURM is not scheduling any more work to that node. Right now there are more than 2000 jobs pending, many of which could run on that node. But SLURM is not scheduling them, and I don’t know why. One thing I’ve seen cause this is a job trying to use more CPUs than it has been allocated. The cgroup stops this being a real problem of course, but it does cause the load average to go high. Is this what’s causing SLURM to stop sending anything to the node? Is there a configuration change that might help in this situation? Thanks in advance, Tim -- Tim Cutts Scientific Computing Platform Lead AstraZeneca Find out more about R&D IT Data, Analytics & AI and how we can support you by visiting our Service Catalogue<https://azcollaboration.sharepoint.com/sites/CMU993> | ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com<https://www.astrazeneca.com>

2 1

Replacing MUNGE with SACK (auth/slurm)
by Daniel Letai 11 Jul '24

11 Jul '24

Does SACK replace MUNGE? As in - MUNGE is not required when building Slurm or on compute? If so, can the Requires and BuildRequires for munge be made optional on bcond_without_munge in the spec file? Or is there a reason MUNGE must remain a hard require for Slurm? Thanks, --Dani_L.

1 0

2025

2024

slurm-users ----- 2025 ----- June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users