- slurm-users - lists.schedmd.com

Transfer from GPFS via slurm
by Ratnasamy, Fritz 06 Jun '25

06 Jun '25

Hi, We were told by our hardware provider that large datasets copied from NFS location to GPFS could be conducted via slurm to monitor the transfer. I am not sure of this works as I could not find much online. Otherwise apart from globus and rsync, what would you suggest as a tool/command to copy files from on a cluster between NFS and GPFS? Best, *Fritz Ratnasamy*Data Scientist Information Technology

2 1

Changing account names in sacctmgr
by Burian, John 05 Jun '25

05 Jun '25

All, For reasons I won’t go into, we find we need to change the POSIX group names for all of our users. My understanding is that in the absence of an --account option, sbatch/salloc assumes the account is the user’s primary POSIX group. So, to change group names, we’ll need to change the account names on all the associations in sacctmgr. And ideally, we would change account names in accounting records so we don’t lose accounting history. One is tempted to get into the Slurm DB and start running SQL updates. Is that path too fraught with peril to consider? Or should I just recreate the associations with the new names and take my lumps on the accounting history? John

4 4

error building debug packages on ubuntu 24.04
by Charles Hedrick 03 Jun '25

03 Jun '25

It was unable to find lib/security/pam_slurm_adopt.so It was built as lib64/security, not lib/security. ./debian/slurm-smd-libpam-slurm-adopt.install needs to be lib64/security/pam_slurm_adopt.so More likely, it needs to be built dynamically, so it will work everywhere.

1 0

Slurm version 25.05 is now available
by Tim Wickberg 29 May '25

29 May '25

We are pleased to announce the availability of Slurm 25.05. The release notes summarizing the new features, and including links to the corresponding documentation, can be found at: https://slurm.schedmd.com/release_notes.html A more extensive list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md The Slurm documentation has also been updated to the 25.05 release: https://slurm.schedmd.com Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

Slurm accounts managed via ansible
by Ratnasamy, Fritz 28 May '25

28 May '25

Hi, I was wondering whether there might be built in support for managing slurm accounts,users,associations in ansible. it would be nice to be able to organize accounts in a yaml style file and modify accounts settings via gitlab CI/CD. For example, in a format as such: accounts: - name: "Boss1" parent: "root" description: "Big Boos" fairshare: 20 - name: "UnderBoss1" parent: "root" description: "working for Boss1" fairshare: 1 users: -username: "user1" -username: "user2" *Fritz Ratnasamy* Data Scientist Information Technology

3 2

Account getting duplicated
by Ratnasamy, Fritz 28 May '25

28 May '25

Hi, the slurm db duplicates all our account associations. one with cluster=cluster and another where cluster=venus (which is our actual cluster). is that intended? Or should I make any changes? *Fritz Ratnasamy*Data Scientist Information Technology

2 1

Slurm Reporting Difference between sreport and sacct
by Passant Hafez 23 May '25

23 May '25

Hi all, I was wondering if someone can help explaining this discrepancy. I have different values for project gpu consumption using sreport vs sacct (+ some calculations) This is an example that shows this: sreport -t hours -T gres/gpu cluster AccountUtilizationByuser start=2025-04-01 end=2025-04-05 | grep project1234 gives 178 while sacct -n -X --allusers --accounts=project1234 --start=2025-04-01 --end=2025-04-05 -o elapsedraw,AllocTRES%80,user,partition gives 213480 billing=128,cpu=128,gres/gpu=8,mem=1000G,node=2 gpuplus 249507 billing=128,cpu=128,gres/gpu=8,mem=1000G,node=2 gpuplus 13908 billing=64,cpu=64,gres/gpu=4,mem=500G,node=1 gpuplus 9552 billing=64,cpu=64,gres/gpu=4,mem=500G,node=1 gpuplus 4 billing=16,cpu=16,gres/gpu=1,mem=200G,node=1 gpu 11 billing=16,cpu=16,gres/gpu=1,mem=200G,node=1 gpu ... I will not bore you with the full output and its calculation, but the first job alone consumed 213480 seconds/60/60 * 8 gpus that's 474.4 gpu hours which is way more than the 178 hrs reported by sreport Any clue why these are inconsistent? or how sreport calculated the 178 value? All the best, Passant

3 3

Crash in "slurmd -C" when latest NVIDIA drivers are used
by Taras Shapovalov 23 May '25

23 May '25

Hello, Does someone have idea why "slurmd -C" crashes when it unloads gpu_nrt.so with latest NVIDIA drivers (570 and 575)? We checked, there is no crash in cuda at the moment and gpu_nvml.so works fine, all nvml calls finish successfully, dlclose on gpu_nvml.so works fine. The crash does not depend whether real GPUs present or not. Steps to reproduce: 1. Install Ubuntu 24.04 2. wget https://download.schedmd.com/slurm/slurm-24.11.4.tar.bz2<https://download.schedmd.com/slurm/slurm-24.11.4.tar.bz2;> 3. tar fx ./slurm-24.11.4.tar.bz2 4. cd slurm-24.11.4 5. apt-get install cuda-12-8 hwloc libmunge-dev -y 6. ./configure 7. make && make install 8. Run "slurmd -C", or sometimes "slurmd -vvv -C" to get the crash. Stack trace: #0 0x0000155555544b2a strlen (ld-linux-x86-64.so.2 + 0x28b2a) #1 0x000015555551fc08 __GI__dl_exception_create (ld-linux-x86-64.so.2 + 0x3c08) #2 0x000015555551d298 __GI__dl_signal_error (ld-linux-x86-64.so.2 + 0x1298) #3 0x000015555551e81d _dl_close (ld-linux-x86-64.so.2 + 0x281d) #4 0x000015555551d51c __GI__dl_catch_exception (ld-linux-x86-64.so.2 + 0x151c) #5 0x000015555551d669 _dl_catch_error (ld-linux-x86-64.so.2 + 0x1669) #6 0x0000155554e97c73 _dlerror_run (libc.so.6 + 0x97c73) #7 0x0000155554e979a6 __dlclose (libc.so.6 + 0x979a6) #8 0x0000155555388a25 gpu_plugin_fini (libslurmfull.so + 0x188a25) #9 0x000015555538f2ef gres_get_autodetected_gpus (libslurmfull.so + 0x18f2ef) #10 0x0000555555564828 _print_config (slurmd + 0x10828) #11 0x0000155554e2a1ca __libc_start_call_main (libc.so.6 + 0x2a1ca) #12 0x0000155554e2a28b __libc_start_main_impl (libc.so.6 + 0x2a28b) #13 0x000055555555fc75 _start (slurmd + 0xbc75) I don't really think the problem is in gpu_nrt itself, seems the problem is in memory corruption somewhere else, but I am not sure. The issue is reproduced constantly. A Best regards, Taras

2 3

Slurm 24.11.x + pyxis + TaskProlog -> fail
by Stefan Staeglich 21 May '25

21 May '25

Hi, we are currently evaluating Slurm 24.11. We are using pyxis in our cluster and have encountered the following problem in connection with TaskProlog: https://github.com/NVIDIA/pyxis/issues/163 Do you know more about the problem and the cause? Best, Stefan

1 0

sacctmgr: error: Sending PersistInit msg: Connection refused
by Ratnasamy, Fritz 18 May '25

18 May '25

Hi, We are working on a test cluster with slurm 24.11.3 and I am getting this error message from the login or compute nodes (note that the error does not show when run from the controller node): sacctmgr list associations tree format=cluster,account,user,maxnodes sacctmgr: error: _open_persist_conn: failed to open persistent connection to host:localhost:6819: Connection refused sacctmgr: error: Sending PersistInit msg: Connection refused Does someone know how fix that issue? Thanks, *Fritz Ratnasamy*Data Scientist Information Technology

2 1