Hello,
I’m trying to figure out why we’ve been seeing an increase in network traffic in our AWS-based cluster, which uses Amazon’s parallel cluster tool. After an incident a couple weeks ago, I turned on debug2 logging
on the slurmd processes, and I’m seeing huge numbers of `REQUEST_GETPW` and `REQUEST_GETGR` requests going to the slurmd processes. I briefly turned on debug2 logging for `slurmctld` as well, and I’m seeing lots of RPC calls, but not as many as the `REQUEST_GETPW`
requests that I’ve seen on compute node slurmd processes.
Here’s a sample from the slurmctld log:
```
[2025-04-28T15:11:05.436] debug2: _slurm_rpc_dump_partitions, size=1253 usec=20
[2025-04-28T15:11:05.450] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2971
[2025-04-28T15:11:05.451] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2971
[2025-04-28T15:11:05.451] debug2: _slurm_rpc_dump_partitions, size=1253 usec=16
[2025-04-28T15:11:05.461] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2788
[2025-04-28T15:11:05.461] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2788
[2025-04-28T15:11:05.461] debug2: _slurm_rpc_dump_partitions, size=1253 usec=12
[2025-04-28T15:11:05.517] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2916
[2025-04-28T15:11:05.518] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2916
[2025-04-28T15:11:05.518] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14
[2025-04-28T15:11:05.628] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3405
[2025-04-28T15:11:05.629] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3405
[2025-04-28T15:11:05.629] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14
[2025-04-28T15:11:05.740] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2189
[2025-04-28T15:11:05.740] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2189
[2025-04-28T15:11:05.740] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15
[2025-04-28T15:11:05.845] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2209
[2025-04-28T15:11:05.846] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2209
[2025-04-28T15:11:05.846] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=4106
[2025-04-28T15:11:05.846] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14
[2025-04-28T15:11:05.847] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=4106
[2025-04-28T15:11:05.847] debug2: _slurm_rpc_dump_partitions, size=1253 usec=11
[2025-04-28T15:11:05.938] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3400
[2025-04-28T15:11:05.938] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3400
[2025-04-28T15:11:05.938] debug2: _slurm_rpc_dump_partitions, size=1253 usec=18
[2025-04-28T15:11:06.903] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3449
[2025-04-28T15:11:06.904] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3449
[2025-04-28T15:11:06.904] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15
[2025-04-28T15:11:07.175] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3722
[2025-04-28T15:11:07.176] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3722
[2025-04-28T15:11:07.177] debug2: _slurm_rpc_dump_partitions, size=1253 usec=254
[2025-04-28T15:11:07.205] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=4040
[2025-04-28T15:11:07.206] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=4040
[2025-04-28T15:11:07.206] debug2: _slurm_rpc_dump_partitions, size=1253 usec=17
[2025-04-28T15:11:07.237] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2990
[2025-04-28T15:11:07.238] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2990
[2025-04-28T15:11:07.239] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15
[2025-04-28T15:11:07.284] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2920
[2025-04-28T15:11:07.285] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2920
[2025-04-28T15:11:07.285] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15
[2025-04-28T15:11:07.370] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3236
[2025-04-28T15:11:07.371] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3236
[2025-04-28T15:11:07.371] debug2: _slurm_rpc_dump_partitions, size=1253 usec=17
[2025-04-28T15:11:08.463] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2848
[2025-04-28T15:11:08.464] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2848
[2025-04-28T15:11:08.464] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14
[2025-04-28T15:11:08.691] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2627
[2025-04-28T15:11:08.692] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2627
[2025-04-28T15:11:08.692] debug2: _slurm_rpc_dump_partitions, size=1253 usec=18
[2025-04-28T15:11:08.873] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3729
[2025-04-28T15:11:08.874] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3729
[2025-04-28T15:11:08.875] debug2: _slurm_rpc_dump_partitions, size=1253 usec=196
[2025-04-28T15:11:08.881] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3461
[2025-04-28T15:11:08.882] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3461
[2025-04-28T15:11:08.882] debug2: _slurm_rpc_dump_partitions, size=1253 usec=10
```
And from slurmd:
```
[2025-04-27T19:45:01.353] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:02.475] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:02.491] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:02.496] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:02.496] [59253.batch] debug: Handling REQUEST_GETGR
[2025-04-27T19:45:02.497] [59253.batch] debug: Handling REQUEST_GETGR
[2025-04-27T19:45:02.497] [59253.batch] debug: Handling REQUEST_GETGR
[2025-04-27T19:45:02.501] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:02.504] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:02.507] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:02.513] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:02.518] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:02.518] [59253.batch] debug: Handling REQUEST_GETGR
[2025-04-27T19:45:02.606] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:02.607] [59253.batch] debug: Handling REQUEST_GETGR
[2025-04-27T19:45:04.988] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:04.992] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:04.995] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:04.999] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.011] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.016] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.033] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.045] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.048] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.057] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.073] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.077] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.110] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.143] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.143] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.144] [59253.batch] debug: Handling REQUEST_GETGR
[2025-04-27T19:45:05.152] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.152] [59253.batch] debug: Handling REQUEST_GETGR
[2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETGR
[2025-04-27T19:45:05.167] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.167] [59253.batch] debug: Handling REQUEST_GETGR
[2025-04-27T19:45:05.170] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.172] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.174] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.203] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.204] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.207] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.316] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.318] [59253.batch] debug: Handling REQUEST_GETPW
[2025-04-27T19:45:05.321] [59253.batch] debug: Handling REQUEST_GETPW
```
This level of debugging makes the logs pretty huge, but if seeing a whole log file is helpful, I can make something available.
Any ideas on next steps for figuring out what’s going on? It seems like something is asking for authentication a whole lot, but it’s not clear to me what or why. We do use munge for Slurm authentication, and
SSSD to work with LDAP for user authentication.
-Jeremy Guillette
—
Jeremy Guillette
Software Engineer, FAS Academic Technology | Academic Technology
Harvard University Information Technology
P: (617) 998-1826 | W: huit.harvard.edu
(he/him/his)