[slurm-users] Slurm 22.05.8 - salloc not starting shell on remote host

Fri May 19 14:32:47 UTC 2023

I'm setting up Slurm from scratch for the first time ever. Using 22.05.8 
since I haven't had a changed to upgrade our DB server to 23.02 yet. 
When I try to use salloc to get a shell on a compute node 
(ranger-s22-07), I end up with a shell on the login node (ranger):

[pbisbal at ranger ~]$ salloc -n 1 -t 00:10:00  --mem=1G salloc: Granted 
job allocation 23 salloc: Waiting for resource configuration salloc: 
Nodes ranger-s22-07 are ready for job [pbisbal at ranger ~]$

Any ideas what's going wrong here? I have the following line in my 
slurm.conf:

LaunchParameters=user_interactive_step

When I run salloc with -vvvvv, here's what I see:

[pbisbal at ranger ~]$ salloc -vvvvv -n 1 -t 00:10:00  --mem=1G
salloc: defined options
salloc: -------------------- --------------------
salloc: mem                 : 1G
salloc: ntasks              : 1
salloc: time                : 00:10:00
salloc: verbose             : 5
salloc: -------------------- --------------------
salloc: end of defined options
salloc: debug3: Trying to load plugin /usr/lib64/slurm/select_cons_res.so
salloc: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Consumable Resources (CR) Node Selection plugin type:select/cons_res version:0x160508
salloc: select/cons_res: common_init: select/cons_res loaded
salloc: debug3: Success.
salloc: debug3: Trying to load plugin /usr/lib64/slurm/select_cons_tres.so
salloc: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Trackable RESources (TRES) Selection plugin type:select/cons_tres version:0x160508
salloc: select/cons_tres: common_init: select/cons_tres loaded
salloc: debug3: Success.
salloc: debug3: Trying to load plugin /usr/lib64/slurm/select_cray_aries.so
salloc: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Cray/Aries node selection plugin type:select/cray_aries version:0x160508
salloc: select/cray_aries: init: Cray/Aries node selection plugin loaded
salloc: debug3: Success.
salloc: debug3: Trying to load plugin /usr/lib64/slurm/select_linear.so
salloc: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Linear node selection plugin type:select/linear version:0x160508
salloc: select/linear: init: Linear node selection plugin loaded with argument 20
salloc: debug3: Success.
salloc: debug:  Entering slurm_allocation_msg_thr_create()
salloc: debug:  port from net_stream_listen is 43881
salloc: debug:  Entering _msg_thr_internal
salloc: debug4: eio: handling events for 1 objects
salloc: debug3: eio_message_socket_readable: shutdown 0 fd 6
salloc: debug3: Trying to load plugin /usr/lib64/slurm/auth_munge.so
salloc: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge authentication plugin type:auth/munge version:0x160508
salloc: debug:  auth/munge: init: Munge authentication plugin loaded
salloc: debug3: Success.
salloc: debug3: Trying to load plugin /usr/lib64/slurm/hash_k12.so
salloc: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:KangarooTwelve hash plugin type:hash/k12 version:0x160508
salloc: debug:  hash/k12: init: init: KangarooTwelve hash plugin loaded
salloc: debug3: Success.
salloc: Granted job allocation 24
salloc: Waiting for resource configuration
salloc: Nodes ranger-s22-07 are ready for job
salloc: debug:  laying out the 1 tasks on 1 hosts ranger-s22-07 dist 8192
[pbisbal at ranger ~]$

This is all I see in /var/log/slurm/slurmd.log on the compute node:

[2023-05-19T10:21:36.898] [24.extern] task/cgroup: _memcg_initialize: job: alloc=1024MB mem.limit=1024MB memsw.limit=unlimited
[2023-05-19T10:21:36.899] [24.extern] task/cgroup: _memcg_initialize: step: alloc=1024MB mem.limit=1024MB memsw.limit=unlimited

And this is all I see in /var/log/slurm/slurmctld.log on the controller:

[2023-05-19T10:18:16.815] sched: _slurm_rpc_allocate_resources JobId=23 NodeList=ranger-s22-07 usec=1136
[2023-05-19T10:18:22.423] Time limit exhausted for JobId=22
[2023-05-19T10:21:36.861] sched: _slurm_rpc_allocate_resources JobId=24 NodeList=ranger-s22-07 usec=1039

Here's my slurm.conf file:

# grep -v ^# /etc/slurm/slurm.conf  | grep -v ^$

ClusterName=ranger
SlurmctldHost=ranger-master
EnforcePartLimits=ALL
JobSubmitPlugins=lua,require_timelimit
LaunchParameters=user_interactive_step
MaxStepCount=2500
MaxTasksPerNode=32
MpiDefault=none
ProctrackType=proctrack/cgroup
PrologFlags=contain
ReturnToService=0
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
StateSaveLocation=/var/spool/slurmctld
SwitchType=switch/none
TaskPlugin=task/affinity,task/cgroup
TopologyPlugin=topology/tree
CompleteWait=32
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0|
DefMemPerCPU=5000
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory
PriorityType=priority/multifactor
PriorityDecayHalfLife=15-0
PriorityCalcPeriod=15
PriorityFavorSmall=NO
PriorityMaxAge=180-0
PriorityWeightAge=5000
PriorityWeightFairshare=5000
PriorityWeightJobSize=5000
AccountingStorageEnforce=all
AccountingStorageHost=slurm.pppl.gov
AccountingStorageType=accounting_storage/slurmdbd
AccountingStoreFlags=job_script
JobCompType=jobcomp/none
JobAcctGatherFrequency=30
JobAcctGatherParams=UsePss
JobAcctGatherType=jobacct_gather/cgroup
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurm/slurmd.log
NodeName=ranger-s22-07 CPUs=72 Boards=1 SocketsPerBoard=4 CoresPerSocket=18 ThreadsPerCore=1 RealMemory=384880 State=UNKNOWN
PartitionName=all Nodes=ALL Default=YES GraceTime=300 MaxTime=24:00:00 State=UP

-- 
Prentice
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20230519/4db74f24/attachment.htm>