[slurm-users] Help with Con_Res Plugin Error

[Julius, Chad]

Slurm Users,

I am hoping that you all can help me with the problem below.

We just spun up a new cluster using Bright and have been trying to change the default behavior of slurm from linear to con_res.  Should be simple enough but I am plagued by the following error:

error: we don't have select plugin type 102

Both the select_linear.so and select_cons_res.so are located in /cm/shared_tmp/apps/slurm/17.11.8/lib64/slurm/

I have been testing with just the compute nodes and not the GPU nodes etc...  I added the following to my slurm.conf file:

# Scheduler
SchedulerType=sched/backfill
SelectType=select/cons_res
SelectTypeParameters=CR_Core

# Nodes
# NodeName=big-mem[001-005],node[001-056]   # Entry from default install
# NodeName=gpu[001-004]  Gres=gpu:2   # Entry from default install
NodeName=node[001-056] CPUs=2 RealMemory=196000 Sockets=2 CoresPerSocket=20 ThreadsPerCore=1 State=UNKNOWN


# Partitions
PartitionName=defq Default=YES MinNodes=1 AllowGroups=ALL PriorityJobFactor=1 PriorityTier=1 DisableRootJobs=NO RootOnly=NO Hidden=NO Shared=YES GraceTime=0 PreemptMode=OFF ReqResv=NO AllowAccounts=ALL AllowQos=ALL LLN=NO ExclusiveUser=NO OverSubscribe=NO OverTimeLimit=0 State=UP Nodes=gpu[001-004],big-mem[001-005],node[001-056]
PartitionName=test Default=NO MinNodes=1 AllowGroups=ALL PriorityJobFactor=1 PriorityTier=1 DisableRootJobs=NO RootOnly=NO Hidden=NO Shared=YES GraceTime=0 PreemptMode=OFF ReqResv=NO AllowAccounts=ALL AllowQos=ALL LLN=NO ExclusiveUser=NO OverSubscribe=NO OverTimeLimit=0 State=UP Nodes=node[001-056]

When I issue the scontrol reconfigure I get the following:

[root at thunder ~]# scontrol reconfigure
slurm_reconfigure error: Unable to contact slurm controller (connect failure)
[root at thunder ~]# systemctl status slurmctld.service
● slurmctld.service - Slurm controller daemon
   Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2018-12-13 08:46:18 CST; 5s ago
  Process: 31416 ExecStart=/cm/shared/apps/slurm/17.11.8/sbin/slurmctld $SLURMCTLD_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 31418 (code=exited, status=1/FAILURE)

When I revert the changes, it goes back to an active working state.

The /var/log/slurmctld log shows this erorr message:

error: we don't have select plugin type 102

Has anyone else run into this problem?  If so, can you recommend a fix?

Thanks,

Chad

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20181213/79cac0c4/attachment.html>