[slurm-users] Having a possible cgroup issue?

[Anderson, Wes R]

PERSONAL/NONWORK // EXTERNAL
I took a look through the archives, and I did not see an clear answer to the issue I was seeing, so I thought I would go ahead and ask.

I am having a cluster issue with SLURM and I hoped you might be able to help me out. I built a small test cluster to determine if it might meet some compute needs I have but seem to keep running into an issue where SLURM is restricting MATLAB to using a single CPU regardless of how many we request.

During testing I found the following:

When I login into a MATLAB interactive session and run "feature numcores"

I get the following:

[cid:image002.jpg at 01D48D35.180C4F30]

Which is correct, as I have 14 cores and they are all available.


However when I go into SLURM and request a MATLAB interactive session and run the same command on the same computer:

[cid:image004.png at 01D48D35.180C4F30]


So, what I understand is that my cgroups settings in SLURM are restricting MATLAB to a single core. Is that correct?  Also, how do I fix this?

Here is my cgroups.conf

###
#
# Slurm cgroup support configuration file
#
# See man slurm.conf and man cgroup.conf for further
# information on cgroup configuration parameters
#--

########################################################
# W A R N I N G:  This file is managed by Puppet       #
# - - - - - - -   changes are likely to be overwritten #
########################################################

#######################
CgroupAutomount=yes
#######################

# testing -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
#ConstrainCores=no
#ConstrainRAMSpace=no
#ConstrainSwapSpace=no
# testing -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


ConstrainCores=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes

#
ConstrainDevices=no

AllowedSwapSpace=0
MaxRAMPercent=100
MaxSwapPercent=100
##MinRAMSpace=30

# TASK/CGROUP PLUGIN

# Constrain the job cgroup RAM to this percentage of the allocated memory.
#AllowedRAMSpace=10
AllowedRamSpace=100

# TaskAffinity=<yes|no>
#  If configured to "yes" then set a default task affinity to bind each
#  step task to a subset of the allocated cores using
#  sched_setaffinity. The default value is "no". Note: This feature
#  requires the Portable Hardware Locality (hwloc) library to be
#  installed.
TaskAffinity=yes

# MemorySwappiness=<number>
# Configure the kernel's priority for swapping out anonymous pages (such as program data)
# verses file cache pages for the job cgroup. Valid values are between 0 and 100, inclusive. A
# value of 0 prevents the kernel from swapping out program data. A value of 100 gives equal
# priorioty to swapping out file cache or anonymous pages. If not set, then the kernel's default
# swappiness value will be used. Either ConstrainRAMSpace or ConstrainSwapSpace must
# be set to yes in order for this parameter to be applied.

MemorySwappiness=0

#####################################################################################
# If compute nodes mount Lustre or NFS file systems, it may be a good idea to         #
#  configure cgroup.conf with:                                                                                                        #
# ConstrainKmemSpace=no                                                                                                                            #
#                                                                                                                                                                 #
# From <https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#activating-cgroups>  #
#####################################################################################
ConstrainKmemSpace=no                                                                                                                                #

########################################################
# W A R N I N G:  This file is managed by Puppet       #
# - - - - - - -   changes are likely to be overwritten #
########################################################

Thanks,
Wes
(A slurm neophyte)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20181206/9eeba43b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.jpg
Type: image/jpeg
Size: 13392 bytes
Desc: image006.jpg
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20181206/9eeba43b/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.png
Type: image/png
Size: 15203 bytes
Desc: image007.png
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20181206/9eeba43b/attachment.png>