knl_generic.conf seems to be ignored and Slurm adopts default settings which specify Intel motherboard and syscfg - need help to figure out why Slurm is not reading my Dell knl_generic.conf parameters
Hi folks I'm in the process of standing up some Stampede2 Dell C3620p KNL nodes and I seem to be hitting a blind spot. I previously "successfully" configured KNL's on an Intel board (S7200AP), with OpenHPC and Rocky8. I say "successfully" because it works but evidently my latest troubleshooting has revealed that I may have been lucky rather than an expert KNL integrator :-) I thought I knew what I was doing, but after repeating my Intel KNL recipe with the Dell system, I have unearthed my ignorance with this wonderful (but deprecated) technology (anecdotally, the KNLs offer excellent performance and power efficiency for our workloads, particularly when contrasted with our alternative available hardware). The first discovery was the "syscfg" for Intel boards is not the same as the "syscfg" for Dell boards. I've since sorted this out. The second discovery was made while troubleshooting an issue that I'm hitting. After realising that the slurmd client nodes don't seem to be reading the "knl_generic.conf" parameters that are specified in /etc/slurm on the smshost (OpenHPC parlance for head node ... And it's a Slurm config less set up), I think my original Intel solution was working out of luck more than ingenuity. For reference , the Slurm configuration for KNL now includes: ``` NodeFeaturesPlugins=knl_generic DebugFlags=NodeFeatures GresTypes=hbm ``` And I've created a separate "knl_generic.conf" that points to the Dell specific tools and features. For the Dell system, slurmd seems to ignore my knl_generic.conf file and is drawing defaults from somewhere else. Slurm still considers SystemType to be Intel, SyscfgPath to be the default location, and SyscfgTimeout to be 1000. For Dell systems, Slurm needs to have SystemType=Dell and Timeout to be 1000. I don't understand why the nodes are not reading the knl_generic file - any help or clues would be appreciated. Here's my theory on what is happening: The Intel KNL system was successful by luck ... It probably exhibited the same ignore-the-config-file but ran default NodeFeatures for some generic knl_generic settings which are stored somewhere as default parameters. I must have just lucked out when I was using my Intel KNL system because it was using the defaults (that are compatible with Intel). If this assumption is correct, the Dell system is not working because it isn't compatible with the Intel defaults. Any clues on how to successfully invoke the config file (or better debuggingtechniwues to figure out why it isn't) would be appreciated. I can share journalctl feedback if necessary. For now, I've tried changing ownership of the config files to root:slurm, copied knl_generic.conf to the compute nodes' /etc/slurm/ and also tried to specify the config file by running (on the compute nodes) "slurmd" with "-f" ... No joy; if slurmd runs successfully (when I don't screw up some random experimental settings) then it always seems to ignore knl_generic.conf and loads some default settings from somewhere. A few questions: 1. Are there default settings stored somewhere? I might be barking up the wrong tree, although I've looked for files that may clash with the config file I've created but can't find any. 2. Is there a better way to force the knl_generic file to be loaded? 3. Is the configless Slurm somehow not reading the knl_generic file to the clients? I understand that all configuration files are read from the host server. Many thanks for any help! Regards / Groete / Sala(ni) Kahle - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Bryan Johnston Senior HPC Technologist II Lead: HPC Ecosystems Project HPCwire 2024's Outstanding Leader in HPC CHPC | www.chpc.ac.za | NICIS | nicis.ac.za Centre for High Performance Computing If you receive an email from me out of office hours for you, please do not feel obliged to respond during off-hours! Book time to meet with me<https://outlook.office.com/bookwithme/user/87af4906a703488386578f34e4473c74@csir.co.za?anonymous&ep=signature>
Hi Bryan Just to close this thread, here's the TLDR of the work-around / solution: * copy knl_generic.conf to the compute nodes and store in the configless-slurm config staging area configless-slurm behaviour * As per Slurm Workload Manager -<https://slurm.schedmd.com/configless_slurm.html> (configless-slurm page), only the following configuration files are supported with configless-slurm: * slurm.conf * acct_gather.conf * cgroup.conf * cli_filter.lua * gres.conf * helpers.conf * job_container.conf * mpi.conf * oci.conf * plugstack.conf * scrun.lua * topology.conf * knl_generic.conf is not one of those files, so to keep using configless-slurm and propagate the knl_generic.conf file, it has to be done manually solution / work-around with configless-slurm * manually copy knl_generic.conf to /var/spool/slurmd/conf-cache/ on compute node/s and then restart slurmd. * there is a symbolic link to /run/slurm/conf to point to the above directory Thanks Bryan! Regards / Groete / Sala(ni) Kahle - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Bryan Johnston Senior HPC Technologist II Lead: HPC Ecosystems Project HPCwire 2024's Outstanding Leader in HPC ________________________________ From: Bryan Johnston via slurm-users <slurm-users@lists.schedmd.com> Sent: Saturday, 19 April 2025 18:20 To: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com> Subject: [slurm-users] knl_generic.conf seems to be ignored and Slurm adopts default settings which specify Intel motherboard and syscfg - need help to figure out why Slurm is not reading my Dell knl_generic.conf parameters Hi folks I'm in the process of standing up some Stampede2 Dell C3620p KNL nodes and I seem to be hitting a blind spot. I previously "successfully" configured KNL's on an Intel board (S7200AP), with OpenHPC and Rocky8. I say "successfully" because it works but evidently my latest troubleshooting has revealed that I may have been lucky rather than an expert KNL integrator :-) I thought I knew what I was doing, but after repeating my Intel KNL recipe with the Dell system, I have unearthed my ignorance with this wonderful (but deprecated) technology (anecdotally, the KNLs offer excellent performance and power efficiency for our workloads, particularly when contrasted with our alternative available hardware). The first discovery was the "syscfg" for Intel boards is not the same as the "syscfg" for Dell boards. I've since sorted this out. The second discovery was made while troubleshooting an issue that I'm hitting. After realising that the slurmd client nodes don't seem to be reading the "knl_generic.conf" parameters that are specified in /etc/slurm on the smshost (OpenHPC parlance for head node ... And it's a Slurm config less set up), I think my original Intel solution was working out of luck more than ingenuity. For reference , the Slurm configuration for KNL now includes: ``` NodeFeaturesPlugins=knl_generic DebugFlags=NodeFeatures GresTypes=hbm ``` And I've created a separate "knl_generic.conf" that points to the Dell specific tools and features. For the Dell system, slurmd seems to ignore my knl_generic.conf file and is drawing defaults from somewhere else. Slurm still considers SystemType to be Intel, SyscfgPath to be the default location, and SyscfgTimeout to be 1000. For Dell systems, Slurm needs to have SystemType=Dell and Timeout to be 1000. I don't understand why the nodes are not reading the knl_generic file - any help or clues would be appreciated. Here's my theory on what is happening: The Intel KNL system was successful by luck ... It probably exhibited the same ignore-the-config-file but ran default NodeFeatures for some generic knl_generic settings which are stored somewhere as default parameters. I must have just lucked out when I was using my Intel KNL system because it was using the defaults (that are compatible with Intel). If this assumption is correct, the Dell system is not working because it isn't compatible with the Intel defaults. Any clues on how to successfully invoke the config file (or better debuggingtechniwues to figure out why it isn't) would be appreciated. I can share journalctl feedback if necessary. For now, I've tried changing ownership of the config files to root:slurm, copied knl_generic.conf to the compute nodes' /etc/slurm/ and also tried to specify the config file by running (on the compute nodes) "slurmd" with "-f" ... No joy; if slurmd runs successfully (when I don't screw up some random experimental settings) then it always seems to ignore knl_generic.conf and loads some default settings from somewhere. A few questions: 1. Are there default settings stored somewhere? I might be barking up the wrong tree, although I've looked for files that may clash with the config file I've created but can't find any. 2. Is there a better way to force the knl_generic file to be loaded? 3. Is the configless Slurm somehow not reading the knl_generic file to the clients? I understand that all configuration files are read from the host server. Many thanks for any help! Regards / Groete / Sala(ni) Kahle - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Bryan Johnston Senior HPC Technologist II Lead: HPC Ecosystems Project HPCwire 2024's Outstanding Leader in HPC CHPC | www.chpc.ac.za<https://eu-west-1.protection.sophos.com?d=chpc.ac.za&u=d3d3LmNocGMuYWMuemE=&p=m&i=NjU0NjA2MWNiMzM3ZDMzYTFlMGNkZjg4&t=eXVqRnRpb0Q0VGFFN1l3dWhOUUZjaDYwUnVJY2xQUVcxMExQMmFkSjB6az0=&h=0f498e10f92f47dcb51521b434acf373&s=AVNPUEhUT0NFTkNSWVBUSVafegCoFVwHpbi5-eo-vXlDr6jwXShynM3GPug0j8p_iJ17sOQKeUhBTnTbLUWIJyY> | NICIS | nicis.ac.za Centre for High Performance Computing If you receive an email from me out of office hours for you, please do not feel obliged to respond during off-hours! Book time to meet with me<https://eu-west-1.protection.sophos.com?d=office.com&u=aHR0cHM6Ly9vdXRsb29rLm9mZmljZS5jb20vYm9va3dpdGhtZS91c2VyLzg3YWY0OTA2YTcwMzQ4ODM4NjU3OGYzNGU0NDczYzc0QGNzaXIuY28uemE_YW5vbnltb3VzJmVwPXNpZ25hdHVyZQ==&p=m&i=NjU0NjA2MWNiMzM3ZDMzYTFlMGNkZjg4&t=M3FYc0ZTMDUvS0FxS1VjZ1c5ZU15QzZkTlRVMXB3VWhseitMMVArb2tMND0=&h=0f498e10f92f47dcb51521b434acf373&s=AVNPUEhUT0NFTkNSWVBUSVafegCoFVwHpbi5-eo-vXlDr6jwXShynM3GPug0j8p_iJ17sOQKeUhBTnTbLUWIJyY>
participants (1)
-
Bryan Johnston