Guillaume, Jobs shouldn't fail if they are requesting the max amount of memory they intend to use. If that is not there, the job would not start (perhaps that is what you meant). If they 'need' 100% of the available memory, you will definitely have some issues, as the OS itself needs some of that memory. That is the idea behind the setting. It will also give you the buffer for the few bytes that can deviate when doing 'slurmd -C' to read the memory reported by the node. I used to just truncate down to the nearest '00' (eg: 1675 became 1600) before the node_reg_mem_percent was shown to me. Now I just set that at 95% which allows for any deviations that occur. Brian Andrus On 8/18/2025 4:27 AM, Xaver Stiensmeier via slurm-users wrote:

Hey,

while the *node_reg_mem_percent* parameter sounds interesting, it would only be feasible for us on a per job basis (I wasn't able to find it there at first glance). Many of our users need a certain amount of RAM and jobs would fail if they have less. Therefore, this doesn't solve our issue.

Best regards, Xaver

On 8/14/25 10:42, Guillaume COCHARD via slurm-users wrote:

...

Hello,

You might want to use the *node_reg_mem_percent *parameter ( https://slurm.schedmd.com/slurm.conf.html#OPT_node_reg_mem_percent ). For example, if set to 80, it will allow a node to work even if it has only 80% of the declared memory.

Guillaume

------------------------------------------------------------------------ *De: *"Xaver Stiensmeier via slurm-users" <slurm-users@lists.schedmd.com> *À: *slurm-users@lists.schedmd.com *Envoyé: *Jeudi 14 Août 2025 10:01:26 *Objet: *[slurm-users] Nodes Become Invalid Due to Less Total RAM Than Expected

Dear slurm-user list,

in the past we had a bigger buffer between RealMemory <https://slurm.schedmd.com/slurm.conf.html#OPT_RealMemory> and the instance memory. We then discovered that the right way is to activating the *memory option* (SelectTypeParameters=CR_Core_Memory) and setting MemSpecLimit <https://slurm.schedmd.com/slurm.conf.html#OPT_MemSpecLimit> to secure RAM for system processes.

However, now we run into the problem that due to *on demand scheduling*, we have to setup the slurm.conf in advance by using the RAM values from our flavors as reported by our cloud provider (OpenStack). These RAM values are higher than the RAM values the machines actually have later on:

ram_in_mib by openstack total_ram_in_mib by top/slurm 2048 1968 16384 15991 32768 32093 65536 64297 122880 120749 245760 241608 491520 483528

Given that we have to define the slurm.conf in advance, we kinda have to predict how much total ram the instances have once created. Of course I used linear regression to approximate the total ram and then lowered it a bit to have some cushion, but this feels unsafe given that future flavors could differ from that.

From the kernel documentation <https://www.kernel.org/doc/Documentation/filesystems/proc.txt> I know that MemTotal is

MemTotal: Total usable ram (i.e. physical ram minus a few reserved bits and the kernel binary code)

but given that the concrete reserved bits are quite complex <https://witekio.com/blog/cat-proc-meminfo-memtotal/>, I am wondering whether I am doing something wrong as this issue doesn't feel niche enough to be that complicated.

---

Anyway, setting the RAM value in the slurm.conf above total ram by predicting too much, leads to errors and nodes being marked as invalid:

[2025-08-11T08:19:04.736] debug:  Node NODE_NAME has low real_memory size (241607 / 245760) < 100.00% [2025-08-11T08:19:04.736] error: _slurm_rpc_node_registration node=NODE_NAME: Invalid argument

or

|[2025-07-03T12:57:18.486] error: Setting node NODE_NAME state to INVAL with reason:Low RealMemory (reported:64295 < 100.00% of configured:68719)|

|Any hint on how to solve this is much appreciated! |

Best regards, Xaver

-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com

If a user wants all the memory on a node, I believe they can spec --mem=0 in order to not worry about matching the calculated RealMemory the node ends up with. If you're then concerned a job might end up on under-sized nodes you could have Features giving the round figure, like Feature=mem240g or whatever. I'll say also though that we don't tend to run into this much on our own clusters because either nodes are configured to only run one job at a time and therefore no memory spec is needed, or on shared nodes the jobs tend to have tight memory requests because we monitor for unused memory and email users -- whose jobs often do not actually use the amount the job was originally stated to require. ________________________________________ Od: Xaver Stiensmeier via slurm-users <slurm-users@lists.schedmd.com> Poslano: četrtek, 6. avgust 2026 05:21 Za: slurm-users@lists.schedmd.com Zadeva: [slurm-users] Re: Nodes Become Invalid Due to Less Total RAM Than Expected Dear slurm-user list, I want to revive this topic as I am still not 100% satisfied with the solution. As OpenStack flavors are often promoting RAM in "nice" GiB numbers like 240, many users/workflows also require "nice" numbers on job submission. So if we actually respect the total ram available (e.g. 245760 MiB in OpenStack, total ram 241608) and guesstimate it , jobs requiring the "nice" number 241608 will only schedule on nodes larger than the one given. It seems like this makes "lying" to the system by applying the 90 (like for State=CLOUD) https://slurm.schedmd.com/slurm.conf.html#OPT_node_reg_mem_percent and using the "in OpenStack" value instead of the total ram guesstimate necessary to make scheduling smooth. But this feels like opening a door for possible RAM issues if a job really needs the whole RAM. Maybe this is just not very common and swap and so on further minimizes this issue in real life, but I had to bring it up once more. Best, Xaver --- Old Discussion --- Brian Andrus wrote: Guillaume, Jobs shouldn't fail if they are requesting the max amount of memory they intend to use. If that is not there, the job would not start (perhaps that is what you meant). If they 'need' 100% of the available memory, you will definitely have some issues, as the OS itself needs some of that memory. That is the idea behind the setting. It will also give you the buffer for the few bytes that can deviate when doing 'slurmd -C' to read the memory reported by the node. I used to just truncate down to the nearest '00' (eg: 1675 became 1600) before the node_reg_mem_percent was shown to me. Now I just set that at 95% which allows for any deviations that occur. Brian Andrus On 8/18/25 13:27, Xaver Stiensmeier wrote: Hey, while the node_reg_mem_percent parameter sounds interesting, it would only be feasible for us on a per job basis (I wasn't able to find it there at first glance). Many of our users need a certain amount of RAM and jobs would fail if they have less. Therefore, this doesn't solve our issue. Best regards, Xaver On 8/14/25 10:42, Guillaume COCHARD via slurm-users wrote: Hello, You might want to use the node_reg_mem_percent parameter ( https://slurm.schedmd.com/slurm.conf.html#OPT_node_reg_mem_percent ). For example, if set to 80, it will allow a node to work even if it has only 80% of the declared memory. Guillaume ________________________________ De: "Xaver Stiensmeier via slurm-users" <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> À: slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> Envoyé: Jeudi 14 Août 2025 10:01:26 Objet: [slurm-users] Nodes Become Invalid Due to Less Total RAM Than Expected Dear slurm-user list, in the past we had a bigger buffer between RealMemory<https://slurm.schedmd.com/slurm.conf.html#OPT_RealMemory> and the instance memory. We then discovered that the right way is to activating the memory option (SelectTypeParameters=CR_Core_Memory) and setting MemSpecLimit<https://slurm.schedmd.com/slurm.conf.html#OPT_MemSpecLimit> to secure RAM for system processes. However, now we run into the problem that due to on demand scheduling, we have to setup the slurm.conf in advance by using the RAM values from our flavors as reported by our cloud provider (OpenStack). These RAM values are higher than the RAM values the machines actually have later on: ram_in_mib by openstack total_ram_in_mib by top/slurm 2048 1968 16384 15991 32768 32093 65536 64297 122880 120749 245760 241608 491520 483528 Given that we have to define the slurm.conf in advance, we kinda have to predict how much total ram the instances have once created. Of course I used linear regression to approximate the total ram and then lowered it a bit to have some cushion, but this feels unsafe given that future flavors could differ from that. From the kernel documentation<https://www.kernel.org/doc/Documentation/filesystems/proc.txt> I know that MemTotal is MemTotal: Total usable ram (i.e. physical ram minus a few reserved bits and the kernel binary code) but given that the concrete reserved bits are quite complex<https://witekio.com/blog/cat-proc-meminfo-memtotal/>, I am wondering whether I am doing something wrong as this issue doesn't feel niche enough to be that complicated. --- Anyway, setting the RAM value in the slurm.conf above total ram by predicting too much, leads to errors and nodes being marked as invalid: [2025-08-11T08:19:04.736] debug: Node NODE_NAME has low real_memory size (241607 / 245760) < 100.00% [2025-08-11T08:19:04.736] error: _slurm_rpc_node_registration node=NODE_NAME: Invalid argument or [2025-07-03T12:57:18.486] error: Setting node NODE_NAME state to INVAL with reason:Low RealMemory (reported:64295 < 100.00% of configured:68719) Any hint on how to solve this is much appreciated! Best regards, Xaver -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com