We’re planning to add a few new compute nodes to a small Slurm cluster, and I’m trying to work out a sensible CPU-to-memory balance before choosing the hardware. My current thought is to look at historical job data first, especially allocated CPUs, requested memory, MaxRSS, runtime, and how often jobs are waiting because of CPU versus memory availability. That seems more useful than simply buying the highest core-count CPUs available. For the new nodes, I’m mainly trying to decide: Whether fewer higher-clocked cores or more lower-clocked cores would give us better overall throughput How much RAM per physical core makes sense based on actual job usage Whether it is better to standardize all nodes or have a separate high-memory partition How much physical RAM should be left outside Slurm's RealMemory value for the OS and system processes Whether NUMA layout and memory bandwidth should influence the CPU choice as much as core count I’ve also been comparing different rack-server configurations here to get an idea of available Xeon, ECC memory, and storage combinations: https://ormsystems.com/servers-price.html For those managing production Slurm clusters, do you normally size new nodes from something like the 80th/90th percentile of historical memory-per-core usage, or do you use a different method? I’d be interested to hear what metrics you’ve found most useful before purchasing additional compute nodes.
It really depends on what your users require. At least in our environment (which runs pretty much every different workload under the sun) we have found that moving to 8 GB/core is the new standard. RAM prices though are making that less appealing. More specifically we have found that having 1TB of total memory per node has been invaluable because it allows us to accommodate the vast majority of large memory jobs while still being able to schedule the other cores for work. This means we have far less demand for high memory nodes. As for what to reserve for the OS, it really depends on what OS you run. From practical experience we reserve 16 GB for Rocky 8. -Paul Edmon- On 9/23/26 10:58 AM, wesleycolton238--- via slurm-users wrote:
We’re planning to add a few new compute nodes to a small Slurm cluster, and I’m trying to work out a sensible CPU-to-memory balance before choosing the hardware.
My current thought is to look at historical job data first, especially allocated CPUs, requested memory, MaxRSS, runtime, and how often jobs are waiting because of CPU versus memory availability. That seems more useful than simply buying the highest core-count CPUs available.
For the new nodes, I’m mainly trying to decide:
Whether fewer higher-clocked cores or more lower-clocked cores would give us better overall throughput How much RAM per physical core makes sense based on actual job usage Whether it is better to standardize all nodes or have a separate high-memory partition How much physical RAM should be left outside Slurm's RealMemory value for the OS and system processes Whether NUMA layout and memory bandwidth should influence the CPU choice as much as core count
I’ve also been comparing different rack-server configurations here to get an idea of available Xeon, ECC memory, and storage combinations:
https://ormsystems.com/servers-price.html
For those managing production Slurm clusters, do you normally size new nodes from something like the 80th/90th percentile of historical memory-per-core usage, or do you use a different method?
I’d be interested to hear what metrics you’ve found most useful before purchasing additional compute nodes.
On 23 September 2026 15:58:08 GMT+01:00, wesleycolton238--- via slurm-users <slurm-users@lists.schedmd.com> wrote:
We’re planning to add a few new compute nodes to a small Slurm cluster, and I’m trying to work out a sensible CPU-to-memory balance before choosing the hardware.
My current thought is to look at historical job data first, especially allocated CPUs, requested memory, MaxRSS, runtime, and how often jobs are waiting because of CPU versus memory availability. That seems more useful than simply buying the highest core-count CPUs available.
For the new nodes, I’m mainly trying to decide:
Whether fewer higher-clocked cores or more lower-clocked cores would give us better overall throughput How much RAM per physical core makes sense based on actual job usage Whether it is better to standardize all nodes or have a separate high-memory partition How much physical RAM should be left outside Slurm's RealMemory value for the OS and system processes Whether NUMA layout and memory bandwidth should influence the CPU choice as much as core count
I’ve also been comparing different rack-server configurations here to get an idea of available Xeon, ECC memory, and storage combinations:
https://ormsystems.com/servers-price.html
For those managing production Slurm clusters, do you normally size new nodes from something like the 80th/90th percentile of historical memory-per-core usage, or do you use a different method?
I’d be interested to hear what metrics you’ve found most useful before purchasing additional compute nodes.
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
I would not advise maxing the core country as that will mean slower clock speed. We have 2 small (very small) clusters and fixed on 32GB/core for no calculated reason as we had no previous experience. We have a lot of genomics users whose software mostly does not do MPI but also needs quite a lot of RAM. Currently the most we have is 768GB which is not enough for several of our users. I would suggest having at least 1 node with as much RAM as you can afford; 2-3TB not unreasonable for assembly, and have a partition for it. That allows you to manage access better. I like that many of our nodes are the same CPU speed and GB/core. I expect the scheduler doesn't care but unless you made it a resource, it has no knowledge of core clock speed. Whatever you can afford!
participants (3)
-
Paul Edmon -
wesleycolton238＠gmail.com -
William Brown