[slurm-users] AWS SLURM Burst Cluster, fill configuring nodes

[John Hearns]

Hi Jordan.
Regarding filling up the nodes look at
https://slurm.schedmd.com/elastic_computing.html

*SelectType* Generally must be "select/linear". If Slurm is configured to
allocate individual CPUs to jobs rather than whole nodes (e.g.
SelectType=select/cons_res rather than SelectType=select/linear), then
Slurm maintains bitmaps to track the state of every CPU in the system. If
the number of CPUs to be allocated on each node is not known when the
slurmctld daemon is started, one must allocate whole nodes to jobs rather
than individual processors. The use of "select/cons_res" requires each node
to have a CPU count set and the node eventually selected must have at least
that number of CPUs.

If I am not wrong you can configure the number of CPUs per node as a fixed
amount - if you select a fixed instance type


NOTE: This demo uses c4.2xlarge instance types for the compute nodes, which
have statically set the number of CPUs=8 in slurm_nodes.conf. If you want
to expierment with different instance types (in slurm-aws-startup.sh)
ensure you change the CPUs in slurm_nodes.conf.






On Fri, 26 Oct 2018 at 07:13, J.R. W <jwillis0720 at gmail.com> wrote:

>
>
>
> Hello everyone,
>
> I setup a SLURM cluster based on this post and plugin.
> https://aws.amazon.com/blogs/compute/deploying-a-burstable-and-event-driven-hpc-cluster-on-aws-using-slurm-part-1/
>
> When I submit jobs to the queue, the AWS instances start configuring.
> Because I have so many potential instances, for each job, they spool up one
> instance. For example, if I submit 10 job, AWS will configure 10 instances.
> What would be ideal is if there is a slurm.conf option I’m missing that
> will tell the power-save plugin to only configure N amount of nodes, even
> though there hundreds of “available” nodes to configure in the cloud. Some
> potential solutions I have thought of.
>
> 1. Have the scheduler fill up nodes even if they are in the configuring
> state. SLURM knows how many CPUs are available for the nodes that are being
> configured. Is there a way to have jobs all fill up a node, even if it’s in
> the configuring state? That way, a queued job will not trigger the “power
> save resume” of a new node.
>
> 2. Some parameter in slurm.conf that has maximum nodes that can be
> available.
>
> 3. Modify my slurm_resum script to check for how many nodes are
> configured. If that number is greater than my N amount of nodes I want spun
> up, then do nothing. Hopefully that will just send the job back to the
> queue to await one of those configured nodes.
>
> I hope I’m making sense. I know the elastic computing is a new feature
>
> Jordan
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20181026/c6829788/attachment.html>