<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Don't worry, I'm well past the "is this a sensible thing". Let's just call it an experiment.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
I have oversubscribe=FORCE:4 set on the partition, and nothing set on the sbatch command itself. And with that setting, I can execute a job that requires all of the node's cores 4x and it will put all of those jobs on that node. When I execute a 5<span class="elementToProof"><sup>th</sup> job,
it goes pending for resources. But in the meantime, only one of the jobs is running at any given time, the rest are suspended. That's just not what I would have thought it would be for "more than one job can execute simultaneously on the same compute resources."
I don't consider them to be executing simultaneously if they're suspended.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<span class="elementToProof"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<span class="elementToProof">Rob</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<span class="elementToProof"><br>
</span></div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Loris Bennett <loris.bennett@fu-berlin.de><br>
<b>Sent:</b> Friday, January 20, 2023 1:48 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Using oversubscribe to hammer a node</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Hi Rob,<br>
<br>
"Groner, Rob" <rug262@psu.edu> writes:<br>
<br>
> I'm trying to setup a specific partition where users can fight with the OS for dominance, The oversubscribe property sounds like what I want, as it says<br>
> "More than one job can execute simultaneously on the same compute resource." That's exactly what I want. I've setup a node with 48 CPU and<br>
> oversubscribe set to force:4. I then execute a job that requests 48 cpus, and that starts running. I execute another job asking for 48 cores, and it gets<br>
> assigned to the node...but it is not running, it's suspended. I can execute 2 more jobs, and they'll all go on the node (so, 4x) but 3 will be suspended at<br>
> any time. I see the time slicing going on, but that isn't what I though it would be...I thought all 4 tasks per cpu would be running at the same time.
<br>
> Basically, I want the CPU/OS to work out the sharing of resources. Otherwise, if one of the tasks that is running is just sitting there doing nothing, it's<br>
> going to do that for its 30 seconds while other tasks are suspended, right? <br>
<br>
Is --oversubscribe set for the jobs?<br>
<br>
> What I want to see is 4x the nodes CPUs in tasks all running at the same time, not time slicing, just for jobs using this partition. Is that a thing?<br>
<br>
It might be thing. I'm not sure it is a very sensible thing. Time<br>
slicing and context switching is still going to take place, with each<br>
process getting a quarter of a core on average. It is not clear that<br>
you will actually increase throughput this way. I would probably first<br>
turn on hyperthreading to deal with jobs which have intermittent<br>
CPU-usage.<br>
<br>
Still, since Slurm offers the possibility of oversubscription, I assume<br>
there must be a use-case.<br>
<br>
Cheers,<br>
<br>
Loris<br>
<br>
-- <br>
Dr. Loris Bennett (Herr/Mr)<br>
ZEDAT, Freie Universität Berlin<br>
<br>
</div>
</span></font></div>
</body>
</html>