<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Hi,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
scancel the job, then set the nodes to a "down" state like so "<span style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt;">scontrol update nodename=<nodename> state=down reason=cg" <span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;">and
resume them afterwards.</span></span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt;">However, if there are tasks stuck, then in most cases a reboot is needed to bring the node back with in a clean state.</span><br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt;"><br>
</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt;">Best,</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt;">Florian</span></div>
<div>
<div id="Signature">
<div>
<div style="">
<div style=""></div>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Durai Arasan <arasan.durai@gmail.com><br>
<b>Sent:</b> Friday, 20 August 2021 10:31<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [External][slurm-users] jobs stuck in "CG" state</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hello!<br>
<div><br>
</div>
<div>We have a huge number of jobs stuck in CG state from a user who probably wrote code with bad I/O. "scancel" does not make them go away. Is there a way for admins to get rid of these jobs without draining and rebooting the nodes. I read somewhere that killing
the respective slurmstepd process will do the job. Is this possible? Any other solutions? Also are there any parameters in slurm.conf one can set to manage such situations better?</div>
<div><br>
</div>
<div>Best,</div>
<div>Durai</div>
<div>MPI Tübingen</div>
</div>
</div>
</body>
</html>