<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" id="owaParaStyle">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;"><font size="2" face="Tahoma" color="black"><span style="font-size:10pt;" dir="ltr"></span></font><font size="2" face="Tahoma" color="black"><span style="font-size:10pt;" dir="ltr">
<div>Hi,</div>
</span></font><font size="2" face="Tahoma" color="black"><span style="font-size:10pt;" dir="ltr"></span></font>
<div><font size="2" face="Tahoma" color="black"><span style="font-size:10pt;" dir="ltr">
<div><br>
</div>
<div>I have a federation of 2 clusters 'merlin5' and 'merlin6'. However for some reason I have two jobs in a strange state, one in FedJobLock and the second in Priority (which never gets allocated and I am not able to cancel):</div>
<div><br>
</div>
<div> 134225868 gpu bash bliven_s PD 0:00 1 (FedJobLock)<br>
134225867 gpu bash bliven_s PD 0:00 1 (Priority)<br>
</div>
<div><br>
</div>
<div>I try to cancel the jobs, no way. From the Slurm server logs I see the following:</div>
<div><br>
</div>
<div>Merlin5:<br>
</div>
<div><br>
</div>
<div>[2019-07-02T14:20:14.252] backfill test for JobId=134225868 Prio=3559 Partition=gpu<br>
[2019-07-02T14:20:14.293] backfill: JobId=134225868 can't get fed job lock from origin cluster to backfill job<br>
[2019-07-02T14:20:14.293] backfill: planned start of JobId=134225868 failed: Job locked by another sibling<br>
[2019-07-02T14:20:14.293] JobId=134225868 to start at 2019-07-02T14:20:14, end at 2019-07-07T14:20:00 on nodes merlin-g-01 in partition gpu<br>
[2019-07-02T14:20:14.294] backfill test for JobId=134225867 Prio=3559 Partition=gpu<br>
[2019-07-02T14:20:14.374] backfill: JobId=134225867 can't get fed job lock from origin cluster to backfill job<br>
[2019-07-02T14:20:14.374] backfill: planned start of JobId=134225867 failed: Job locked by another sibling<br>
[2019-07-02T14:20:14.374] JobId=134225867 to start at 2019-07-02T14:20:14, end at 2019-07-07T14:20:00 on nodes merlin-g-04 in partition gpu<br>
[2019-07-02T14:20:14.374] backfill: reached end of job queue<br>
[2019-07-02T14:20:14.374] backfill: completed testing 2(2) jobs, usec=122038<br>
[2019-07-02T14:20:18.052] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=134225868 uid 0 routed to merlin6<br>
[2019-07-02T14:20:18.052] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=134225867 uid 0 routed to merlin6<br>
</div>
<div><br>
</div>
<div>Merlin6:<br>
</div>
<div><br>
</div>
<div>[2019-07-02T14:20:21.755] backfill: beginning<br>
[2019-07-02T14:20:21.756] backfill: no jobs to backfill<br>
[2019-07-02T14:20:44.415] error: Didn't find JobId=134225868 in fed_job_list<br>
[2019-07-02T14:20:44.456] error: Didn't find JobId=134225867 in fed_job_list<br>
[2019-07-02T14:20:51.756] backfill: beginning<br>
[2019-07-02T14:20:51.756] backfill: no jobs to backfill<br>
[2019-07-02T14:21:09.721] error: Didn't find JobId=134225868 in fed_job_list<br>
[2019-07-02T14:21:14.537] error: Didn't find JobId=134225868 in fed_job_list<br>
[2019-07-02T14:21:14.578] error: Didn't find JobId=134225867 in fed_job_list<br>
<br>
</div>
<div>While from the accounting server:</div>
<div><br>
</div>
<div> bliven_s 134225867 bash gpu PENDING Partition+ Unknown Unknown 00:00:00 1 1 00:00:00 None assigned merlin5
<br>
bliven_s 134225868 bash gpu PENDING Partition+ Unknown Unknown 00:00:00 1 1 00:00:00 None assigned merlin5
<br>
<br>
</div>
<div><br>
</div>
<div>Any idea how to fix that and what could trigger this?</div>
<div><br>
</div>
<div>Thanks a lot,</div>
<div class="PlainText">Marc <br>
_________________________________________________________<br>
Paul Scherrer Institut <br>
High Performance Computing & Emerging Technologies<br>
Marc Caubet Serrabou<br>
Building/Room: OHSA/014</div>
</span></font>
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText"><font size="2"><span style="font-size:10pt">Forschungsstrasse, 111</span></font></div>
<div class="PlainText">5232 Villigen PSI<br>
Switzerland<br>
<br>
Telephone: +41 56 310 46 67<br>
E-Mail: marc.caubet@psi.ch</div>
</span></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>