<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi,</p>
<p>we're not using Omni-Path but also had issues with Infiniband
taking too long and slurmd failing to start due to that.</p>
<p>Our solution was to implement a little wait-for-interface systemd
service which delays the network.target until the ib interface has
come up.</p>
<p>Our discovery was that the network-online.target is triggered by
the NetworkManager as soon as the first interface is connected.</p>
<p>I've put the solution we use on my GitHub: <a
href="https://github.com/maxlxl/network.target_wait-for-interfaces"
class="moz-txt-link-freetext">https://github.com/maxlxl/network.target_wait-for-interfaces</a></p>
<p>You may need to do small adjustments, but it's pretty straight
forward in general.</p>
<p><br>
</p>
<p>Kind regards<br>
Max<br>
</p>
<div class="moz-cite-prefix">On 30.10.23 13:50, Ole Holm Nielsen
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:140a10f1-0668-4024-b041-32f9e1385c85@fysik.dtu.dk">I'm
fighting this strange scenario where slurmd is started before the
Infiniband/OPA network is fully up. The Node Health Check (NHC)
executed by slurmd then fails the node (as it should). This
happens only on EL8 Linux (AlmaLinux 8.8) nodes, whereas our
CentOS 7.9 nodes with Infiniband/OPA network work without
problems.
<br>
<br>
Question: Does anyone know how to reliably delay the start of the
slurmd Systemd service until the Infiniband/OPA network is fully
up?
<br>
<br>
Note: Our Infiniband/OPA network fabric is Omni-Path 100 Gbit/s,
not Mellanox IB. On AlmaLinux 8.8 we use the in-distro OPA
drivers since the CornelisNetworks drivers are not available for
RHEL 8.8.
<br>
<br>
The details:
<br>
<br>
The slurmd service is started by the service file
/usr/lib/systemd/system/slurmd.service after the
"network-online.target" has been reached.
<br>
<br>
It seems that NetworkManager reports "network-online.target"
BEFORE the Infiniband/OPA device ib0 is actually up, and this
seems to be the cause of our problems!
<br>
<br>
Here are some important sequences of events from the syslog
showing that the network goes online before the Infiniband/OPA
network (hfi1_0 adapter) is up:
<br>
<br>
Oct 30 13:01:40 d064 systemd[1]: Reached target Network is Online.
<br>
(lines deleted)
<br>
Oct 30 13:01:41 d064 slurmd[2333]: slurmd: error: health_check
failed: rc:1 output:ERROR: nhc: Health check failed:
check_hw_ib: No IB port is ACTIVE (LinkUp 100 Gb/sec).
<br>
(lines deleted)
<br>
Oct 30 13:01:41 d064 kernel: hfi1 0000:4b:00.0: hfi1_0: 8051: Link
up
<br>
Oct 30 13:01:41 d064 kernel: hfi1 0000:4b:00.0: hfi1_0:
set_link_state: current GOING_UP, new INIT (LINKUP)
<br>
Oct 30 13:01:41 d064 kernel: hfi1 0000:4b:00.0: hfi1_0: physical
state changed to PHYS_LINKUP (0x5), phy 0x50
<br>
<br>
I tried to delay the NetworkManager "network-online.target" by
setting a wait on the ib0 device and reboot, but that seems to be
ignored:
<br>
<br>
$ nmcli -p connection modify "System ib0"
connection.connection.wait-device-timeout 20
<br>
<br>
I'm hoping that other sites using Omni-Path have seen this and
maybe can share a fix or workaround?
<br>
<br>
Of course we could remove the Infiniband check in Node Health
Check (NHC), but that would not really be acceptable during
operations.
<br>
<br>
Thanks for sharing any insights,
<br>
Ole
<br>
<br>
</blockquote>
<div class="moz-signature">-- <br>
Max Rutkowski<br>
IT-Services und IT-Betrieb<br>
Tel.: +49 (0)331/6264-2341<br>
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:max.rutkowski@gfz-potsdam.de">max.rutkowski@gfz-potsdam.de</a><br>
___________________________________<br>
<br>
Helmholtz-Zentrum Potsdam<br>
<b>Deutsches GeoForschungsZentrum GFZ</b><br>
Stiftung des öff. Rechts Land Brandenburg<br>
Telegrafenberg, 14473 Potsdam</div>
</body>
</html>