We're in the process of installing some racks with Lenovo SD665 V3 [1] water-cooled servers. A Lenovo DW612S chassis contains 6 1U trays with 2 SD665 V3 servers mounted side-by-side in each tray. Lenovo delivers SD665 V3 servers including water-cooled NVIDIA InfiniBand "SharedIO" adapters [2] so that one node is the Primary including a PCIe adapter, and the other is Auxiliary with just a cable to the Primary's adapter. Obviously, servicing 2 "Siamese twin" Slurm nodes requires a bit of care and planning. What is worse is that when the Primary node is rebooted or powered down, the Auxiliary node will lose its Infiniband connection and may have a PCIe fault or an NMI as documented in [3]. And when nodes are powered up, the Primary must have completed POST before the Auxiliary gets started. I wonder how to best deal with power failures? It seems that when Slurm jobs are running on Auxiliary nodes, these jobs are going to crash when the possibly unrelated Primary node goes down. This looks like a pretty bad system design on the part of Lenovo :-( The goal was apparently to same some money on IB adapters and having fewer IB cables. Question: Do any Slurm sites out there already have experiences with Lenovo "Siamese twin" nodes with SharedIO IB? Have you developed some operational strategies, for example dealing with node pairs as a single entity for job scheduling? Thanks for sharing any ideas and insights! Ole [1] https://lenovopress.lenovo.com/lp1612-lenovo-thinksystem-sd665-v3-server [2] https://lenovopress.lenovo.com/lp1693-thinksystem-nvidia-connectx-7-ndr200-i... [3] https://support.lenovo.com/us/en/solutions/ht510888-thinksystem-sd650-and-co... -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark
Thats a Very interesting design and looking at the SD665 V3 documentation am I correct each node has dual 25GBs SFP28 interfaces? If so, the despite dual nodes in a 1u configuration, you actually have 2 separate servers? Sid On Fri, 23 Feb 2024, 22:40 Ole Holm Nielsen via slurm-users, < slurm-users@lists.schedmd.com> wrote:
We're in the process of installing some racks with Lenovo SD665 V3 [1] water-cooled servers. A Lenovo DW612S chassis contains 6 1U trays with 2 SD665 V3 servers mounted side-by-side in each tray.
Lenovo delivers SD665 V3 servers including water-cooled NVIDIA InfiniBand "SharedIO" adapters [2] so that one node is the Primary including a PCIe adapter, and the other is Auxiliary with just a cable to the Primary's adapter.
Obviously, servicing 2 "Siamese twin" Slurm nodes requires a bit of care and planning. What is worse is that when the Primary node is rebooted or powered down, the Auxiliary node will lose its Infiniband connection and may have a PCIe fault or an NMI as documented in [3]. And when nodes are powered up, the Primary must have completed POST before the Auxiliary gets started. I wonder how to best deal with power failures?
It seems that when Slurm jobs are running on Auxiliary nodes, these jobs are going to crash when the possibly unrelated Primary node goes down.
This looks like a pretty bad system design on the part of Lenovo :-( The goal was apparently to same some money on IB adapters and having fewer IB cables.
Question: Do any Slurm sites out there already have experiences with Lenovo "Siamese twin" nodes with SharedIO IB? Have you developed some operational strategies, for example dealing with node pairs as a single entity for job scheduling?
Thanks for sharing any ideas and insights!
Ole
[1] https://lenovopress.lenovo.com/lp1612-lenovo-thinksystem-sd665-v3-server [2]
https://lenovopress.lenovo.com/lp1693-thinksystem-nvidia-connectx-7-ndr200-i... [3]
https://support.lenovo.com/us/en/solutions/ht510888-thinksystem-sd650-and-co...
-- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
We have these cards in some sd650v1 servers. You get 2 nodes in a 1u configuration, but they are attached, you can only pull both out of the rack at once. Ours are slightly older, so we only have 1x 1Gb on-board per server, plus 1x 200Gb HDR port on the B server, which provides a “virtual” 200G 4xHDR port on each node, although I think in practice they function as a 2xHDR 100G port on each server. I think the sd655v3 will have 2x 25G SPF+ NICs on each node, plus the 1 or 2 200G NDR QSFP112 ports provided by the ConnectX7 card on the B server, shared with the A server. You can totally reboot either server without affecting the other. You will just see something like: [ 356.799171] mlx5_core 0000:58:00.0: mlx5_fw_tracer_start:821:(pid 819): FWTracer: Ownership granted and active As the “owner” fails over from one node to the other. However, doing a full power off of the B server, will crash the A node: bear-pg0206u28a: 02/24/2024 15:36:57 OS Stop, Run-time critical Stop (panic/BSOD) (Sensor 0x46) bear-pg0206u28a: 02/24/2024 15:36:58 Critical Interrupt, Bus Uncorrectable Error (PCIs) bear-pg0206u28a: 02/24/2024 15:37:01 Slot / Connector, Fault Status asserted PCIe 1 (PCIe 1) bear-pg0206u28a: 02/24/2024 15:37:03 Critical Interrupt, Software NMI (NMI State) bear-pg0206u28a: 02/24/2024 15:37:07 Module / Board, State Asserted (SharedIO fail) And this will put the fault light on. Once the B node is back, the A node will recover OK and you need to do virtual reseat or just restart the BMC to clear the fault. So in day to day usage we don’t generally notice. It can be a bit of pain during outages or reinstall – obviously updating firmware on the B node will take out the A node too as the card needs to be reset. But they are fairly rare, and we don’t do anything special except rebooting/reinstalling the A nodes after the B nodes are all done to clear any errors. Oh, and the A node doesn’t show up in the IB fabric: [root@bear-pg0206u28a ~]# ibnetdiscover ibwarn: [24150] _do_madrpc: send failed; Invalid argument ibwarn: [24150] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0) /var/tmp/OFED_topdir/BUILD/rdma-core-58mlnx43/libibnetdisc/ibnetdisc.c:811; Failed to resolve self ibnetdiscover: iberror: failed: discover failed [root@bear-pg0206u28a ~]# So we can’t use automated topology generation scripts (without a little special casing). Cheers, Luke -- Luke Sudbery Principal Engineer (HPC and Storage). Architecture, Infrastructure and Systems Advanced Research Computing, IT Services Room 132, Computer Centre G5, Elms Road Please note I don’t work on Monday. From: Sid Young via slurm-users <slurm-users@lists.schedmd.com> Sent: Friday, February 23, 2024 9:49 PM To: Ole Holm Nielsen <Ole.H.Nielsen@fysik.dtu.dk> Cc: Slurm User Community List <slurm-users@lists.schedmd.com> Subject: [slurm-users] Re: Slurm management of dual-node server trays? CAUTION: This email originated from outside the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Thats a Very interesting design and looking at the SD665 V3 documentation am I correct each node has dual 25GBs SFP28 interfaces? If so, the despite dual nodes in a 1u configuration, you actually have 2 separate servers? Sid On Fri, 23 Feb 2024, 22:40 Ole Holm Nielsen via slurm-users, <slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> wrote: We're in the process of installing some racks with Lenovo SD665 V3 [1] water-cooled servers. A Lenovo DW612S chassis contains 6 1U trays with 2 SD665 V3 servers mounted side-by-side in each tray. Lenovo delivers SD665 V3 servers including water-cooled NVIDIA InfiniBand "SharedIO" adapters [2] so that one node is the Primary including a PCIe adapter, and the other is Auxiliary with just a cable to the Primary's adapter. Obviously, servicing 2 "Siamese twin" Slurm nodes requires a bit of care and planning. What is worse is that when the Primary node is rebooted or powered down, the Auxiliary node will lose its Infiniband connection and may have a PCIe fault or an NMI as documented in [3]. And when nodes are powered up, the Primary must have completed POST before the Auxiliary gets started. I wonder how to best deal with power failures? It seems that when Slurm jobs are running on Auxiliary nodes, these jobs are going to crash when the possibly unrelated Primary node goes down. This looks like a pretty bad system design on the part of Lenovo :-( The goal was apparently to same some money on IB adapters and having fewer IB cables. Question: Do any Slurm sites out there already have experiences with Lenovo "Siamese twin" nodes with SharedIO IB? Have you developed some operational strategies, for example dealing with node pairs as a single entity for job scheduling? Thanks for sharing any ideas and insights! Ole [1] https://lenovopress.lenovo.com/lp1612-lenovo-thinksystem-sd665-v3-server [2] https://lenovopress.lenovo.com/lp1693-thinksystem-nvidia-connectx-7-ndr200-i... [3] https://support.lenovo.com/us/en/solutions/ht510888-thinksystem-sd650-and-co... -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark -- slurm-users mailing list -- slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave@lists.schedmd.com<mailto:slurm-users-leave@lists.schedmd.com>
Hi Luke, Thanks very much for your feedback about Lenovo SD650 V1 water-cooled servers. The new SD665 V3 also consists of 2 AMD Genoa servers in a shared tray. I have now installed RockyLinux 8.9 on the nodes and tested the Infiniband connectivity. Fortunately, Lenovo/Mellanox/Nvidia seem to have fixed the Infiniband "SharedIO" adapters in the latest generation (V3 ?) of servers so that the Primary node can be rebooted or even powered off *without* causing any Infiniband glitches on the Auxiliary (left-hand) node. This is a great relief to me :-) What I did was to run a ping command on the IPoIB interface on the Auxiliary (left-hand) node to another node in the cluster, while rebooting or powering down the Primary node. Not a single IP packet was lost. Best regards, Ole On 3/1/24 18:25, Luke Sudbery wrote:
We have these cards in some sd650v1 servers.
You get 2 nodes in a 1u configuration, but they are attached, you can only pull both out of the rack at once.
Ours are slightly older, so we only have 1x 1Gb on-board per server, plus 1x 200Gb HDR port on the B server, which provides a “virtual” 200G 4xHDR port on each node, although I think in practice they function as a 2xHDR 100G port on each server. I think the sd655v3 will have 2x 25G SPF+ NICs on each node, plus the 1 or 2 200G NDR QSFP112 ports provided by the ConnectX7 card on the B server, shared with the A server.
You can totally reboot either server without affecting the other. You will just see something like:
[ 356.799171] mlx5_core 0000:58:00.0: mlx5_fw_tracer_start:821:(pid 819): FWTracer: Ownership granted and active
As the “owner” fails over from one node to the other.
However, doing a full power off of the B server, will crash the A node:
bear-pg0206u28a: 02/24/2024 15:36:57 OS Stop, Run-time critical Stop (panic/BSOD) (Sensor 0x46)
bear-pg0206u28a: 02/24/2024 15:36:58 Critical Interrupt, Bus Uncorrectable Error (PCIs)
bear-pg0206u28a: 02/24/2024 15:37:01 Slot / Connector, Fault Status asserted PCIe 1 (PCIe 1)
bear-pg0206u28a: 02/24/2024 15:37:03 Critical Interrupt, Software NMI (NMI State)
bear-pg0206u28a: 02/24/2024 15:37:07 Module / Board, State Asserted (SharedIO fail)
And this will put the fault light on. Once the B node is back, the A node will recover OK and you need to do virtual reseat or just restart the BMC to clear the fault.
So in day to day usage we don’t generally notice. It can be a bit of pain during outages or reinstall – obviously updating firmware on the B node will take out the A node too as the card needs to be reset. But they are fairly rare, and we don’t do anything special except rebooting/reinstalling the A nodes after the B nodes are all done to clear any errors.
Oh, and the A node doesn’t show up in the IB fabric:
[root@bear-pg0206u28a ~]# ibnetdiscover
ibwarn: [24150] _do_madrpc: send failed; Invalid argument
ibwarn: [24150] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0)
/var/tmp/OFED_topdir/BUILD/rdma-core-58mlnx43/libibnetdisc/ibnetdisc.c:811; Failed to resolve self
ibnetdiscover: iberror: failed: discover failed
[root@bear-pg0206u28a ~]#
So we can’t use automated topology generation scripts (without a little special casing).
Cheers,
Luke
--
Luke Sudbery
Principal Engineer (HPC and Storage).
Architecture, Infrastructure and Systems
Advanced Research Computing, IT Services
Room 132, Computer Centre G5, Elms Road
*Please note I don’t work on Monday.*
*From:*Sid Young via slurm-users <slurm-users@lists.schedmd.com> *Sent:* Friday, February 23, 2024 9:49 PM *To:* Ole Holm Nielsen <Ole.H.Nielsen@fysik.dtu.dk> *Cc:* Slurm User Community List <slurm-users@lists.schedmd.com> *Subject:* [slurm-users] Re: Slurm management of dual-node server trays?
*CAUTION:*This email originated from outside the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe.
Thats a Very interesting design and looking at the SD665 V3 documentation am I correct each node has dual 25GBs SFP28 interfaces?
If so, the despite dual nodes in a 1u configuration, you actually have 2 separate servers?
Sid
On Fri, 23 Feb 2024, 22:40 Ole Holm Nielsen via slurm-users, <slurm-users@lists.schedmd.com <mailto:slurm-users@lists.schedmd.com>> wrote:
We're in the process of installing some racks with Lenovo SD665 V3 [1] water-cooled servers. A Lenovo DW612S chassis contains 6 1U trays with 2 SD665 V3 servers mounted side-by-side in each tray.
Lenovo delivers SD665 V3 servers including water-cooled NVIDIA InfiniBand "SharedIO" adapters [2] so that one node is the Primary including a PCIe adapter, and the other is Auxiliary with just a cable to the Primary's adapter.
Obviously, servicing 2 "Siamese twin" Slurm nodes requires a bit of care and planning. What is worse is that when the Primary node is rebooted or powered down, the Auxiliary node will lose its Infiniband connection and may have a PCIe fault or an NMI as documented in [3]. And when nodes are powered up, the Primary must have completed POST before the Auxiliary gets started. I wonder how to best deal with power failures?
It seems that when Slurm jobs are running on Auxiliary nodes, these jobs are going to crash when the possibly unrelated Primary node goes down.
This looks like a pretty bad system design on the part of Lenovo :-( The goal was apparently to same some money on IB adapters and having fewer IB cables.
Question: Do any Slurm sites out there already have experiences with Lenovo "Siamese twin" nodes with SharedIO IB? Have you developed some operational strategies, for example dealing with node pairs as a single entity for job scheduling?
Thanks for sharing any ideas and insights!
Ole
[1] https://lenovopress.lenovo.com/lp1612-lenovo-thinksystem-sd665-v3-server <https://lenovopress.lenovo.com/lp1612-lenovo-thinksystem-sd665-v3-server> [2] https://lenovopress.lenovo.com/lp1693-thinksystem-nvidia-connectx-7-ndr200-i... <https://lenovopress.lenovo.com/lp1693-thinksystem-nvidia-connectx-7-ndr200-infiniband-qsfp112-adapters> [3] https://support.lenovo.com/us/en/solutions/ht510888-thinksystem-sd650-and-co... <https://support.lenovo.com/us/en/solutions/ht510888-thinksystem-sd650-and-connectx-6-hdr-sharedio-lenovo-servers-and-storage>
-- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark
participants (3)
-
Luke Sudbery -
Ole Holm Nielsen -
Sid Young