B300 32-Server Farm + Standard-Aligned Network Tower — Architecture Specification

Date: March 30, 2026

Scope: 32-node deployed NVIDIA B300 compute farm with 72-node-standard-aligned network, management, and border architecture

Zones: 32-rack compute zone + refreshed network/services placement envelope

---

Overview

The Kedios package remains a 32-node NVIDIA Blackwell Ultra B300 training farm, but the network-side architecture basis is now refreshed. The compute population stays fixed at 32 servers / 256 GPUs. What changes is the way the report presents the surrounding network, management, border, and storage-network layers: they now follow the accepted 72-node standard B300 baseline rather than the old minimal custom-stack narrative.

This file therefore uses a dual-layer rule:

1. Current deployment layer: 32 compute nodes, 256 GPUs, live Q3400-RA compute fabric
2. Standard-aligned network layer: retained Spectrum-4 plus SN5610, SN4700, SN2201, and dual UFM

The old N1/N2 occupied, N3–N6 empty, single-UFM, and generic one-switch OOB wording should no longer be treated as authoritative source text.

---

1. Physical Layout and Zone Structure

Locked Design Rules

Compute Zone — 32-Rack Grid

The compute zone remains unchanged:

• 32 rack positions
• 32 ASUS XA NB3I-E12 B300 servers
• no networking hardware inside the compute racks as the base reporting assumption
• unchanged cold-aisle / hot-aisle and dual-feed PDU-A / PDU-B logic

The racks remain numbered C01 through C32 and still land within the working range of the existing inter-zone cable-length assumptions.

Network / Services Placement Envelope

The network-side narrative changes materially. The refreshed topology should no longer assume that only N1 and N2 are populated and that N3–N6 are simply future reserve. Instead, treat the broader network/services area as the placement envelope for:

• Q3400-RA compute fabric core
• retained Spectrum-4 pair
• SN5610 converged / storage-network layer
• SN4700 border and control layer
• SN2201 management layer
• dual UFM nodes

---

2. Compute Zone — Per-Rack Build (×32)

Each of the 32 compute rack positions is built identically. The server is an ASUS XA NB3I-E12 carrying one NVIDIA HGX B300 × 8 tray. Each rack carries exactly one server.

The server occupies the bottom 9U of the rack (U1–U9). A patch panel at U11 terminates the management cables, and a cable management arm at U12–U13 organises the 10 heavy AOC cable bundles. The remaining 29U above U13 is intentionally left empty.

Power is delivered via two vertical in-rack PDUs on the rear posts — one fed from PDU-A, one from PDU-B. The 5+5 PSU array in the server pairs each bank with one PDU, giving N+5 PSU redundancy plus dual-feed facility redundancy simultaneously.

Each rack's external cabling consists of:

• 8× NDR800 AOC cables (5–10 m) carrying the 8 ConnectX-8 IB ports to the 8 leaf switches in Rack N1
• 2× NDR400 AOC cables (5–10 m) carrying the 2 BlueField-3 DPU ports to the 2 Spectrum-4 switches in Rack N2
• 2× management links (5–10 m) carrying the X710 management NIC ports (OS management + BMC/IPMI) into the SN2201-based management layer

Every rack produces the same 12-cable bundle. Across all 32 racks, this creates 256 IB AOC cables, 64 Ethernet AOC cables, and 64 Cat6A management cables landing in the network tower.

---

3. IB Compute Fabric — Live Deployment

The Q3400-RA compute fabric remains the live cross-server training fabric for the current 32-node farm. The refresh does not change the compute-fabric count or rail design.

Rail Logic

Each server still maps one CX8 port per GPU rail:

• rail 0 to leaf L0
• rail 1 to leaf L1
• through rail 7 to leaf L7

This keeps AllReduce traffic rail-local whenever possible and preserves the current 32-node training performance assumptions.

SHARP and Fabric Management

All 12 Q3400-RA switches still support SHARP Gen 4 and the usual UFM-managed functions:

• topology discovery
• routing computation
• adaptive routing
• congestion response
• telemetry aggregation

What changes in this refresh is the management posture around the IB fabric: the base architecture now assumes 2 UFM nodes rather than a single-UFM default.

Inter-Switch Cabling

The locked compute-fabric inter-switch count remains:

• 256 passive DAC links for Q3400 leaf-to-spine connectivity

Those counts remain valid because the live 8-leaf / 4-spine compute fabric itself has not changed.

---

4. Standard-Aligned Ethernet, Management, Border, and UFM Layers

The refreshed network stack now carries four explicit non-compute layers.

Spectrum-4

Spectrum-4 remains part of the accepted topology by explicit owner direction. It is not replaced by the standard-aligned additions. The 64 BF3-facing server links therefore stay valid and continue to represent the live Ethernet side-fabric path for the current 32-node farm.

SN5610

The SN5610 layer is locked at 6 total units, using the owner-confirmed 2 + (2 + 2) formula. In this source revision, the count and the role are both locked, while the final per-unit rack and port labels are deferred to the draw.io refresh.

SN4700

The SN4700 layer is locked at 4 total units:

• 2 border leaf
• 2 C-spine / OOB-related

This is now part of the accepted external-connectivity and control-plane story.

SN2201

The management layer is locked at 17 total units, using the owner-confirmed 9 + 8 split. This replaces the previous generic single-switch OOB narrative as the base design position.

The server-side management-port mapping remains unchanged until the final ASUS integration package says otherwise:

• X710 Port 0 = OS management
• X710 Port 1 = BMC / IPMI

UFM

The topology now assumes 2 UFM nodes as the base architecture. Present them as production + standby / HA pair unless a stronger implementation detail is later confirmed.

The reason for moving from one to two UFM nodes is operational resilience and standards alignment, not raw port-count exhaustion.

---

5. Inter-Zone Cable Infrastructure

All cables from the 32 compute racks still run through the overhead tray path into the refreshed network/services placement envelope. The locked server-side cable counts are:

Additional cable counts for SN5610, SN4700, SN2201, and the second UFM node must be finalized in the draw.io refresh. Do not reuse the old all-in cable total from the pre-refresh topology as the final number for the refreshed design.

---

6. Power and Capacity Framing

Important distinction:

• 1.5 MW is the facility-allocation envelope
• 680 kW is the current rack-constrained hard ceiling under the unchanged rack rule

Because the refreshed network baseline introduces SN5610, SN4700, SN2201, and a second UFM node, this source revision does not claim a new all-network watt total until the final vendor power sheets are attached to the refreshed procurement BOM.

---

7. Split Baseline vs Current Deployment Snapshot

This is now the mandatory reporting rule for downstream summaries, questionnaires, and diagrams:

• compute population differs between the standard baseline and the live deployment
• network-side switch counts do not in the current working design