B300 Single Rack Server Unit — Architecture Specification

Date: February 28, 2026

Scope: One compute rack unit as deployed across all 32 positions in the Kedios B300 farm

Server: ASUS XA NB3I-E12 with NVIDIA HGX B300 × 8

---

Overview

Each rack in the compute zone is a self-contained, single-server compute unit. The ASUS XA NB3I-E12 is a 9U air-cooled behemoth purpose-built for the NVIDIA HGX B300 tray assembly. The philosophy is simple: one rack, one server, every slot dedicated to compute or its direct support. There is no cable aggregation inside the rack — every external link exits the chassis directly to the network tower in the adjacent zone.

Physically, the server occupies units U1 through U9 from the bottom of the rack. Above it sits a single 1U rear-mounted patch tray and a 2U cable management arm. The rest of the rack is empty by design except for the in-rack PDU mounted vertically on the rear post, which feeds the dual redundant PSU banks.

---

1. Chassis and Form Factor

The ASUS XA NB3I-E12 is a 9U rackmount chassis designed exclusively to house the NVIDIA HGX B300 tray. It uses a front-to-back forced air cooling architecture — cold aisle air enters the front face through GPU tunnel shrouds and exits into the hot aisle at the rear. There are no liquid cooling loops. The chassis dimensions are standard 19-inch rack width.

The server sits at the bottom of the rack (U1–U9) to keep the centre of mass low and to align the cable exits — which are mid-chassis at approximately U5 — with the cable management arm directly above.

Above the server from U10 onward:

• U10: 1U blanking panel (thermal buffer)
• U11: 1U 24-port patch panel — used only for the 10 GbE management NIC breakout cables (neat bundling before the management cable run exits the rack top)
• U12–U13: 2U cable management arm — horizontal channels and retention clips for the 8× NDR800 IB AOC trunk cables and 2× NDR400 Ethernet AOC trunk cables
• U14–U42: Empty — intentionally unused, available for cold-aisle bypass baffles if airflow optimization is required later

The rear of the rack carries:

• Dual vertical in-rack PDUs (one on each rear post) — each PDU fed from a separate building feed for redundancy (PDU-A branch and PDU-B branch)
• Earthing bar at the bottom of the rear, bonded to rack frame

---

2. Power System

The server ships with 10× 3,200 W 80 PLUS Titanium PSUs arranged in a 5+5 hot-swap redundant configuration. In normal operation all ten PSUs share load equally. The loss of five PSUs simultaneously is survivable without shutdown, though GPU power limits would be imposed.

Each PSU pair is fed from a dedicated circuit:

• PSU bay 1–5 → draws from the PDU-A vertical strip (right rear post)
• PSU bay 6–10 → draws from the PDU-B vertical strip (left rear post)

The in-rack PDUs are rated for 32A per phase, three-phase input. Each PDU provides a monitored and switched outlet per PSU, giving remote power cycling capability per PSU individually without touching neighbouring units.

At sustained training load the server draws approximately 14.5 kW at the wall. Using the same instantaneous GPU boost overshoot assumption (~+6%), the absolute peak is ~15.4 kW, well within the updated 20 kW per-rack wall-output ceiling with ~4.6 kW thermal/electrical headroom remaining.

No UPS is inside the rack. UPS protection is building-level, handled by UPS Rooms #1, #2, and #3, with power routed through PCU 1 and PCU 2 before reaching the rack-mounted PDUs.

---

3. GPU Subsystem — NVIDIA HGX B300 × 8

The HGX B300 tray is the defining hardware of this server. It is a board-level assembly carrying eight NVIDIA Blackwell Ultra B300 GPUs, all interconnected via NVLink 5 at 1.8 TB/s per GPU (14.4 TB/s full bisection across the tray). Every GPU is co-located on the same tray, so all intra-server GPU-to-GPU communication happens over NVLink on the board — no IB hop required for within-server traffic.

Each of the eight GPUs carries:

• 288 GB HBM3e (stacked in 12-high HBM3e packages) — giving a total of 2.304 TB of GPU memory per server
• 1,100 W TDP (ASUS datasheet confirmed) — total GPU thermal load 8,800 W in the worst case
• Full NVLink 5 mesh to all seven other GPUs on the same tray
• One dedicated on-board ConnectX-8 (CX8) NIC, soldered directly to the HGX baseboard, providing 800 Gb/s NDR InfiniBand connectivity for inter-server traffic

The HGX tray sits in a sealed tunnel shroud inside the chassis. Cooling airflow is directed specifically across the GPU package and HBM stacks by the shroud geometry. GPU junction temperature is monitored per device via the CX8 telemetry path through UFM Agent.

---

4. CPU and Host Subsystem

Two Intel Xeon Platinum 6776P processors sit on the host motherboard, separate from the HGX tray electrically but connected via PCIe Gen 5. Each 6776P runs at a 350 W TDP (ASUS official) for a combined CPU thermal budget of 700 W. The CPUs serve primarily as orchestrators — issuing kernel launches to the GPUs, managing NVMe storage, and running the UFM Agent and OS daemons. They are not in the training compute critical path for MFU (model FLOP utilization) purposes.

System memory is 32× 128 GB Samsung DDR5-6400 RDIMM modules (4 TB total per server). All 32 DIMMs run at rated 6,400 MT/s speed. Memory is used for model staging, host-side communication buffers, and OS/hypervisor needs.

---

5. Storage

Three storage tiers are present on each server:

Tier 1 — Boot / OS (2× 1.92 TB U.2 NVMe):

Samsung PM9D3a Gen5 drives in a mirrored pair. Houses the operating system, container runtime, and the NCCL/MPI libraries. Not used for training data I/O.

Tier 2 — Fast scratch / checkpoint (8× 3.84 TB U.2 NVMe):

Eight Samsung PM9D3a Gen5 drives in a non-RAID stripe for maximum sequential write bandwidth. Used for model checkpointing, gradient accumulation scratch, and fast-load dataset staging. Aggregate raw capacity: 30.72 TB. Total server NVMe: 34.56 TB.

Tier 3 — Network storage (via BlueField-3 DPU):

The two BF3 DPUs (see section 7) provide a high-speed path to the farm-wide shared storage fabric over NDR400 Ethernet. Distributed storage (e.g., Lustre, GPFS, or object store) is accessible at up to 400 Gb/s per DPU, 800 Gb/s aggregate.

---

6. InfiniBand Compute Fabric Connections (CX8 × 8)

Each of the eight ConnectX-8 adapters on the HGX baseboard connects to the external IB compute fabric via a single 800 Gb/s NDR OSFP port. Inside the chassis, this exits through eight OSFP breakout panels on the rear of the server at approximately U4–U6. Outside the chassis, the cables run up through the cable management arm and exit the top of the rack toward the network tower.

Cable type for all eight CX8 links: Active Optical Cable (AOC), 5–10 m, selected because the compute zone and network tower zone are physically adjacent but not co-located in the same row. Passive DAC is not viable beyond ~3 m.

Each CX8 port is assigned to a specific GPU rail:

• CX8[0] (GPU 0) → Leaf Switch L0 in the network tower
• CX8[1] (GPU 1) → Leaf Switch L1 in the network tower
• CX8[2] (GPU 2) → Leaf Switch L2 in the network tower
• CX8[3] (GPU 3) → Leaf Switch L3 in the network tower
• CX8[4] (GPU 4) → Leaf Switch L4 in the network tower
• CX8[5] (GPU 5) → Leaf Switch L5 in the network tower
• CX8[6] (GPU 6) → Leaf Switch L6 in the network tower
• CX8[7] (GPU 7) → Leaf Switch L7 in the network tower

This rail-optimized assignment means that all AllReduce operations within a single GPU rail (e.g., GPU 0 across all 32 servers) traverse only one leaf switch — no cross-rail traffic until the spine layer is reached. This dramatically reduces hot-spot congestion during distributed training.

---

7. Ethernet Storage and DPU Connections (BlueField-3 × 2)

Two NVIDIA BlueField-3 3220 DPUs are installed in PCIe slots on the host motherboard. Each BF3 provides a single 400 Gb/s NDR400 OSFP port facing the Spectrum-4 AI Ethernet fabric in the network tower.

The two BF3 are connected to different Spectrum-4 switches (BF3[0] → Spectrum-4 #1, BF3[1] → Spectrum-4 #2) for active-active redundancy. Under normal operation both paths carry load. BF3 adapters also run the onboard ARM cores for SmartNIC offload functions including RDMA over Converged Ethernet (RoCE), storage protocol acceleration, and network security policy enforcement — offloaded from the host CPU.

Cable type for both BF3 links: AOC 5–10 m, same reason as CX8.

---

8. Out-of-Band Management

The current rack-level management design maps the Intel X710-AT2 dual-port 10 GbE management NIC to two independent management paths:

• Port 0 → OS-level management network (SSH, monitoring agents, Prometheus exporters, UFM Agent communications)
• Port 1 → BMC/IPMI out-of-band management path (used for remote power cycling, BIOS configuration, KVM-over-IP, and sensor polling)

Both ports break out through Cat6A patch cables to the in-rack patch panel at U11, then run as bundled Cat6A trunks to the OOB management switch in the network tower. The 10 GbE copper standard is fully viable for these inter-zone management runs — copper Ethernet is rated to 100 m at 10G, which comfortably spans the adjacent-zone distance of ~5–15 m.

This port mapping is consistent with the current topology design and supporting hardware notes, but it should still be validated against the final ASUS server integration before production sign-off.

---

9. Cable Exit Summary per Rack

The 8 CX8 AOC cables are the dominant cable consideration per rack. Across 32 server racks, this produces 256 CX8 AOC cables total terminating at the 8 leaf switches in the network tower — an average of 32 AOC cables landing on each leaf switch, exactly matching the 32 downlink ports per leaf.

---

10. Power Consumption Summary