Kedios Infrastructure Report

Kedios B300 Server Farm — Planning Summary

March 30, 202632-node deployed B300 compute farm with 72-node-standard-aligned network baseline

Kedios B300 Server Farm — Planning Summary

Date: March 30, 2026

Scope: 32-node deployed B300 compute farm with 72-node-standard-aligned network baseline

Purpose: High-level planning and review summary for the current source-of-truth markdown set


1. Executive Position

The Kedios package remains a 32-node NVIDIA B300 training farm. The change in this revision is not a move to 72 deployed servers. Instead, the network, management, border, and BOM basis now aligns to the 72-node B300 standard reference pattern, while the live compute population remains 32 nodes / 256 GPUs.

This summary therefore follows a two-layer rule:

  1. current deployment = 32 compute nodes, 256 GPUs
  2. standard-aligned network baseline = retained Spectrum-4 plus SN5610, SN4700, SN2201, and dual UFM

2. Locked Planning Figures

ItemLocked value
Current deployed compute nodes32
Current deployed GPUs256
Standard network reference capacity72 nodes / 576 GPUs
IB compute fabric8 × Q3400-RA leaf + 4 × Q3400-RA spine
Retained Ethernet side fabric2 × Spectrum-4
Added standard-aligned layersSN5610 ×6, SN4700 ×4, SN2201 ×17, UFM ×2
Contracted package34 occupied racks
Facility allocation1.5 MW
Rack-constrained ceiling680 kW under the retained 20 kW/rack rule

3. Power and Rack Framing

Validated Compute-Side Anchors

LayerValue
32 compute racks sustained~464 kW
32 compute racks burst peak~492 kW
32 compute rack hard ceiling640 kW
34-rack package hard ceiling680 kW

Allocation Rule

Use these two facts together:

  • 1.5 MW = facility-allocation envelope
  • 680 kW = current rack-constrained hard ceiling

Because the refreshed network baseline adds SN5610, SN4700, SN2201, and a second UFM node, this planning summary does not publish a new all-in network watt total until the refreshed procurement BOM carries the missing per-model power entries.


4. Network Architecture Summary

Live Compute Fabric

ComponentCountPlanning note
Q3400-RA leaf8One per GPU rail
Q3400-RA spine4Keeps 1:1 non-blocking bisection
Total Q3400-RA12Live compute-fabric count remains unchanged
IB bisection bandwidth204.8 Tb/sLive 32-node compute-fabric value

Non-Compute Network Layers

Component familyCountPlanning note
Spectrum-42Retained BF3-facing Ethernet side fabric
UFM nodes2Production + standby / HA pair
SN56106Locked owner formula: 2 + (2 + 2)
SN470042 border leaf + 2 C-spine / OOB
SN220117Locked owner formula: 9 + 8

Management Rule

The old single 96-port OOB switch story is withdrawn. The base management model is now the SN2201-based management layer.

The server-side port mapping remains:

  • X710 Port 0 = OS management
  • X710 Port 1 = BMC / IPMI

5. Split Baseline vs Current Deployment

Component family72-node standard baselineCurrent 32-node populated deployment
Compute nodes7232
GPUs576256
Q3400-RA leaf88
Q3400-RA spine44
Spectrum-422
UFM nodes22
SN561066
SN470044
SN22011717
UFM Agent (software)7232

The practical meaning is simple:

  • the compute population differs between baseline and deployment
  • the network-side switch counts do not in the current working design

6. Placement and Cabling Rules

Locked Server-Side Cable Counts

Link familyCount
CX8 to Q3400 leaf256
BF3 to Spectrum-464
Server management links64
Q3400 leaf to Q3400 spine256

Placement Rule

The old assumption that only N1 and N2 are occupied is no longer authoritative. Treat N1–N6 as the logical network/services placement envelope until the refreshed draw.io diagrams lock the final physical placement of:

  • Spectrum-4
  • SN5610
  • SN4700
  • SN2201
  • dual UFM nodes

7. Downstream Reporting Rules

Every downstream summary, questionnaire answer, or viewer card must preserve these distinctions:

  1. Never imply that 72 compute nodes are deployed today.
  2. Present 32 deployed nodes and 72-node standard baseline as separate labeled facts.
  3. Do not reuse the old 1 MW, 1 UFM, 1 generic OOB switch, or N3–N6 empty wording.
  4. Do not publish a refreshed all-network watt total until the final BOM provides the missing switch power entries.

Glossary

NDR
Next Data Rate — InfiniBand generation at 400 Gb/s (NDR400) or 800 Gb/s (NDR800) per physical port.
NDR400
InfiniBand NDR at 400 Gb/s per port, used by the BlueField-3 DPU for side-fabric connections.
NDR800
InfiniBand NDR at 800 Gb/s per port, used by ConnectX-8 HCAs on the HGX B300 GPU-to-fabric links.
ConnectX-8
NVIDIA ConnectX-8 NDR800 InfiniBand HCA integrated on the HGX B300 tray — 8 per server, one per GPU rail.
BlueField-3
NVIDIA BF-3220 DPU — 400G NDR400 InfiniBand, provides side-fabric connectivity and in-network compute offload.
Q3400-RA
NVIDIA Quantum-X800 Q3400 Rail-Accelerated InfiniBand switch — 144 NDR ports; deployed as 8 leaf + 4 spine.
Spectrum-4
NVIDIA Spectrum-4 400GbE/InfiniBand Ethernet switch — 51.2 Tb/s; retained as active-active side-fabric pair.
SN5610
NVIDIA Spectrum-SN5610 converged 400G Ethernet switch — 6 units in the storage/converged service plane.
SN4700
NVIDIA Spectrum-SN4700 400G Ethernet switch — 4 units for border/WAN handoff and control-plane.
SN2201
NVIDIA Spectrum-SN2201 1G/10G management switch — 17 units covering the full OOB management layer.
UFM
Unified Fabric Manager — NVIDIA IB fabric management; deployed as 2-node HA pair (production + standby).
SHARP
Scalable Hierarchical Aggregation and Reduction Protocol — in-network collective offload on Q3400-RA.
HGX B300
NVIDIA HGX Blackwell Ultra B300 — 8-GPU tray with NVLink Gen 5 at 1.8 TB/s per GPU, 14.4 TB/s aggregate across the tray.
B300 GPU
NVIDIA Blackwell Ultra B300 — 288 GB HBM3e, 1.1 kW TDP; current report basis uses ~4.5 PFLOPS FP8 dense / ~9 PFLOPS FP8 sparse and ~15 PFLOPS NVFP4 dense / ~30 PFLOPS NVFP4 sparse per GPU.
NVLink
NVIDIA direct GPU interconnect — Gen 5 on Blackwell at 1.8 TB/s per GPU, yielding 14.4 TB/s across an 8-GPU HGX B300 tray.
HBM3e
High Bandwidth Memory 3e — stacked DRAM in B300 GPUs at 288 GB per GPU, 8 TB/s peak bandwidth.
Fat-Tree
Network topology providing non-blocking bisection bandwidth; IB compute fabric is a 2-tier rail-optimised fat-tree.
Rail-Optimised
IB fabric layout: each GPU rail maps to a dedicated leaf switch, keeping AllReduce traffic rail-local.
AOC
Active Optical Cable — fibre-based cable with integrated E/O conversion, used for all NDR800 IB inter-rack links.
IPMI / BMC
Intelligent Platform Management Interface / Baseboard Management Controller — out-of-band server management.
PDU-A / PDU-B
Dual-feed power distribution: each PSU bank pairs with one PDU, giving N+5 PSU + dual-feed facility redundancy.
CRAC / CRAH
Computer Room Air Conditioner / Air Handler — precision cooling units, N+1 target coverage in the Kedios facility.
DPU
Data Processing Unit — BlueField-3 Smart NIC providing network/storage offload and security isolation.
XA NB3I-E12
ASUS server SKU: 9U air-cooled, dual Xeon 6776P, 32 × 128 GB DDR5 (4 TB total), 10× NVMe, HGX B300 ×8, CX-8 ×8, BF-3 ×2.
Xeon 6776P
Intel Xeon 6 Granite Rapids-SP — 56-core, PCIe 5.0 host CPU in the XA NB3I-E12 server; current server power tables in this repo model ~350 W per socket.
NVFP4
NVIDIA FP4 format — current report basis uses ~15 PFLOPS dense / ~30 PFLOPS sparse per B300 GPU, reported in this repo as ~240 PFLOPS sparse per 8-GPU server.
FP8
8-bit float — current report basis uses ~4.5 PFLOPS dense / ~9 PFLOPS sparse per B300 GPU, with the report itself citing ~36 PFLOPS dense per 8-GPU server.
AllReduce
Distributed-training collective operation across all GPUs; accelerated by IB fat-tree fabric and SHARP.
Fat-Tree Bisection BW
204.8 Tb/s across the full 32-server farm — 1:1 non-blocking, no fabric oversubscription.
20 kW Rack Limit
Hard power cap per rack in the Kedios facility; servers draw ~14.5 kW sustained, leaving 5.5 kW margin.