Kedios B300 Full Network
& Topology Report

The Kedios B300 farm is a 32-node NVIDIA Blackwell Ultra B300 GPU training cluster deployed within a 20 MW data center, consuming approximately 530 kW peak out of a 1 MW allocated budget (53%). The farm is purpose-built for large-scale distributed AI training — specifically AllReduce/AllGather collective operations — backed by a 1:1 non-blocking InfiniBand NDR800 fabric delivering 204.8 Tb/s bisection bandwidth.

Facility Overview

Per-Server Power Audit (Component-Level)

Rack Power vs 20 kW Hard Limit

Farm Zone Power Summary

Compute Zone — 32-Rack Grid (C01–C32)

• Alternating cold aisle / hot aisle layout — conditioned air from CRAC #1 and CRAC #2
• Dual-feed from PDU-A and PDU-B across all rows — N+1 power path redundancy
• Each rack: server at U1–U9; patch panel at U11; cable management U12–U13; upper 29U empty
• Each rack exits 12 cables: 8× IB AOC + 2× Ethernet AOC + 2× Cat6A → network tower

Network Tower — N1 & N2 Rack Layout

Form factor: 9U, air-cooled (front-to-back), direct airflow  |  Quantity: 32 units (one per rack, C01–C32)

Per-Server Hardware BOM

Per-Server Performance

32-Server Farm Aggregates

Network Ports Per Server

32-Server Farm Port Totals

InfiniBand GPU Rail Assignment

Each server's 8 CX8 ports map to 8 independent GPU rails. All GPU-i ports across all 32 servers land on the same leaf switch Li, isolating GPU-to-GPU communication by rail.

         ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐
SPINES   │  Spine S0  │ │  Spine S1  │ │  Spine S2  │ │  Spine S3  │
(×4)     │ Q3400-RA   │ │ Q3400-RA   │ │ Q3400-RA   │ │ Q3400-RA   │
         │ 144p NDR   │ │ 144p NDR   │ │ 144p NDR   │ │ 144p NDR   │
         └──────┬─────┘ └──┬─────────┘ └────────┬───┘ └─────┬──────┘
                │  (32 uplinks per leaf × 8 leaves = 256 spine ports)
   ┌────────────┼──────────┼────────────────────┼───────────┼─────┐
   │            │          │                    │           │     │
 ┌─┴───────┐ ┌──┴──────┐  ...  ┌───────────┐  ... ┌────────┴──┐  │
 │ Leaf L0 │ │ Leaf L1 │       │  Leaf L6  │      │  Leaf L7  │  │
 │ Rail 0  │ │ Rail 1  │       │  Rail 6   │      │  Rail 7   │  │
 │Q3400-RA │ │Q3400-RA │       │  Q3400-RA │      │ Q3400-RA  │  │
 └────┬────┘ └────┬────┘       └─────┬─────┘      └─────┬─────┘  │
      │           │                  │                   │
 (32 downlinks per leaf — one CX8[n] from every server)
      │           │                  │                   │
 C01..C32    C01..C32           C01..C32             C01..C32
 CX8[0]      CX8[1]             CX8[6]               CX8[7]

NVIDIA Quantum-X800 Q3400-RA Specifications

Leaf Switch Design (×8 — L0 through L7)

Spine Switch Design (×4 — S0 through S3)

Fat-Tree Bandwidth & Latency

Rack N1 houses all 12 × Q3400-RA switches co-located. Co-location enables all inter-switch links to use short passive DAC — no active components in the spine-to-leaf forwarding path.

Internal Wiring (Leaf → Spine)

SHARP Gen 4 In-Network Computation

• All 12 Q3400-RA support SHARP Gen 4
• Partial gradient sums computed inside switches during AllReduce — reduces fabric BW by up to ~50%
• Particularly impactful for LLM pretraining (GPT/LLaMA scale) with large gradient tensors
• Orchestrated automatically by UFM Appliance — no application code changes required

Physical Layout

Spectrum-4 AI Ethernet Fabric

UFM Appliance

UFM Appliance functions: topology discovery, routing table computation, adaptive routing, SHARP orchestration, fault detection/recovery, per-port telemetry aggregation.

Cable Totals

Networking Hardware

Rack Infrastructure

Three diagrams were produced, available in .drawio (editable) and .png (rendered) in network-topology-diagrams/.

Diagram 1

IDC Farm Planning & Connectivity

farm-idc-topology.drawio / .png

End-to-end connectivity block flow — shows all major components and their interconnections:

• External uplink path: Internet → Router → Firewall → Border Leaf
• Compute fabric: Spectrum-4 active-active → Q3400-RA IB Fabric (leaves + spines) → 32× B300 servers
• Management path: OOB switch → all servers (BMC + OS) + UFM Appliance → IB fabric managers
• Zone separation clearly indicated: compute zone (right) vs. network tower (center)

Diagram 2

Farm InfiniBand Topology — Rail View

farm-ib-topology.drawio / .png

NVIDIA-style full IB topology — 3-row layout matching NVIDIA reference diagrams:

• Row 1 (Spines): 4× Q3400-RA S0–S3 (orange)
• Row 2 (Leaves): 8× Q3400-RA L0–L7 — each color-coded by rail (matching the 8 GPU rail colors)
• Row 3 (Servers): 32× compute nodes C01–C32 grouped in 4 sets of 8
• 256 rail-colored CX8→Leaf connections (colored by rail) + 32 orange spine-to-leaf connections
• Ethernet section: Spectrum-4 #1/#2, UFM, OOB — 64 BF3→Spectrum-4 connections

Diagram 3

Single Rack Leaf/Spine Connection Detail

single-rack-topology.drawio / .png

Per-port detail for a single representative server rack (C01) — C02–C32 are identical:

• 8× CX8 NDR800 AOC connections → 8 leaf switches (color-coded by rail, 2.5 px)
• 8× leaf → 4× spine links (orange, 8×4 = 32 total leaf-spine connections)
• BF3-0 → Spectrum-4 #1 | BF3-1 → Spectrum-4 #2 (purple AOC)
• X710 Port 0 + Port 1 → OOB switch (yellow dashed = management)
• UFM Appliance → server (blue dashed = UFM Agent SW) + UFM → Leaf L0 (blue dashed = IB fabric mgmt)
• Connection color legend at bottom