Kedios B300 — Full Network & Topology Report

Organization: Kedios

Date: March 30, 2026

Scope: 32-Node NVIDIA Blackwell Ultra B300 GPU Farm with 72-Node-Standard-Aligned Network Baseline

Status: Source Architecture Refresh Locked · Diagram Redraw Pending

---

1. Executive Summary

The Kedios B300 package remains a 32-node NVIDIA Blackwell Ultra B300 GPU training cluster inside a 20 MW facility, but the network architecture, BOM framing, and management-plane narrative are now refreshed to align with the 72-node B300 standard baseline. The compute population stays fixed at 32 servers / 256 GPUs. The standards-aligned layer now governs how the report presents InfiniBand control, Ethernet side fabric, border, storage-network, and management components.

This means the report intentionally carries two truths at once:

1. the current deployed compute scope is still a 32-node farm
2. the accepted network baseline is now a 72-node-standard-aligned design pattern

The compute fabric remains a 1:1 non-blocking Q3400-RA InfiniBand fat-tree delivering 204.8 Tb/s of server-side bisection bandwidth for the currently deployed 32-node farm. At the same time, the Ethernet, storage-network, management, and border layers are no longer described as a minimal custom add-on. They are now expressed as the standard-aligned design basis for the current package.

The platform remains optimized for collective-heavy distributed training and post-training workflows. Selective inference and checkpoint-adjacent workloads remain supportable, but they are not the sizing basis for the fabric or rack-power architecture.

Farm at a Glance

The earlier ~530 kW all-in peak remains the last fully validated package-level figure from the pre-refresh network stack. Because the refreshed baseline adds SN5610, SN4700, SN2201, and a second UFM node, this revision separates validated compute-side power anchors from the refreshed network-side BOM instead of overstating a new all-in watt total before the final vendor power sheets are attached.

---

2. Data Center Infrastructure & Power

Facility Overview

Per-Server Power Audit (Component-Level)

Server Rack vs 20 kW Hard Limit

Server fits within the 20 kW hard limit. Sustained training peak of ~14.5 kW leaves 5.5 kW margin. Even at absolute burst (~15.4 kW) the margin is 4.6 kW — no risk of breaker trips.

Validated Power Anchors

Network-Side Power Refresh Status

The refreshed network baseline adds hardware families that were not part of the earlier all-in package roll-up. The current source set supports counts, roles, and topology placement intent for these elements, but it does not yet support a final consolidated watt total for every added model.

Use 1.5 MW as the facility-allocation answer and 680 kW as the current rack-constrained ceiling under the unchanged 20 kW/rack rule. Do not promote a new all-network aggregate watt number until the refreshed procurement BOM carries the missing per-model power entries.

---

3. Physical Layout & Zone Structure

Zone Allocation

The project still remains a 34-rack package. The updated draw.io sources must finalize the physical placement of the refreshed network layers instead of reusing the previous assumption that only N1 and N2 are populated.

Compute Zone — 32-Rack Grid (C01–C32)

• Racks numbered C01 through C32 — pure compute, 1 server per rack
• Alternating cold aisle / hot aisle layout — conditioned air from CRAC #1 and CRAC #2
• Dual-feed from PDU-A and PDU-B across all rows (N+1 power path redundancy)
• Each rack: server occupies bottom U1–U9 (9U); patch panel at U11; cable management U12–U13; upper 29U intentionally empty
• Each rack exits 12 cables: 8× IB AOC + 2× Ethernet AOC + 2× Cat6A copper to network tower

Network Zone — Standard-Aligned Placement Envelope

The compute-side physical rule still holds: long server-to-network runs cross the inter-zone gap and therefore use AOC or copper management cabling rather than passive DAC. What changes in this revision is the network-layer occupancy narrative:

• the Q3400-RA core remains the live compute-fabric anchor
• the Spectrum-4 pair remains the live BF3 side fabric
• the management, border, and storage-network layers are now part of the accepted standard-aligned design basis
• the prior statement that N3–N6 are simply empty spare racks is withdrawn

Until the diagram refresh is completed, treat N1–N6 as the logical network/services placement envelope rather than as a final U-by-U rack map.

---

4. Server Specification — ASUS XA NB3I-E12

Model: ASUS XA NB3I-E12

Form factor: 9U, air-cooled (front-to-back), direct airflow

Quantity deployed: 32 units (one per rack, C01–C32)

Per-Server Hardware Bill of Materials

Per-Server Performance

32-Server Farm Aggregate

---

5. Network Architecture — InfiniBand Compute Fabric

Network Ports Per Server

32-Server Farm Port Totals

The old 80 total ports on one generic OOB switch figure is retired in this revision. Management, border, and control-plane connectivity now follow the standard-aligned SN2201 + SN4700 + dual-UFM model rather than a single generic 96-port switch story.

InfiniBand Rail Assignment

Each server's 8 CX8 ports are assigned to 8 independent GPU rails:

Rail isolation principle: GPU-i on server X and GPU-i on server Y share the same leaf switch. This means AllReduce within a rail (the dominant training communication pattern) traverses zero spine hops — leaf-only latency. Only cross-rail operations (AllToAll) require the 2-hop leaf→spine→leaf path.

---

6. Switch Topology — 2-Tier Rail-Optimized Fat-Tree

Topology Design

                    ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
  SPINES (×4)       │  Spine   │ │  Spine   │ │  Spine   │ │  Spine   │
                    │   S0     │ │   S1     │ │   S2     │ │   S3     │
                    │Q3400-RA  │ │Q3400-RA  │ │Q3400-RA  │ │Q3400-RA  │
                    │144p NDR  │ │144p NDR  │ │144p NDR  │ │144p NDR  │
                    └──┬───────┘ └──┬───────┘ └──┬───────┘ └──┬───────┘
                       │ (32 links per spine, 8 per leaf)      │
        ┌──────┬────────┼──────┬─────┬──────┬────┼────┬────────┼──────┐
        │      │        │      │     │      │    │    │        │      │
  ┌─────┴──┐ ┌─┴──────┐ ... ┌──┴─────┴──┐ ... ┌─┴────┴──┐ ┌──┴─────┐
  │ Leaf L0│ │ Leaf L1│     │  Leaf L6  │     │  Leaf L7│ │        │
  │Rail 0  │ │Rail 1  │     │  Rail 6   │     │  Rail 7 │ │        │
  │Q3400-RA│ │Q3400-RA│     │  Q3400-RA │     │Q3400-RA │ │        │
  └────┬───┘ └────┬───┘     └─────┬─────┘     └────┬────┘ └────────┘
       │          │               │                 │
  (32 downlinks per leaf — one CX8[n] port from each server)
       │          │               │                 │
  C01..C32   C01..C32        C01..C32           C01..C32
  CX8[0]     CX8[1]          CX8[6]             CX8[7]

Q3400-RA Specifications

Leaf Switch Design (×8 — L0 through L7)

Spine Switch Design (×4 — S0 through S3)

Fat-Tree Bandwidth & Latency

The 8-leaf / 4-spine Q3400 topology remains the live compute fabric for the current 32-node farm. The 72-node standard alignment changes the surrounding Ethernet, management, border, and storage-network layers rather than the rail mapping or the 32-node compute-population count.

---

7. Network Tower — Rack N1 (InfiniBand Fabric)

Physical Content — 12 × Q3400-RA (48U total)

All 12 Q3400-RA switches are co-located in the network tower (Rack N1, or split across N1 IB bay and N2 upper bay as needed for physical depth). Co-location enables all inter-switch links to be short passive DAC, eliminating active components from the spine-to-leaf path.

Leaf-to-Spine Internal Wiring

SHARP In-Network Computation

• All 12 Q3400-RA support SHARP Gen 4
• AllReduce operations partially executed inside switches — partial gradient sums computed cooperatively by leaf and spine
• Reduces fabric bandwidth consumed per AllReduce iteration by up to ~50% for training batch sizes common in LLM pretraining
• Orchestrated automatically by UFM Appliance; no application code changes required
• Particularly impactful for gradient synchronization in large transformer models (GPT, LLaMA scale)

---

8. Standard-Aligned Ethernet, Management & UFM Layers

Layering Rule

This revision separates the non-compute network stack into four explicit layers:

1. Retained Spectrum-4 side fabric for the current 32-node BF3 path
2. SN5610 standard-aligned converged / storage-network layer
3. SN4700 border and control layer
4. SN2201 management layer with dual UFM nodes

These are now treated as accepted topology components for the Kedios package. The report therefore stops describing management and border networking as a single generic custom switch and instead presents them as the standard-aligned design basis serving the current 32-node farm.

Spectrum-4 AI Ethernet Fabric (Retained)

The Spectrum-4 pair remains in the topology by explicit design decision. It continues to carry the BF3-facing Ethernet path for the current 32-node farm.

Traffic carried by the retained Spectrum-4 fabric:

• BF3-based storage and service traffic for the 32 deployed servers
• secondary compute-adjacent Ethernet traffic
• orchestration and control-plane services that are intentionally kept off the IB data fabric

SN5610 Converged / Storage-Network Layer

The refreshed standard baseline introduces an SN5610 layer with a locked count of 6 total units, using the owner-confirmed 2 + (2 + 2) formula. In this report revision, the count is locked and the role is locked, while the final sub-role placement is deferred to the updated diagrams.

UFM High-Availability Pair

The refreshed baseline upgrades UFM from a single-appliance default to a dual-node base architecture.

UFM responsibilities remain unchanged in type:

• topology discovery
• routing computation
• adaptive routing and congestion response
• SHARP orchestration
• fault detection and recovery
• telemetry aggregation

The reason for moving to 2 × UFM is operational resilience and standards alignment, not raw port-count exhaustion. One UFM node is still likely sufficient for raw capacity, but it is no longer the preferred base-design statement for the standards-aligned network package.

SN2201 Management Layer

The management plane now moves from a generic 96-port 10 GbE OOB switch narrative to an SN2201-based management layer.

The current management-port mapping rule remains unchanged until final server integration says otherwise:

• X710 Port 0 = OS management
• X710 Port 1 = BMC / IPMI path

SN4700 Border / Control Layer

The refreshed baseline also adds a fixed SN4700 border and control layer:

These units should now be treated as part of the IDC-facing and control-layer architecture rather than as optional future add-ons.

Placement Rule

This source revision locks the counts and roles above, but it does not claim a final U-by-U rack drawing for them yet. The next draw.io refresh must convert these locked counts into the final physical placement view.

---

9. Cabling Plan

Locked Cable Counts From The Current 32-Node Compute Population

Standard-Aligned Network-Layer Additions

The refreshed baseline adds cable families that were not present in the old ~654 total cable assemblies roll-up:

• SN5610 interconnects
• SN4700 border and control-layer patching
• SN2201 management fan-out and uplinks
• second-UFM management links

Those counts should be finalized in the diagram redraw and the refreshed procurement BOM, because the exact values depend on the final physical placement and link map. For that reason:

• keep the compute-side cable counts above as locked
• do not reuse the old ~654 total as a final all-in cable total for the refreshed topology

---

10. Complete Hardware Bill of Materials

Compute Hardware — Current Deployment

Split Baseline vs Current Deployment BOM

Rack Infrastructure

---

11. Aggregate Bandwidth Summary

---

12. Farm Compute Performance

---

13. Topology Diagrams

Three rendered SVG topology views are published through the shared diagram viewer. Each one opens in a centered review window with fit, 100%, zoom, and scroll controls for detailed inspection.

Diagram 1 — IDC Farm Planning & Connectivity

Open rendered SVG viewer

Block-flow diagram showing the full end-to-end connectivity path:

Internet → Router → Firewall → Border Leaf → Spectrum-4 (active-active) → Q3400-RA IB Fabric → 32× B300 Compute Servers

Also shows: OOB Management Switch, UFM Appliance, Management networking path, zone separation, fabric management connections.

---

Diagram 2 — Farm InfiniBand Topology (NVIDIA-Style Rail View)

Open rendered SVG viewer

Full IB topology in the NVIDIA reference style:

• Row 1: 4× Q3400-RA Spine switches (S0–S3)
• Row 2: 8× Q3400-RA Leaf switches (L0–L7), each rail color-coded
• Row 3: 32× server nodes (C01–C32), grouped in 4 sets of 8
• 256 rail-colored CX8→Leaf connections (8 colors, one per rail)
• 32 orange spine-to-leaf connections
• Ethernet section: Spectrum-4 #1/#2, UFM, OOB, Storage (future)
• 64 BF3→Spectrum-4 connections

---

Diagram 3 — Single Rack Leaf/Spine Connection Detail

Open rendered SVG viewer

Per-port detail for a single representative server rack (C01):

• All 8 CX8 ports to 8 leaf switches (color-coded by rail)
• All 8 leaf switches to 4 spine switches (orange, 8×4 = 32 links)
• BF3-0 → Spectrum-4 #1 (purple AOC)
• BF3-1 → Spectrum-4 #2 (purple AOC)
• X710 Port 0 + Port 1 → OOB Management Switch (yellow dashed)
• UFM Agent (SW) ← UFM Appliance (blue dashed)
• UFM Appliance → Leaf switches for IB fabric management (blue dashed)
• Connection legend (color key for all edge types)

Racks C02–C32 are identical — same 12-port topology, same cable types, same switch targets.

---

14. Key Design Decisions & Rationale

---

15. Risk & Mitigation Notes

---

Report generated: February 28, 2026

Architecture status: Locked

Diagrams: ✅ farm-idc-topology · ✅ farm-ib-topology · ✅ single-rack-topology

Next actions: Commit to repository · Stakeholder review · Procurement kick-off