Complete Bring-Up Flow — Whitebox Switch

 

🧭 Complete Bring-Up Flow — Whitebox Switch (Example: UfiSpace S9600-32X with SONiC)

              ┌─────────────────────────────────────┐
              │           BMC (AST2500/AST2600)      │
              │  • IPMI / Web console / SOL          │
              │  • Power control, sensors, fans      │
              │  • Out-of-band port (eth0)           │
              └─────────────────────────────────────┘
                               │
                               ▼
   ┌────────────────────────────────────────────────────────────┐
   │                   x86/ARM Host CPU                         │
   │  (where ONIE and later SONiC run)                          │
   └────────────────────────────────────────────────────────────┘
                               │
                         (boot process)
                               ▼

---

⚙️ Stage 1 – Power-On & BMC Initialization

1. BMC powers up first.

   • It’s an independent microcontroller (running its own Linux).

   • Controls fans, PSU, sensors, LEDs.

   • You can access it via the OOB Ethernet port before the host boots.

2. You can use IPMI commands:

   ipmitool -I lanplus -H <bmc_ip> -U admin -P <pwd> power status
   ipmitool ... power on
   

   So even if the NOS crashes, BMC stays alive.

---

🧩 Stage 2 – ONIE Bootloader

ONIE = Open Network Install Environment (like PXE + grub for switches).

1. The x86 CPU loads ONIE from flash.

2. ONIE brings up the management interface (usually eth0 or ma1).

3. It does a DHCP request with:

   • Option 60 (“onie_vendor”)

   • Option 61 (serial or MAC)

4. DHCP server responds with:

   • IP address for ONIE

   • Optional URL of NOS image (onie-installer)

5. ONIE downloads and installs the NOS image (e.g., SONiC .bin).

So you’ll see something like:

ONIE: Install Mode
Discovering installer via DHCP...
Downloading sonic.bin via HTTP...
Installing SONiC...

---

🐚 Stage 3 – NOS (SONiC) Boot

1. After installation, ONIE reboots → SONiC starts.

2. SONiC is a Linux-based NOS built from containers:

   • Host OS (Debian)

   • Docker containers:

     • bgp (FRR)

     • syncd (ASIC driver)

     • swss (Switch state service)

     • lldp, snmp, teamd, database …

---

🔧 Stage 4 – ASIC SDK & SAI Initialization

1. SONiC loads the vendor SDK (from Broadcom, Marvell, etc.).

2. The SDK exposes low-level APIs through SAI (Switch Abstraction Interface).

3. syncd uses SAI to configure the ASIC:

   SAI_create_port() → SDK → ASIC register → Port enabled
   SAI_create_vlan() → SDK → FDB table programmed
   

So at this stage, the data plane hardware is ready to forward packets.

---

📡 Stage 5 – Control Plane Setup

1. swss and database containers populate Redis with config info.

2. bgp (FRR) reads from Redis → starts routing daemons.

3. LLDP, DHCP relay, SNMP, etc. start in their respective containers.

4. The switch now has:

   • Management IP on eth0 / ma1

   • ASIC ports up and linked

   • Routing/bridging protocols active

---

🌐 Stage 6 – In Operation

Now you can:

• SSH to the management IP (via in-band or OOB)

• Use show interfaces status (reads ASIC counters via SAI)

• Control via gNMI, SNMP, or RESTCONF

The BMC continues to run independently — you can power-cycle, check temps, or do a serial-over-LAN session without touching SONiC.

---

🧱 Overall Architecture Summary

Layer	Component	Purpose
Hardware	ASIC (Q2C, Trident, Tofino etc.)	High-speed packet forwarding
SDK + SAI	Vendor library + SAI interface	Bridge between NOS and ASIC
NOS	SONiC containers	Network features, routing, mgmt
Bootloader	ONIE	NOS installation & recovery
OOB Controller	BMC (IPMI)	Power, sensors, remote access