Router/Switch Architecture

🧩 1️⃣ Physical View — What’s inside a Whitebox Switch

Inside your UfiSpace S9600-32X (or similar platforms), the architecture looks like this:


+------------------------------------------------------+
|        Switch System-on-Chip (Broadcom Q2C)          |
|                                                      |
|   +-----------------+        +------------------+     |
|   |  Packet ASIC    |<-----> |  CPU (x86/ARM)  |     |
|   +-----------------+        +------------------+     |
|         |  |  |  |                   |                 |
|        32x100G ports               Mgmt (ma1)          |
|                                                      |
|  [ BMC Chip (IPMI) ] — out-of-band mgmt               |
+------------------------------------------------------+

So:

ASIC handles all the packet forwarding, port control, VLAN, ACL, routing, etc.
CPU runs Linux/NOS (like RtBrick, SONiC, or ONL) and programs the ASIC via software APIs.
BMC is just for hardware control (fans, power, sensors, boot).

🧩 2️⃣ How the ASIC Ports Become “Visible”

When the switch boots up (ONIE or NOS):

The Linux kernel sees only the management NIC (ma1) — not the ASIC ports.
Those front-panel ports are not standard Ethernet NICs.
The ASIC vendor SDK (like Broadcom’s OpenNSA / BCM SDK) loads, which provides APIs to:
- Initialize the switch silicon
- Create port objects
- Map them to front-panel connectors
- Bring them UP
The NOS (Network OS) uses this SDK to create Linux-visible network interfaces — for example:
```
Ethernet1, Ethernet2, Ethernet3 ... Ethernet32
```
These are virtual representations of the real ASIC ports.

🧩 3️⃣ Software Stack — How Programming Happens

Let's visualize the software stack 👇


+-----------------------------------------+
|            Network OS (e.g., RtBrick)   |
|  - Configures ports, VLANs, BGP, etc.   |
|  - Uses SAI or proprietary driver       |
+---------------------┬-------------------+
                      |
+---------------------┴-------------------+
|   Hardware Abstraction Layer (HAL) / SAI|
|   - Maps OS-level config to ASIC SDK    |
|   - Translates “create VLAN 10” → SDK calls
+---------------------┬-------------------+
                      |
+---------------------┴-------------------+
|         ASIC SDK (e.g. Broadcom BCM SDK)|
|         - Configures PHYs, MACs, SerDes |
|         - Programs TCAMs, LPM tables    |
|         - Handles packet pipeline       |
+---------------------┬-------------------+
                      |
+---------------------┴-------------------+
|              ASIC Hardware              |
|        (Q2C / Trident / Tomahawk)       |
+-----------------------------------------+

🧩 4️⃣ Example — Bringing Up ASIC Ports (Broadcom Platform)

Step 1: Platform Init (ONL or SONiC)

When the NOS boots:


/etc/onl/platform_init.sh

Runs something like:


bcmcmd "port enable 1-32"
bcmcmd "port pbmp show"

→ These commands come from the Broadcom SDK (bcmcmd) to initialize the 32x100G ports.

Step 2: NOS creates interfaces

The NOS (or ONL) maps these to Linux netdevs:


Ethernet1 -> ASIC port 1
Ethernet2 -> ASIC port 2
...
Ethernet32 -> ASIC port 32

You can see them using:


ip link show

They look like interfaces, but traffic bypasses the Linux kernel datapath — it’s all in ASIC silicon.

🧩 5️⃣ Control Plane vs Data Plane

Plane	Runs on	Handles	Example Interface
Control Plane	CPU	Routing protocols, management	`ma1`, `loopback`
Data Plane	ASIC	Actual packet forwarding	`Ethernet1-32`

So, when you do ping, BGP, or DHCP in NOS — control packets come to CPU (ma1),
but once the routes are programmed, forwarding happens entirely in ASIC hardware.

🧩 6️⃣ How the Switch Is “Managed”

Out-of-band (BMC/IPMI): power, console, sensors, remote recovery.
In-band (via ma1 or inband-mgmt): management traffic over IP.
ASIC SDK / HAL / SAI: handles internal port and hardware control.
NOS (like RtBrick / SONiC): provides the configuration, APIs, and protocols.

🧩 Summary

Layer	Role	Example
BMC (IPMI)	Hardware-level mgmt	Power/reset/sensors
ONIE	Bootloader environment	Install NOS
NOS (e.g., RtBrick)	Network OS	Routing, switching, mgmt
ASIC SDK / HAL	Hardware programming	SerDes, VLANs, TCAMs
ASIC ports (front-panel)	Data plane	Ethernet1–Ethernet32
ma1	Mgmt interface	CPU-side mgmt (Linux-visible)

🧩 1️⃣ Boot sequence recap


+----------------------+
| Power on / ONIE boot |
+----------------------+
          ↓
+----------------------+
| Load Linux kernel    |
| Detect CPU NIC (ma1) |
+----------------------+
          ↓
+----------------------+
| Load platform drivers|
| (fans, sensors, I2C, CPLD) |
+----------------------+
          ↓
+----------------------+
| Load Broadcom SDK    |
| (/etc/onl/platform_init.sh) |
+----------------------+
          ↓
+----------------------+
| SDK initializes ASIC |
| Ports, PHYs, SerDes  |
+----------------------+
          ↓
+----------------------+
| NOS starts (RtBrick, SONiC) |
| Maps ASIC ports → Linux netdevs |
+----------------------+

🧩 2️⃣ SDK initialization (Broadcom BCM SDK)

When ONIE or ONL starts, it runs something like this:


/etc/onl/platform_init.sh

Inside that script (for Broadcom platforms), you’ll see:


modprobe linux_user_bde
modprobe linux_kernel_bde
modprobe linux_bcm_knet

bcm.user start
bcmcmd "init"
bcmcmd "port pbmp show"
bcmcmd "port enable 1-32"

🧩 3️⃣ Checking ASIC ports using `bcmcmd`

Once the SDK is loaded, you can run:


root@onl# bcmcmd
BCM.0>

You’re now inside the Broadcom SDK shell — a direct interface to the ASIC.

🧠 Example 1: Show all front-panel ports


BCM.0> port pbmp show
Port Bitmaps:
    fe: <none>
    ge: <none>
    xe: 1-32

➡️ This means your ASIC has 32 x 100G ports (called “xe” ports — X for 10 G base unit, e for Ethernet).

🧠 Example 2: Enable and verify ports


BCM.0> port enable 1-32
BCM.0> ps

You’ll get a table like:


          ena/        speed/ link auto    STP                  lrn  inter   max
port      link  Lns   duplex scan neg?    state  pause  discrd ops   face  frame
----      ----  ----  ------ ---- ----    -----  -----  ------ ----  ----- -----
1          up    4   100G   SW   Yes      FWD    TX/RX   None  FA   CR4    9216
2          up    4   100G   SW   Yes      FWD    TX/RX   None  FA   CR4    9216
...
32         up    4   100G   SW   Yes      FWD    TX/RX   None  FA   CR4    9216

So the SDK directly programs the MAC, PHY, and SerDes lanes for each port.

🧠 Example 3: Show SerDes mapping (lane configuration)


BCM.0> phy info 1

Might output something like:


Port 1: PHY addr=0x1, type=QSFP28, lanes=4, speed=100G, serdes=lane0-3

→ Each 100 G port uses 4 × 25 G SerDes lanes internally.

🧩 4️⃣ How the NOS (like RtBrick or SONiC) maps these ports

Once the ASIC SDK is ready, the NOS runs its HAL (Hardware Abstraction Layer) or SAI (Switch Abstraction Interface) layer.

Example mapping table (simplified):

NOS interface	ASIC logical port	SDK port ID	SerDes lanes	Connector
Ethernet1	xe1	1	0–3	QSFP1
Ethernet2	xe2	2	4–7	QSFP2
Ethernet3	xe3	3	8–11	QSFP3
...	...	...	...	...
Ethernet32	xe32	32	124–127	QSFP32

In SONiC you can see this mapping in /etc/sonic/port_config.ini

In RtBrick, it’s in the platform definition JSON (under /etc/rbfs/platforms/...json).

🧩 5️⃣ Traffic path overview

Once the ports are up and the NOS programs forwarding entries:


   ASIC Hardware Pipeline
+--------------------------------------------+
|  Ingress Parser → VLAN/QoS → L3 FIB → Egress|
|   (All programmed via SDK)                 |
+--------------------------------------------+
             ↓
           Physical QSFP ports (1–32)

So when packets arrive, the ASIC pipeline processes them at line rate — the CPU only handles:

BGP/OSPF control packets
Management / CLI / SSH traffic
Exceptions (e.g., TTL expiry)

🧩 6️⃣ Useful commands for debugging (Broadcom-based)

Command	Purpose
`bcmcmd "ps"`	Show port status
`bcmcmd "show c"`	Show chip info
`bcmcmd "d chg l2 show"`	Show L2 MAC table
`bcmcmd "l3 egress show"`	Show L3 next-hops
`bcmcmd "version"`	Show SDK version
`bcmcmd "port pbmp show"`	Show available ports

🧩 7️⃣ Summary

Layer	Tool / Component	Purpose
ONIE / Linux	Kernel drivers	Boot and detect platform
SDK	`bcmcmd` / `bcm.user`	Initialize ASIC ports
HAL / SAI	Platform-specific layer	Translate NOS config to SDK calls
NOS (RtBrick / SONiC)	`rbfshell` / `config interface`	User CLI, BGP, VLANs, etc.
ASIC Hardware	Q2C	Does all forwarding in silicon

Search This Blog