Day 1 said CXL "runs on the PCIe physical layer." Day 2 unpacks exactly what that means — how a single connector can be either PCIe or CXL, how the two ends decide at power-on, and the clever architectural split that lets CXL be both PCIe-compatible and low-latency at the same time.
CXL could have defined its own physical layer from scratch. It deliberately didn't — and that was one of the most important strategic decisions in its design. PCIe is a mature, high-volume, well-understood electrical standard with an entire ecosystem of connectors, retimers, form factors, and test equipment already in existence. By riding on PCIe's electricals, CXL inherited all of that for free, which is a large part of why it went from specification to shipping silicon so quickly. A brand-new physical layer would have meant years of ecosystem-building before the first real product.
The mechanism that makes this work is called Flex Bus. A Flex Bus port can operate as either native PCIe or CXL, and it decides which based on what device is plugged in — at link training time, not manufacturing time.
This is why you'll often see CXL described as running "over a PCIe slot": physically it is a PCIe slot. The difference is entirely in what protocol the two ends negotiate to speak across it.
The negotiation uses a PCIe 5.0 feature called Alternate Protocol Negotiation (APN) — a mechanism that lets a non-PCIe protocol use the PCIe physical layer while supplying its own higher layers. The way it's woven into PCIe's normal startup is elegant:
Note the neat backward-compatibility property this gives you: the link always starts as ordinary PCIe. Only if both ends explicitly agree on CXL does it switch. Plug in a device that knows nothing about CXL, and the negotiation simply never happens — the link stays native PCIe with zero special handling.
Link-training and APN details per CXL/PCIe specification tutorials and verification-IP documentation.
Here's the design decision that makes CXL genuinely interesting, not just "PCIe with coherence bolted on." CXL doesn't run all three sub-protocols through one shared stack. It runs two:
| Stack | Carries | Design Priority |
|---|---|---|
| CXL.io stack | CXL.io | Nearly identical to a standard PCIe stack, with dynamic message framing — general-purpose, flexible |
| CXL.cache + CXL.mem stack | CXL.cache, CXL.mem | A separate, latency-optimized stack with its own transaction and link layers and fixed message framing |
Why split them? Because the two workloads have opposite requirements. CXL.io is general I/O — discovery, configuration, DMA — where flexibility matters more than raw latency, and the mature PCIe stack is perfect. But CXL.cache and CXL.mem carry coherent memory traffic: a CPU issuing a simple load or store instruction that reaches across the link. That path has to be as low-latency as physically possible, because it sits directly in the critical path of the processor waiting for data.
The payoff: because the coherent stack uses fixed framing and its own streamlined transaction/link layers, CXL.mem lets a processor access remote memory by issuing ordinary load/store instructions at substantially lower latency than CXL.io or plain PCIe could ever provide. A single shared stack could not have optimized for both flexible I/O and lowest-possible memory latency at once — so CXL didn't try.
The latency-optimized stack matters enormously, but it's crucial to be honest about what it does and doesn't achieve: CXL-attached memory is still meaningfully slower than the CPU's own directly-attached DRAM. Real measurements bear this out:
| Memory Type | Typical Access Latency |
|---|---|
| Local (directly-attached) DRAM | ~80–140 ns |
| CXL-attached memory | ~170–250 ns |
| Added latency vs local DRAM | ~100–150 ns (one measured system: +153 ns) |
Latency figures per published CXL performance-characterization studies.
The most useful mental model: CXL memory behaves roughly like a two-hop-away NUMA node. A CPU accessing it pays a penalty similar to reaching across to memory attached to a distant socket — not free, but far cheaper than swapping to storage, and critically, still directly addressable with normal load/store instructions. Most CXL memory controllers add around 200 ns, plus or minus a few tens of nanoseconds depending on how many retimers sit between the CPU and the device (i.e. how physically far away it is). That distance-dependence scales: small pools of 8–16 nodes may add only 70–90 ns, while rack-scale pooling can push added latency past 180 ns.
Why this matters for the rest of the course: this latency profile is exactly why CXL isn't used to replace local DRAM, but to tier alongside it — hot data stays in fast local memory, while colder, capacity-heavy data lives in the CXL tier. That tiering strategy, and the software that manages it, is the entire subject of Days 12 and 13.
Because a CXL link is physically a PCIe link, its bandwidth follows PCIe's lane math directly. CXL runs on a PCIe 5.0 PHY at 32 GT/s per lane, in x4, x8, or x16 lane configurations (the x4/x8 options exist to support bifurcation — splitting a physical x16 slot into multiple narrower links). With 128b/130b encoding contributing under 2% overhead, each lane delivers roughly 3.94 GB/s of usable bandwidth per direction:
| Link Width | Per-Direction Bandwidth | Bidirectional Total |
|---|---|---|
| x4 | ~16 GB/s | ~32 GB/s |
| x8 | ~32 GB/s | ~64 GB/s |
| x16 | ~64 GB/s | ~128 GB/s |
A key property inherited from PCIe: the link is full-duplex. Each lane has independent transmit and receive paths that operate simultaneously, so an x16 CXL 1.1/2.0 link can move ~64 GB/s each way at the same time. When Day 9 reaches CXL 4.0 on the PCIe 7.0 PHY (128 GT/s), these per-lane numbers roughly quadruple relative to PCIe 5.0 — but the lane-width math stays identical.
Like most modern serial protocols, CXL moves data in fixed-size flits (flow-control units). In CXL 1.0, 1.1, and 2.0, that flit is 68 bytes, laid out precisely:
The 2-byte protocol ID identifies which sub-protocol this flit belongs to (this is how three protocols multiplex onto one link). The 64-byte payload is no accident — it's exactly the size of a CPU cache line, which is the natural unit of coherent memory transfer. The 2-byte CRC protects the payload against bit errors. You'll see this flit grow to 256 bytes in CXL 3.x (Day 10) to make room for forward error correction, mirroring PCIe 6.0 — but the 68-byte flit is the foundation the first three CXL generations were built on.