Days 1 and 2 introduced CXL's three sub-protocols by name. Day 3 opens each one up. This is the most protocol-heavy lesson in Phase 1 — but understanding these three, and especially the opposite directions CXL.cache and CXL.mem move memory, is what makes everything about device types, pooling, and coherence click for the rest of the course.
Before the details, hold onto one idea: CXL.cache and CXL.mem both deal with coherent memory, but they point in opposite directions. CXL.cache is a device reaching into the host's memory. CXL.mem is the host reaching into the device's memory. That single distinction explains their entire structure — including why they use completely different terminology.
CXL.io is the foundation every CXL device must implement. It's based on PCIe and handles all the unglamorous but essential plumbing:
If a device supported only CXL.io, it would essentially be a PCIe device — no coherence, no shared memory. CXL.io is what makes the device a citizen of the system; the other two protocols are what make it a coherent one. As covered on Day 2, CXL.io flows through the PCIe-like stack, while the two coherent protocols share the latency-optimized stack.
CXL.cache lets a device keep a coherent cache of the host's memory, using the familiar MESI coherence protocol with a 64-byte cache line (the same cache-line size as the CPU, and the same 64 bytes that form the CXL flit payload from Day 2). This is what lets an accelerator work on shared data without constantly round-tripping to the CPU.
CXL.cache is built on three channels in each direction, and the two directions have specific names:
| Direction | Meaning | Channels |
|---|---|---|
| D2H | Device-to-Host | Request, Response, Data |
| H2D | Host-to-Device | Request, Response, Data |
The channels flow largely independently, which keeps throughput high. The D2H Request channel is where the device asks for things — it defines 15 commands grouped into four categories:
| Category | What the Device Is Asking For |
|---|---|
| Read | Coherence state and data for a cache line (host responds with state on H2D Response + data on H2D Data) |
| Read0 | Coherence state only, no data — e.g. to upgrade S→E, or acquire E-state when about to overwrite the whole line |
| Read0-Write | Write data to the host directly without holding any prior coherence state |
| Write | Write-back of modified data |
A deliberate simplification: the host manages all coherence tracking for peer caches. A CXL device never talks directly to another device's cache. If the host needs to invalidate a cache line held by a second device, the host issues a snoop-invalidate (SnpInv) on that device's H2D Request channel. This keeps every device's coherence logic simple — a device only ever coordinates with the host, never with its peers.
The host communicates coherence decisions back to the device using a Global Observation (GO) message on the H2D Response channel. A GO tells the device two things: the resulting MESI coherence state of the line, and the coherency commitment point — the moment the result becomes globally visible.
The channels are mostly independent, but there is one important ordering rule: a Snoop arriving from the host on the H2D Request channel must push any prior GO message for the same cache-line address on the H2D Response channel. In plain terms — the device must have already seen the outcome of its earlier operation on a line before it's asked to give that line up. This single ordering guarantee is what keeps the device's view of coherence consistent even though the channels otherwise flow freely.
CXL.mem is the mirror image of CXL.cache. Here the host is the one reaching across the link — treating memory physically attached to the device as cacheable system memory it can load from and store to. This is the protocol that underpins memory expanders, pooling, and disaggregation (Type 3 devices, Day 4).
Because the roles are reversed, CXL.mem uses entirely different, role-explicit terminology — master and subordinate:
M2S (Master-to-Subordinate) requests.S2M (Subordinate-to-Master) responses.So a CPU load from CXL-attached memory becomes an M2S request travelling to the device, and the returned data comes back as an S2M response. It's a clean, transactional master/subordinate interface — the CPU commands, the device memory serves. Contrast this directly with CXL.cache's D2H/H2D naming: the terminology tells you at a glance who initiates.
The three sub-protocols aren't all used by every device. A device implements the subset that matches its role — and that subset is its "type," which is exactly where Day 4 goes:
| Protocols Used | Resulting Device |
|---|---|
| CXL.io + CXL.cache | Type 1 — accelerator with a cache (reaches into host memory) |
| CXL.io + CXL.cache + CXL.mem | Type 2 — accelerator with its own coherent memory (both directions) |
| CXL.io + CXL.mem | Type 3 — memory expander (host reaches into device memory only) |