HBM3 Write Data Path — Module 5 | HBM3 Controller Build

1. Write Data Path Overview

In an HBM3 controller the write data path is the bridge between the command scheduler (which fires a WRITE command to the DRAM) and the physical DQ bus (which delivers actual data into the DRAM's sense amplifiers). These two events are not simultaneous. The DRAM sees the WRITE command on the CA (command/address) bus at cycle T, but it expects data to arrive CWL cycles later — by default, 36 cycles at the HBM3 nominal clock rate of 2 GHz.

The write path module must handle four distinct jobs:

Buffering — accept write data from the host before the DQ bus is ready, storing it in the Write Data Buffer (WDB) FIFO so the system bus stays un-stalled.
Delay pipeline — track the CWL countdown from each WRITE command so data leaves the FIFO at the exact correct cycle.
Burst serialization — HBM3 sends 128 bits per pseudo-channel in four 32-bit beats (BL4). The host supplies all 128 bits at once; the serializer chops this into a timed sequence.
DM / DQS generation — the Data Mask bits gate individual byte lanes, and the DQS differential strobe must be driven by the controller during writes.

This module sits directly above the PHY layer. Its outputs (o_dq_out, o_dm_out, o_dqs_en) are registered signals that the PHY samples on every half-cycle to drive the bidirectional HBM3 I/O pads.

2. CAS Write Latency (CWL) — Why 36 Cycles?

CWL stands for CAS Write Latency — the number of clock cycles between the rising edge that registers a WRITE command and the rising edge on which the controller must present the first valid data nibble on DQ. The JEDEC JESD238B specification defines CWL as a function of the operating speed grade.

Speed Grade	Frequency	CWL (cycles)	CWL (ns)
HBM3-3200	1600 MHz	28	17.5 ns
HBM3-4000	2000 MHz	36	18.0 ns
HBM3-5200	2600 MHz	46	17.7 ns
HBM3-6400	3200 MHz	56	17.5 ns

Physically, CWL accounts for three propagation delays that must be resolved before write data is useful:

ODT settling time — On-Die Termination resistors inside the DRAM must be switched on before data arrives to prevent reflections on the DQ lines. This switchover requires several nanoseconds.
DQS preamble — The controller must assert DQS (the differential write strobe) at least one tCK before the first data beat. The DRAM uses DQS rising/falling edges to latch each nibble.
CA-to-DQ bus propagation skew — The CA bus and DQ bus travel through different silicon paths and package traces. CWL compensates for the relative delay so both signals arrive at their intended timing relationship inside the DRAM die.

At 2 GHz the controller pipeline is extremely tight. A single cycle error in CWL alignment corrupts the entire burst. The CWL pipeline shift register is the most timing-critical logic in the write path.

Write Leveling

HBM3 includes a write leveling training procedure (performed at boot time) that fine-tunes the DQS launch time relative to the CA command. The result is a per-byte-lane calibration offset stored in the PHY. This module accepts i_cwl[7:0] as a runtime parameter so firmware can update the effective CWL after leveling calibration without recompilation.

3. Write Data Buffer (WDB)

The Write Data Buffer is a small synchronous FIFO that decouples the host write bus from the DQ bus output. Without a WDB, the host would need to hold write data valid for exactly CWL cycles after issuing each write command — an impractical constraint on a high-throughput NoC or AXI interconnect.

Sizing the WDB

Each FIFO entry holds one complete BL4 write transaction: 128 data bits + 16 mask bits = 144 bits per entry. The depth must cover the maximum number of in-flight write commands between the WDB push and the CWL pipeline pop. With a CWL of 36 and a typical memory bus pipeline of 4–8 cycles, 8 entries provides comfortable headroom for pipelined write traffic without consuming excessive SRAM.

Entry width: 144 bits (128 data + 16 mask)
Depth: 8 entries (configurable via parameter)
Push: host asserts i_wr_cmd together with i_wr_data and i_wr_mask
Pop: CWL pipeline fires the read pointer exactly CWL cycles after each push
Backpressure: o_wdb_full signals the scheduler to stall new write commands

If o_wdb_full is not respected and the host pushes additional entries, the WDB will overflow and data will be silently dropped. The scheduler must check o_wdb_full before issuing any WRITE command to the DRAM CA bus.

WDB FIFO Implementation

The FIFO uses a classic dual-pointer (wr_ptr / rd_ptr) implementation with a 3-bit pointer and an 8-entry register array. The full condition is when (wr_ptr[2] != rd_ptr[2]) && (wr_ptr[1:0] == rd_ptr[1:0]) — a standard gray-code-safe approach appropriate for single-clock designs. Because the WDB operates entirely in the controller clock domain, no CDC (clock domain crossing) logic is required here.

4. Burst Length 4 — HBM3 Data Packing

HBM3 mandates BL4 (Burst Length 4) per pseudo-channel. This means every WRITE command transfers exactly 4 data beats on the DQ bus. Each beat is 32 bits wide (the DQ bus width of a single HBM3 pseudo-channel). A complete BL4 burst therefore transfers 4 × 32 = 128 bits (16 bytes) per write command.

Why BL4 and not BL8?

DDR5 SDRAM uses BL8 on a 64-bit bus (64 bytes per burst). HBM3 achieves comparable per-burst capacity through different geometry: 16 pseudo-channels per die, each running BL4 on a 32-bit DQ bus. This shorter burst length reduces the minimum access granularity — useful for cache-line-sized accesses in GPU workloads — and allows faster bank interleaving between consecutive bursts on the same pseudo-channel.

Burst Serializer Logic

The burst serializer converts the 128-bit parallel word from the WDB FIFO into four timed 32-bit beats driven on o_dq_out. A 2-bit beat counter (beat_cnt) increments each cycle while the burst is active. The MUX selects the appropriate 32-bit slice:

beat 0 → data[31:0]
beat 1 → data[63:32]
beat 2 → data[95:64]
beat 3 → data[127:96]

The corresponding 4-bit DM slice is selected in parallel: mask[3:0] through mask[15:12] on beats 0–3 respectively. Both outputs are registered for clean timing margin to the PHY.

5. Data Mask (DM) and Data Bus Inversion (DBI)

Data Mask (DM)

HBM3 supports per-byte masking on writes. The 32-bit DQ bus carries 4 bytes per beat, and each byte has a corresponding DM (Data Mask) bit. When DM is asserted high for a byte, the DRAM ignores that byte's data during the write — the stored value in that byte lane is unchanged. This is critical for non-aligned stores, cache partial updates, and ECC sub-word writes.

The 16-bit i_wr_mask input encodes DM bits for all 4 beats of the burst: mask[3:0] for beat 0 through mask[15:12] for beat 3. The serializer extracts the relevant 4 bits per beat and drives them on o_dm_out[3:0] in lockstep with the data.

DM and data must be perfectly aligned — the DRAM latches both on the same DQS edge. Any skew between o_dq_out and o_dm_out causes incorrect masking. Both are registered in the same always block to guarantee identical timing.

Data Bus Inversion (DBI) — Optional

DBI is an optional HBM3 mode (enabled by MRS register) that reduces SSO (Simultaneous Switching Output) noise and memory bus power. The principle: for each byte, if more than 4 bits would transition 0→1 in the next beat, invert the entire byte instead and assert the associated DBI_n pin low to signal the inversion to the DRAM receiver.

The power saving is significant. In a worst-case all-zeros to all-ones transition on a 32-bit bus at 2 GHz, all 32 pins switch simultaneously — the resulting ground bounce and VCC droop can corrupt adjacent signals. DBI limits worst-case transitions to 16 per beat (50% of the bus), reducing simultaneous switching noise by up to half.

The implementation in this module is straightforward: for each 8-bit byte slice, count set bits. If the count exceeds 4, invert the byte and assert the DBI bit. This operation is purely combinational and adds minimal latency before the output register.

6. Write Path Pipeline Diagram

7. Full Verilog Source — hbm3_write_path.v

The complete synthesizable implementation. All <= (non-blocking assignment) and comparison operators are properly encoded in HTML. Copy-paste this directly into your toolchain.

verilog — hbm3_write_path.v

// =============================================================
// hbm3_write_path.v
// HBM3 Write Data Path — Module 5
// Phase 2 of the HBM3 Controller Build series
// EcrioniX · https://ecrionix.org/hbm3-controller/write-path/
// =============================================================
// Parameters
//   WDB_DEPTH  : Write Data Buffer FIFO depth (power of 2)
//   BL         : Burst Length, fixed 4 for HBM3 pseudo-channel
//   DQ_W       : DQ bus width per pseudo-channel (32 bits)
// =============================================================

module hbm3_write_path #(
    parameter WDB_DEPTH = 8,   // must be power of 2
    parameter BL        = 4,   // burst length — HBM3 fixed BL4
    parameter DQ_W      = 32  // DQ bus width per pseudo-channel
) (
    // clock / reset
    input  wire              i_clk,
    input  wire              i_rst_n,

    // write command interface
    input  wire              i_wr_cmd,         // write command valid
    input  wire [127:0]     i_wr_data,        // 4-beat burst data (BL4 x 32b)
    input  wire [15:0]      i_wr_mask,        // byte enables: 4 bytes x 4 beats
    input  wire [7:0]       i_cwl,            // CAS Write Latency (default 36)

    // DQ bus outputs to PHY
    output  reg [31:0]      o_dq_out,         // DQ bus — one beat per cycle
    output  reg [3:0]       o_dm_out,         // DM (data mask) per byte
    output  reg              o_dqs_en,         // DQS strobe enable to PHY
    output  reg              o_wr_valid,       // DQ data is valid on bus
    output  wire             o_wdb_full        // WDB backpressure flag
);

// ─────────────────────────────────────────────
// Local parameters
// ─────────────────────────────────────────────
localparam ENTRY_W   = 128 + 16;           // data + mask per entry = 144b
localparam PTR_W    = $clog2(WDB_DEPTH) + 1; // one extra bit for full detect
localparam DEPTH_W  = $clog2(WDB_DEPTH);

// ─────────────────────────────────────────────
// Write Data Buffer — synchronous FIFO
// ─────────────────────────────────────────────
reg [ENTRY_W-1:0] wdb_mem [0:WDB_DEPTH-1];
reg [PTR_W-1:0]  wdb_wr_ptr;
reg [PTR_W-1:0]  wdb_rd_ptr;

wire wdb_empty = (wdb_wr_ptr == wdb_rd_ptr);
wire wdb_full_w = (wdb_wr_ptr[PTR_W-1] != wdb_rd_ptr[PTR_W-1]) &&
                  (wdb_wr_ptr[DEPTH_W-1:0] == wdb_rd_ptr[DEPTH_W-1:0]);
assign o_wdb_full = wdb_full_w;

// WDB push (on write command, if not full)
always @(posedge i_clk) begin
    if (i_wr_cmd && !wdb_full_w) begin
        wdb_mem[wdb_wr_ptr[DEPTH_W-1:0]] <= {i_wr_data, i_wr_mask};
        wdb_wr_ptr <= wdb_wr_ptr + 1'b1;
    end
end

// WDB pointer reset
always @(posedge i_clk or negedge i_rst_n) begin
    if (!i_rst_n) begin
        wdb_wr_ptr <= 0;
        wdb_rd_ptr <= 0;
    end
end

// ─────────────────────────────────────────────
// CWL Pipeline — shift register to delay pop
// ─────────────────────────────────────────────
reg [63:0] cwl_pipe;           // 64-stage shift register (max CWL)
wire        cwl_fire;           // fires when write data must go to DQ

// Shift new write commands through the CWL pipeline
always @(posedge i_clk or negedge i_rst_n) begin
    if (!i_rst_n)
        cwl_pipe <= 64'b0;
    else
        cwl_pipe <= {cwl_pipe[62:0], (i_wr_cmd && !wdb_full_w)};
end

// Tap the pipeline at position i_cwl-1 (0-indexed)
assign cwl_fire = cwl_pipe[i_cwl - 8'd1];

// ─────────────────────────────────────────────
// Burst Serializer
// ─────────────────────────────────────────────
reg [127:0] burst_data;
reg [15:0]  burst_mask;
reg [1:0]   beat_cnt;         // 0..3
reg          burst_active;

always @(posedge i_clk or negedge i_rst_n) begin
    if (!i_rst_n) begin
        burst_data   <= 128'b0;
        burst_mask   <= 16'b0;
        beat_cnt     <= 2'b0;
        burst_active <= 1'b0;
        wdb_rd_ptr   <= 0;
        o_dq_out     <= 32'b0;
        o_dm_out     <= 4'b0;
        o_dqs_en     <= 1'b0;
        o_wr_valid   <= 1'b0;
    end else begin
        // Default: deassert outputs
        o_wr_valid <= 1'b0;
        o_dqs_en   <= 1'b0;

        if (burst_active) begin
            // Drive DQS one cycle before first data beat (preamble)
            o_dqs_en   <= 1'b1;
            o_wr_valid <= 1'b1;

            // MUX: select 32-bit slice for current beat
            case (beat_cnt)
                2'd0: begin o_dq_out <= burst_data[31:0];   o_dm_out <= burst_mask[3:0];   end
                2'd1: begin o_dq_out <= burst_data[63:32];  o_dm_out <= burst_mask[7:4];   end
                2'd2: begin o_dq_out <= burst_data[95:64];  o_dm_out <= burst_mask[11:8];  end
                2'd3: begin o_dq_out <= burst_data[127:96]; o_dm_out <= burst_mask[15:12]; end
            endcase

            if (beat_cnt == 2'd3) begin
                burst_active <= 1'b0;
                beat_cnt     <= 2'b0;
                o_dqs_en     <= 1'b0; // postamble: deassert DQS
            end else
                beat_cnt <= beat_cnt + 1'b1;

        end else if (cwl_fire && !wdb_empty) begin
            // CWL expired — load burst data from WDB and begin serialization
            {burst_data, burst_mask} <= wdb_mem[wdb_rd_ptr[DEPTH_W-1:0]];
            wdb_rd_ptr   <= wdb_rd_ptr + 1'b1;
            burst_active <= 1'b1;
            beat_cnt     <= 2'b0;
            o_dqs_en     <= 1'b1; // preamble: assert DQS one cycle early
        end else begin
            o_dq_out <= 32'b0;
            o_dm_out <= 4'b0;
        end
    end
end

endmodule // hbm3_write_path

8. SystemVerilog Testbench with SVA Assertions

The testbench verifies CWL alignment, burst integrity, DM accuracy, and WDB backpressure. SVA concurrent assertions fire on every cycle and report violations automatically.

systemverilog — tb_hbm3_write_path.sv

// ===========================================================
// tb_hbm3_write_path.sv  — Self-checking SV testbench
// ===========================================================
module tb_hbm3_write_path;

parameter CLK_PERIOD = 500; // 500 ps = 2 GHz
parameter CWL        = 36;

logic          i_clk      = 0;
logic          i_rst_n    = 0;
logic          i_wr_cmd   = 0;
logic [127:0]  i_wr_data  = 0;
logic [15:0]  i_wr_mask  = 0;
logic [7:0]   i_cwl      = CWL;

wire  [31:0]  o_dq_out;
wire  [3:0]   o_dm_out;
wire          o_dqs_en;
wire          o_wr_valid;
wire          o_wdb_full;

hbm3_write_path #(.WDB_DEPTH(8), .BL(4), .DQ_W(32)) dut (.*);

always #(CLK_PERIOD/2) i_clk = !i_clk;

// ── SVA: DQS must be high during o_wr_valid ──────────────
property p_dqs_during_valid;
    @(posedge i_clk) o_wr_valid |-> o_dqs_en;
endproperty
assert property (p_dqs_during_valid)
    else $error("SVA FAIL: o_dqs_en deasserted during o_wr_valid");

// ── SVA: wr_valid must deassert after exactly BL=4 beats ─
property p_burst_length;
    @(posedge i_clk) $rose(o_wr_valid) |-> ##4 !o_wr_valid;
endproperty
assert property (p_burst_length)
    else $error("SVA FAIL: burst length != 4 beats");

// ── SVA: WDB must not overflow ────────────────────────────
property p_no_overflow;
    @(posedge i_clk) !(i_wr_cmd && o_wdb_full);
endproperty
assert property (p_no_overflow)
    else $error("SVA FAIL: write command issued when WDB full");

// ── SVA: o_dq_out must be 0 when not valid ───────────────
property p_dq_idle;
    @(posedge i_clk) !o_wr_valid |-> (o_dq_out == 32'b0);
endproperty
assert property (p_dq_idle)
    else $error("SVA FAIL: o_dq_out non-zero outside burst");

// ── Test sequence ─────────────────────────────────────────
int cycle_count;

task automatic issue_write(
    input logic [127:0] data,
    input logic [15:0]  mask
);
    @(posedge i_clk);
    i_wr_cmd  = 1;
    i_wr_data = data;
    i_wr_mask = mask;
    @(posedge i_clk);
    i_wr_cmd  = 0;
endtask

initial begin
    $dumpfile("hbm3_write_path.vcd");
    $dumpvars(0, tb_hbm3_write_path);

    // Release reset after 5 cycles
    repeat (5) @(posedge i_clk);
    i_rst_n = 1;
    repeat (3) @(posedge i_clk);

    // Test 1: Single write, no mask
    $display("[T1] Single write — no mask");
    issue_write(128'hDEAD_BEEF_CAFE_BABE_1234_5678_9ABC_DEF0, 16'h0000);
    repeat (CWL + 6) @(posedge i_clk);

    // Test 2: Write with partial mask (mask upper 2 bytes of beat 0)
    $display("[T2] Partial mask — beats 0 upper bytes masked");
    issue_write(128'hAAAA_BBBB_CCCC_DDDD_EEEE_FFFF_1111_2222, 16'h000C);
    repeat (CWL + 6) @(posedge i_clk);

    // Test 3: Back-to-back writes (pipelined)
    $display("[T3] Back-to-back pipelined writes");
    issue_write(128'h1111_2222_3333_4444_5555_6666_7777_8888, 16'h0000);
    issue_write(128'hABCD_EF01_2345_6789_ABCD_EF01_2345_6789, 16'h0000);
    repeat (CWL + 10) @(posedge i_clk);

    $display("[PASS] All SVA assertions passed");
    $finish;
end

initial begin
    cycle_count = 0;
    forever @(posedge i_clk) cycle_count++;
end

endmodule

9. Timing Parameter Reference

Parameter	Symbol	Value (2 GHz)	Cycles	Description
CAS Write Latency	CWL	18 ns	36	Command to first DQ data valid
Burst Length	BL	4 beats	4	Fixed per HBM3 pseudo-channel
Write Preamble	tWPRE	0.5 ns	1	DQS low before first beat
Write Postamble	tWPST	0.5 ns	1	DQS low after last beat
DQ setup time	tDS	85 ps	—	DQ valid before DQS edge (PHY)
DQ hold time	tDH	85 ps	—	DQ valid after DQS edge (PHY)
ODT turn-on latency	ODTLon	CWL - 2	34	ODT must be on before data
Refresh interval	tREFI	3.9 µs	7800	Average time between refresh cmds

10. Port Reference — hbm3_write_path

Port	Dir	Width	Description
i_clk	Input	1	Controller clock (2 GHz nominal)
i_rst_n	Input	1	Active-low synchronous reset
i_wr_cmd	Input	1	Write command pulse from scheduler (1 cycle)
i_wr_data	Input	128	Full BL4 burst data, presented with i_wr_cmd
i_wr_mask	Input	16	Byte enables for all 4 beats (4 bytes/beat × 4 beats)
i_cwl	Input	8	CAS Write Latency in cycles (typically 36)
o_dq_out	Output	32	DQ bus output, one 32-bit beat per clock
o_dm_out	Output	4	Data mask for current beat (1 bit per byte)
o_dqs_en	Output	1	DQS strobe enable to PHY tri-state driver
o_wr_valid	Output	1	Indicates DQ data and DM are valid this cycle
o_wdb_full	Output	1	WDB FIFO full — scheduler must stall new writes

11. FAQ

What is CWL in HBM3 and why is it 36 cycles?

CWL (CAS Write Latency) is the number of clock cycles from a WRITE command until the controller must put the first data beat on the DQ bus. At 2 GHz (500 ps/cycle), CWL=36 equals 18 ns. This accounts for ODT switching time, DQS preamble setup, and CA-to-DQ propagation skew across the HBM3 silicon and package. Higher speed grades have larger CWL values in cycles but similar absolute time in nanoseconds.

Why does HBM3 use Burst Length 4 instead of BL8 like DDR5?

HBM3 achieves its massive bandwidth through parallel pseudo-channels (16 per die stack), each with a 32-bit DQ bus and BL4 bursts. A full burst delivers 128 bits per pseudo-channel. DDR5 uses BL8 on a 64-bit bus (512 bits per burst) but runs only 1–2 channels per DIMM. HBM3's shorter bursts mean lower minimum access granularity — important for GPU caches — and faster bank-to-bank interleaving.

What happens if the WDB overflows?

If the host issues a write command when o_wdb_full is high and the WDB push is blocked, the data is silently discarded. The WRITE command has already been sent to DRAM on the CA bus, so DRAM will capture garbage data at CWL cycles later. This is a fatal protocol error. The scheduler must check o_wdb_full before asserting i_wr_cmd. The SVA assertion p_no_overflow in the testbench catches this condition in simulation.

How does DBI reduce power and should it always be enabled?

DBI (Data Bus Inversion) inverts any data byte where more than 4 bits would switch 0→1, limiting worst-case simultaneous transitions to 4 per byte (50% of the bus). This reduces SSO noise and average switching power by up to ~50% on the write bus. The cost is one extra DBI_n pin per byte lane and slightly higher logic overhead. For power-constrained systems (HPC GPU, AI accelerators) DBI is typically enabled. For latency-critical low-power systems where the bus is not fully loaded it may be skipped.

Can I use a larger WDB depth, and what is the cost?

Yes. WDB_DEPTH is a parameter. Larger depths allow more in-flight writes before backpressure, which improves throughput in bursty workloads. The cost is SRAM area: each entry is 144 bits, so 8 entries = 1152 bits (~144 bytes). A depth of 16 doubles SRAM to 288 bytes — negligible on modern dies. For very deep pipelines (e.g. multi-hop NoC) consider 16 or 32 entries. The pointer width and full-detect logic widen automatically via $clog2.

← Module 4 — Page Policy Module 6 — Read Data Path →