Max/average/global pooling hardware, batch normalization folding into conv weights, and the fused BN+ReLU+Pool pipeline that runs the whole stage at one pixel per clock.
After convolution and activation, two more operations shape the feature map: pooling (which shrinks spatial dimensions) and normalization (which stabilizes the data distribution). Both are cheap in hardware compared to convolution — and both are usually fused into the conv pipeline so they cost nothing extra in memory traffic.
Max pooling takes the maximum value in each window (typically 2×2, stride 2), halving the width and height. It keeps the strongest activation and discards the rest — adding translation invariance. In hardware it's just comparators, no multipliers.
// maxpool2x2.v — Streaming 2×2 max pooling (stride 2)
module maxpool2x2 #(parameter IMG_W = 224, parameter DW = 8)(
input wire clk, rst_n,
input wire pix_valid,
input wire signed [DW-1:0] pix_in,
output reg signed [DW-1:0] pool_out,
output reg out_valid
);
reg signed [DW-1:0] lb [0:IMG_W-1]; // 1 line buffer (prev row)
reg signed [DW-1:0] prev_col; // previous pixel this row
reg [$clog2(IMG_W):0] col;
reg row_odd; // toggles each row
// 3-comparator max tree for the 2×2 window
function signed [DW-1:0] max2(input signed [DW-1:0] a, b);
max2 = (a > b) ? a : b;
endfunction
always @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
col <= 0; row_odd <= 0; out_valid <= 0;
end else if (pix_valid) begin
// store this pixel into the line buffer for next row's pairing
lb[col] <= pix_in;
prev_col <= pix_in;
// produce one output every 2 cols on odd rows (2×2 complete)
if (row_odd && col[0]) begin
// window = {lb[col-1], lb[col], prev_col, pix_in}
pool_out <= max2( max2(lb[col-1], lb[col]),
max2(prev_col, pix_in) );
out_valid <= 1;
end else
out_valid <= 0;
if (col == IMG_W-1) begin col <= 0; row_odd <= ~row_odd; end
else col <= col + 1;
end
end
endmoduleAverage pooling outputs the mean of each window. Slightly more hardware than max pooling — an adder tree plus a divide — but the divide by a power of 2 (e.g. /4 for 2×2) is just a right shift, so it's still multiplier-free.
GAP collapses each entire channel into a single number by averaging all its pixels. Modern CNNs (ResNet, MobileNet, EfficientNet) use GAP instead of huge fully-connected layers — slashing parameters and overfitting.
Batch normalization stabilizes training by normalizing each channel. At inference time, all four BN parameters (mean μ, variance σ², scale γ, shift β) are frozen constants — which means the entire BN operation is a per-channel affine transform: y = γ·(x−μ)/√(σ²+ε) + β.
Because it's affine and the previous layer (convolution) is also affine, the two can be merged — this is BN folding.
BN folding is valid because the BN statistics are frozen after training. During training, μ and σ² change every batch, so BN must stay separate. This is purely an inference-time optimization — and it's why you almost never see a standalone BatchNorm block in a deployed FPGA accelerator.
The whole post-conv stage collapses into a single streaming pipeline. BN is already folded into conv, so the engine only needs requantize → ReLU → pool, all in registered stages with no memory round-trips.
// fused_relu_maxpool.v — requantize → ReLU → 2×2 max pool (BN pre-folded)
module fused_relu_maxpool #(parameter ACC_W=32, parameter DW=8, parameter IMG_W=224)(
input wire clk, rst_n,
input wire valid_in,
input wire signed [ACC_W-1:0] acc_in, // conv accumulator (BN already folded in)
input wire [7:0] req_shift, // requantize shift
output reg signed [DW-1:0] y,
output reg valid_out
);
// Stage 1: requantize + ReLU + saturate to INT8
reg signed [DW-1:0] act;
reg act_valid;
always @(posedge clk) begin
if (!rst_n) begin act <= 0; act_valid <= 0; end
else if (valid_in) begin
reg signed [ACC_W-1:0] s;
s = acc_in >>> req_shift; // requantize
s = s[ACC_W-1] ? 0 : s; // ReLU
act <= (s > 127) ? 8'sd127 : s[DW-1:0]; // saturate
act_valid <= 1;
end else act_valid <= 0;
end
// Stage 2: feed activated stream into the 2×2 max-pool engine
maxpool2x2 #(.IMG_W(IMG_W),.DW(DW)) u_pool (
.clk(clk), .rst_n(rst_n),
.pix_valid(act_valid), .pix_in(act),
.pool_out(y), .out_valid(valid_out)
);
endmodule| Operation | Hardware | Multipliers | Throughput |
|---|---|---|---|
| Max pool 2×2 | 3 comparators + 1 line buffer | 0 | 1 out / 4 in |
| Avg pool 2×2 | 3 adders + shifter | 0 | 1 out / 4 in |
| Global avg pool | 1 accumulator/channel + reciprocal | 1 (×1/N) | 1 pixel/clock |
| Batch norm (folded) | none — merged into conv | 0 | free |
| Fused ReLU+Pool | shifter + comparators | 0 | 1 pixel/clock |
Convolution (Day 5) does the heavy lifting with thousands of MACs. Everything in this lesson — pooling, normalization, activation — is essentially free by comparison: no multipliers, fused into the stream. This is why CNN accelerators spend ~95% of their DSPs on convolution and almost none on these layers.
Next — Day 9: Pipelining & Parallelism — layer pipelining, inter-layer FIFOs, the throughput-vs-latency trade-off, and data vs model parallelism for the full CNN.