What Is an AI Chip? How AI Accelerators Really Work

An AI chip (or AI accelerator) is a processor specialised for the mathematics of neural networks. And that math, stripped of mystique, is overwhelmingly one operation: multiply-accumulate — acc += a × b — performed billions of times, arranged as matrix multiplications.

A CPU has a few powerful, general cores optimised for branchy, sequential logic. A neural network instead wants the same simple arithmetic on enormous, uniform data. So an AI chip throws out the generality and packs in thousands of small multiply-accumulate (MAC) units, surrounds them with wide on-chip memory, and feeds them from high-bandwidth external memory. GPUs, TPUs, NPUs and custom ASICs are all points on this same idea.

A neural-network layer computes, in essence, Y = activation(W · X + b) — a matrix of weights W multiplied by a vector/matrix of inputs X. Multiplying matrices means, for every output element, a dot product: a long sequence of multiply-then-add. This dense matrix-multiply is called GEMM (General Matrix Multiply), and modern transformers are >90% GEMM.

One multiply-accumulate is trivial. The catch is scale: a single large language model does trillions of MACs per token. The entire job of an AI chip is to perform that ocean of MACs with maximum throughput per watt.

Want to feel why parallel hardware crushes this? Play the interactive GPU Lab — the CPU-vs-GPU race is the same principle behind every AI chip.

The atom of an AI chip is the MAC unit: a multiplier feeding an adder that accumulates a running sum. Put thousands of them on a die and you can do thousands of multiply-accumulates every clock cycle.

But raw MACs aren't enough — you have to feed them, and memory bandwidth is the enemy. The elegant answer is the systolic array: a 2-D grid of MAC units (processing elements, PEs) through which data flows rhythmically, like a heartbeat (systole). Each PE multiplies, accumulates, and passes operands to its neighbours. Crucially, each value entering the array is reused by an entire row or column of PEs instead of being re-fetched — slashing memory traffic. Google's TPU is built around a large systolic array.

▶ See it run: the interactive Systolic Array Lab lets you edit two matrices and step the array cycle by cycle — watch the data flow and the accumulators build up.

"AI chip" is an umbrella. The members trade flexibility against efficiency:

They all accelerate the same MACs. A GPU keeps programmability (and so dominates fast-moving research/training); a TPU/ASIC hard-wires the dataflow for efficiency; an NPU shrinks it for the edge. See the GPU side hands-on →

Here's the secret most explainers skip: AI chips are usually limited by memory, not compute. A MAC array can demand operands faster than DRAM can deliver them — the famous memory wall. Moving a number from external memory can cost orders of magnitude more energy than the multiply itself. So AI-chip design is largely a memory-movement problem.

Two design levers fight the wall:

• HBM (High-Bandwidth Memory) — DRAM dies stacked with through-silicon vias next to the compute die on a silicon interposer, delivering terabytes per second. (See why HBM demand exploded.)
• On-chip SRAM + data reuse — keep data on-chip and reuse it many times (the systolic array's whole point) so you touch HBM as little as possible.

Neural networks are remarkably tolerant of low numerical precision. That's a gift: smaller numbers mean less memory, less bandwidth, and far more MACs per mm² and per watt. AI chips support a ladder of formats:

Quantization is the process of converting a trained FP model to a lower-precision one (e.g. INT8) with minimal accuracy loss. Halving the bit-width roughly doubles effective throughput and memory capacity — which is why FP8 and INT8 are central to today's AI hardware. A chip's headline TOPS number is always quoted at a precision (e.g. "X TOPS INT8").

Because memory movement dominates energy, the dataflow — the order in which data is loaded, reused and stored — is a first-class architectural decision. Classic strategies keep one operand "stationary" in the PEs to maximise reuse:

• Weight-stationary — keep weights in the PEs, stream activations through (common in systolic arrays/TPUs).
• Output-stationary — keep the accumulating output in place, stream inputs & weights.
• Row-stationary — balance reuse of weights, inputs and partial sums (e.g. the Eyeriss research design).

The goal is always the same: fetch each value from expensive memory once, then reuse it as many times as possible close to the MACs.

Many neural-network weights and activations are zero (or can be pruned to zero). Multiplying by zero is wasted work — so modern AI chips exploit sparsity: hardware that skips zero operands, or supports structured sparsity (e.g. 2-of-4 patterns) to roughly double effective throughput. Combined with quantization, sparsity is a major lever for inference efficiency.

• TOPS / TFLOPS — Tera (trillion) Operations / Floating-point Ops Per Second: raw throughput, quoted at a precision.
• TOPS/W — efficiency: operations per watt. The metric that actually matters at data-centre scale and on the edge.
• Utilisation — what fraction of the MACs are actually busy. Real workloads often hit a fraction of peak TOPS because of memory stalls.

The roofline model ties it together: performance is capped either by compute (the flat "roof") or by memory bandwidth (the sloped part), depending on a workload's arithmetic intensity (operations per byte fetched). Low-intensity workloads (like small-batch LLM inference) are memory-bound — which is exactly why HBM and on-chip SRAM matter so much. Peak TOPS alone is marketing; sustained TOPS at high utilisation is engineering.

Training is a brutal, multi-week, multi-thousand-accelerator effort dominated by compute and interconnect. Inference is about serving predictions cheaply and quickly — and is increasingly pushed to the edge (your phone's NPU) for latency and privacy.

A single die has reached the reticle limit (the max size a lithography tool can print). So AI chips scale outward:

• Chiplets & advanced packaging — multiple dies (compute + HBM stacks) joined on a silicon interposer (2.5D) or stacked (3D). Packaging throughput is now a key constraint on how many accelerators ship.
• Chip-to-chip interconnect — high-speed links (e.g. NVLink-class fabrics) let thousands of accelerators act as one giant machine for training.
• Networking & power — at cluster scale, the bottleneck shifts again to network bandwidth, power delivery and cooling.

This is why the AI boom is a whole-supply-chain event — logic, memory, packaging and networking all at once. More on that here →

• In-memory / near-memory computing — do the MACs inside the memory array to kill data movement (the biggest energy cost). Deep dive: memory alternatives for AI →
• Analog & mixed-signal compute — perform multiply-accumulate with physics (charge/current) for huge efficiency, trading exactness.
• Photonic computing — matrix multiply with light; near-zero interconnect energy, still maturing. How photonics works →
• Wafer-scale & 3D stacking — ever-bigger integrated compute with memory stacked directly on logic.
• Sparsity & sub-8-bit formats — squeezing more useful work per joule.

Related: How a Quantum Chip Works · GPU Lab (interactive) · Why AI Needs So Many Chips · Transistor Evolution · VLSI Hub