What Is Google's TPU?

If you’ve spent any time around AI news lately, you’ve probably seen the term “TPU” thrown around alongside GPU, Nvidia, and words like “exaflops” and “HBM bandwidth.” It can feel like a wall of jargon. So let’s strip all of that away and build this up from scratch — starting with the simplest possible explanation, then layering in the real engineering behind Google’s newest chips, TPU 8t and TPU 8i.

The Five-Year-Old Explanation

Imagine you’re building a giant tower out of LEGO bricks, and you have two kinds of helpers:

Helper A (the GPU) is a helper who’s really good at doing lots of different little tasks — snapping bricks together, sorting colors, painting pieces. They’re flexible and can help with almost any job you give them.
Helper B (the TPU) is a helper who was born and trained to do exactly one thing incredibly fast: multiplying and adding numbers together, over and over, in huge batches. That’s it. That’s their whole superpower. But because that’s all they do, they can do it faster and with less wasted energy than Helper A.

AI models — like the one answering your questions right now — are, underneath all the complexity, mostly just enormous piles of multiplication and addition happening billions of times per second. So Google built Helper B (the TPU) specifically for that one job, instead of using a general-purpose helper. And now, for the first time, Google has split Helper B into two specialized helpers — one built to teach the AI (training) and one built to let the AI talk back to you instantly (inference). That’s TPU 8t and TPU 8i.

That’s the whole idea. Everything below is just the “how” and “why” in more detail.

First, What Even Is a TPU?

TPU stands for Tensor Processing Unit. A “tensor” is just a fancy math term for a grid of numbers (think of a spreadsheet, but it can have more than two dimensions). Neural networks — the technology behind ChatGPT, Gemini, Claude, and basically all modern AI — work by multiplying huge grids of numbers together, over and over, to transform an input (like your question) into an output (like an answer).

A regular computer chip (a CPU) is built to do lots of different jobs one after another, very precisely, in order. A TPU is built to do one specific job — multiplying and adding huge grids of numbers — at massive scale, all at once, using specialized circuits called systolic arrays that are wired specifically for this kind of math. Google has been building these since 2015, originally just for internal use, and opened them up to outside developers and cloud customers starting in 2018.

Okay, But What’s Wrong With GPUs? Why Not Just Use Those?

This is the real question, and it deserves a real answer.

GPUs (Graphics Processing Units) were originally built for video games — rendering millions of pixels on screen at once. It turns out that job (lots of simple math done in parallel) is mathematically similar to what neural networks need, which is why Nvidia’s GPUs became the default hardware for AI training over the past decade. Nvidia GPUs are genuinely excellent at this, and they remain dominant — they still control the large majority of the AI chip market today.

But GPUs are still general-purpose parallel processors. They were designed to handle graphics rendering, scientific simulation, video decoding, and AI workloads all with the same underlying architecture. That flexibility is a strength, but it’s also a cost — some of the chip’s real estate and power budget goes toward capabilities that a pure AI workload doesn’t need.

A TPU throws that flexibility away on purpose. Google designs TPUs only for the matrix-multiplication-heavy math that neural networks need, and co-designs the chip, the cooling, the networking between chips, and the software stack all together, as one system, instead of buying commodity hardware and bolting software on top. This “co-design everything together” approach is the philosophy behind TPUs, and it’s what lets Google claim dramatically better efficiency — more useful AI computation per dollar and per watt of electricity — for the specific workloads it cares about.

In short: GPUs are a very good generalist. TPUs are a specialist built for exactly one job, at Google’s exact scale, tuned to Google’s exact software.

It’s a bit like the difference between a Swiss Army knife and a scalpel. The Swiss Army knife (GPU) can do a hundred things reasonably well and is what most of the world reaches for. The scalpel (TPU) can only really do one thing — but for that one thing, in the hands of someone who owns the whole operating room (Google owns the chip, the data center, the software, and the AI models running on it), it can be faster, cheaper to run, and more precise.

The Big New Idea: Splitting the Chip in Two

For nearly a decade, Google shipped one TPU generation at a time, and that single chip design had to handle both halves of the AI lifecycle: training (teaching a model by showing it huge amounts of data) and inference (actually using the trained model to answer questions in real time). With its eighth generation, announced in April 2026, Google did something it had never done before: it split the TPU into two distinct, purpose-built chips.

TPU 8t — built for training
TPU 8i — built for inference

Why split them now? Because training and inference are, in an important sense, opposite problems.

Training Is a Throughput Problem

Training a frontier AI model means feeding it trillions of examples and adjusting billions (or trillions) of internal numbers slowly, over weeks or months, until the model gets good at predicting the next word, pixel, or action. This job cares mostly about throughput — how much total computation you can chew through, as efficiently as possible, across tens of thousands of chips working together. A training run doesn’t necessarily need to respond to any single request quickly; it needs to grind through an enormous, sustained workload without wasting a single cycle. It’s like a factory assembly line optimized to produce the maximum number of cars per day.

Inference Is a Latency Problem

Inference is the opposite: it’s what happens the instant you type a message and hit send. Nobody wants to wait ten seconds for an AI agent to “think.” And with the rise of AI agents — systems where multiple AI models talk to each other, plan multi-step tasks, and reason step-by-step through a chain of thought before answering — even tiny delays compound fast. If ten AI agents are passing information back and forth to solve a task, and each hop adds a small delay, the whole system starts to feel sluggish. Inference cares about latency — how fast a single response comes back — not just total throughput. It’s like a restaurant kitchen optimized to get your one order out fast, not to cook the most meals per day.

Trying to build one chip that’s equally great at both a “produce the most over time” job and a “respond to me instantly” job means compromising on both. So Google decided to stop compromising and build two chips instead.

What Makes TPU 8t (Training) Special

TPU 8t is the “compute-intensive” chip, and a few of its design choices explain why:

SparseCore — a dedicated piece of the chip that handles the messy, irregular parts of training (like looking up embeddings) separately from the clean, repetitive matrix math, so the main compute engine never sits idle waiting on that irregular work.
Native FP4 — this refers to using a lower-precision number format for calculations. Think of it like doing math with fewer decimal places when you don’t need perfect precision — it’s faster and uses less memory bandwidth, roughly doubling throughput compared to the older format, while special techniques keep accuracy high enough to still be trustworthy.
Virgo Network — a new, flatter way of wiring thousands of chips together so they can talk to each other with fewer “hops,” which matters enormously when tens of thousands of chips need to coordinate during a single training run.
TPUDirect RDMA and TPU Direct Storage — these let the chip pull data directly from storage and network cards without routing everything through a slower, traditional CPU-and-memory bottleneck, dramatically speeding up how fast the chip gets fed the data it needs to chew through.

Put simply: TPU 8t is built to keep a gigantic factory floor of chips running at near-100% productivity for weeks at a time, without wasting compute on inefficiencies, and to shrink model training cycles from months down to weeks.

What Makes TPU 8i (Inference) Special

TPU 8i takes a very different approach, because its job — responding fast, over and over, to real users and AI agents — has different bottlenecks:

Large on-chip SRAM (3x more than before) — SRAM is a small but extremely fast type of memory sitting right on the chip. More of it means the chip can keep more of the model’s “working memory” (called the KV cache, which stores context from the ongoing conversation) close at hand instead of constantly fetching it from slower memory — reducing idle time between responses.
Collectives Acceleration Engine (CAE) — a new dedicated piece of hardware that speeds up the “synchronization” steps that happen when a model is generating a step-by-step chain of thought or coordinating across chips — cutting that specific bottleneck’s latency dramatically.
Boardfly network topology — instead of the older “3D torus” wiring pattern (which is efficient for training but adds unnecessary delay for quick request/response patterns), Boardfly connects chips in a flatter, more directly connected way, cutting the number of network hops a message has to take by more than half.

Put simply: TPU 8i is built so that when you talk to an AI agent — or when a swarm of AI agents are talking to each other — the responses feel instant, even under massive concurrent load.

The Parts They Still Share

Even though the two chips are specialized, Google didn’t rebuild everything from scratch. Both TPU 8t and TPU 8i share a common foundation, which keeps things manageable for the engineers building on top of them:

Axion, Google’s own ARM-based CPU host — both chips run on the same custom host processor instead of relying on traditional x86 chips, removing a potential bottleneck at the “front door” of the system.
4th-generation liquid cooling — chips this powerful generate enormous heat, and both use the same advanced liquid cooling design to stay stable.
The same software stack — developers writing code in JAX, PyTorch, or Keras don’t need to write different code for each chip. A compiler layer called XLA automatically handles translating the same code to run efficiently on either chip.

This shared foundation is a deliberate trade-off: Google gets the efficiency benefits of specialized silicon without doubling the cost and complexity of maintaining two totally separate ecosystems for developers.

So, What Problem Does This Actually Solve?

Here’s the heart of it, in plain language:

The old way — one general chip generation trying to handle both training and inference — was a compromise. As AI shifted from “answer one question” chatbots toward complex, multi-step AI agents that reason, plan, retry, and coordinate with other agents in real time, that compromise started costing real money and real speed. Inference now makes up the majority of total AI computing demand, not training — a huge shift from just a few years ago when training dominated. A chip tuned for the old priorities was increasingly mismatched with how AI is actually being used today.

The new way — two specialized chips — lets Google (and the customers who rent this hardware through Google Cloud) get dramatically better results for each half of the job: faster, cheaper training runs, and snappier, cheaper real-time responses, without either job dragging the other down.

As for TPUs versus GPUs specifically: GPUs aren’t going anywhere, and Google isn’t claiming TPUs replace them outright — Google even offers Nvidia GPU instances alongside its own TPUs in the cloud. But for Google’s own AI models, and for customers who commit to Google’s software ecosystem, TPUs offer a path to significantly better cost-per-task and power-per-task by trading flexibility for specialization. It’s the classic engineering trade-off: general-purpose hardware is easier to adopt and more versatile, but purpose-built hardware, when it fits your exact use case, is hard to beat on raw efficiency.

Why This Matters Beyond the Tech Specs

A few bigger-picture reasons this launch matters, even if you never touch a TPU yourself:

It’s a bet on how AI will be used going forward. By building a chip specifically optimized for agentic, chain-of-thought, multi-step reasoning workloads, Google is betting that this — not simple one-shot chatbot replies — is where AI usage is heading.
It’s a challenge to Nvidia’s dominance. Nvidia currently controls the large majority of the AI accelerator market. By building increasingly specialized, increasingly capable custom silicon — and now recruiting a second major chip design partner alongside its longtime partner Broadcom — Google is signaling it wants to be a credible alternative, not just for itself, but for the cloud customers who rent its infrastructure.
Efficiency is becoming the real competitive battleground. As AI models get more expensive to train and serve at scale, the winners increasingly won’t just be whoever has the most raw compute — they’ll be whoever can deliver the most useful AI work per dollar and per watt of electricity. Specialized silicon, tightly co-designed with the software and models running on it, is one of the clearest ways to win that efficiency race.

Summary

Google split its AI chip into two specialists — one built to teach AI models as efficiently as possible, and one built to let AI models respond and reason instantly — because trying to make one general-purpose chip do both jobs was leaving performance and money on the table, and because the rise of fast, real-time AI agents demanded hardware built specifically for that world.

Cheers,

Sim