Must-Know AI Infrastructure Concepts

Concepts and original illustrations referenced from a carousel by @vishakha.sadhwani.

If you’ve ever heard people talking about GPUs, NVLink, quantization, or “TTFT” and just nodded along pretending you understood — this post is for you.

AI infrastructure sounds intimidating because it’s full of acronyms, but almost every concept underneath is actually pretty simple once someone breaks it down in plain English.

Think of this post as a tour through the engine room of AI. When you type a prompt into ChatGPT or Claude, that request doesn’t just float in the cloud — it lands on real physical hardware: chips, memory, cables, and cooling systems, all working together in a very specific, very deliberate way. Understanding the pieces helps you understand why AI is expensive, why some companies build “AI superclusters,” and why a chatbot sometimes feels instant and sometimes feels like it’s thinking.

We’ll go through this in four parts:

Part	What it covers
🧩 1. The Building Blocks	The actual hardware chips and memory
🔌 2. How GPUs Talk to Each Other	The wiring and connections
🏗️ 3. Scaling Up	Clusters and parallelism
⚡ 4. Making AI Fast	The tricks used once a model is actually running

Here’s the map of where we’re headed:

flowchart LR
    A[🧩 Building Blocks
    GPU, HBM, CPU, PCIe] --> B[🔌 Connecting GPUs
    NVLink, NVSwitch, InfiniBand]
    B --> C[🏗️ Scaling Up
    Clusters & Parallelism]
    C --> D[⚡ Serving Users Fast
    Quantization, KV Cache, TTFT/TPOT]

Grab a coffee. Let’s go concept by concept. ☕

---

🧩 Part 1: The Building Blocks (Hardware 101)

1️⃣ GPU (Graphics Processing Unit)

Let’s start with the star of the show. A GPU is the main compute engine that runs the math behind AI — both training a model and using it afterward (called “inference”).

Here’s the simplest way to think about it: a CPU is like one incredibly smart person doing one task at a time, very carefully, one after another. A GPU is like a stadium full of thousands of moderately smart people, all doing simple math problems at the same time.

AI is, underneath all the magic, mostly just giant multiplication problems — specifically, multiplying huge grids of numbers together (called matrices). GPUs are built to do exactly that kind of repetitive math in parallel, which makes them dramatically faster than CPUs for AI work, even though a single GPU “core” is much simpler than a CPU core.

flowchart TB
    subgraph CPU["🧑‍💼 CPU — 'One Genius'"]
        direction TB
        c1[Task 1] --> c2[Task 2] --> c3[Task 3]
    end
    subgraph GPU["🏟️ GPU — 'Stadium of Workers'"]
        direction LR
        g1[Task] ~~~ g2[Task] ~~~ g3[Task] ~~~ g4[Task]
        g5[Task] ~~~ g6[Task] ~~~ g7[Task] ~~~ g8[...]
    end

CPU does things one after another. GPU does thousands of simple things at once.

2️⃣ High-Bandwidth Memory (HBM)

If the GPU is the “brain” doing the math, HBM is its short-term memory — the notepad it uses while working.

This memory has to be extremely fast, because a GPU can do so much math per second that if the memory feeding it numbers is slow, the GPU just sits there waiting, wasting all that expensive compute power. That’s the whole point of HBM — it’s memory built and physically stacked right next to the GPU chip so it can feed data in and out at extremely high speed.

💡 Why it matters to you: the size of this memory decides how big of a model, how much data, or how long a conversation (“context”) can actually fit at once. When you hear “this model needs 80GB of VRAM,” that’s HBM they’re talking about. Run out of it, and the model simply can’t load or can’t handle your request.

3️⃣ Host CPU

The CPU doesn’t disappear just because GPUs do the heavy lifting — it’s still the coordinator. Think of the CPU as the project manager and the GPU as the factory floor.

The CPU’s job is to organize the server, prepare and clean up the data, run the general application logic (like handling your web request), and then hand off the actual heavy-duty math to the GPU.

4️⃣ PCIe (Peripheral Component Interconnect Express)

Now that we have a CPU and a GPU, they need to talk to each other. PCIe is essentially the road connecting them.

It matters because it determines how fast data can move into the GPU in the first place. You could have the fastest GPU in the world, but if the road bringing data to it is narrow and slow, you get a traffic jam — the GPU finishes its work and then just waits for the next batch of data to arrive.

flowchart LR
    CPU1[🧑‍💼 CPU] <-->|PCIe road| GPU1[🎮 GPU]
    style CPU1 fill:#f9d976,stroke:#333
    style GPU1 fill:#a8e6a1,stroke:#333

5️⃣ Tensor Cores

Inside modern NVIDIA GPUs, there are specialized mini-units called Tensor Cores. Regular GPU cores are already good at parallel math, but Tensor Cores are built specifically for the exact type of matrix multiplication that AI models rely on constantly.

Think of it like this: a GPU is a factory good at assembly-line work in general, but Tensor Cores are a specific assembly line built only for the one product (matrix math) the factory makes the most of. Because they’re so specialized, they can do that particular job dramatically faster than general-purpose GPU cores could.

6️⃣ CUDA

CUDA is NVIDIA’s software platform — think of it as the instruction manual and toolkit that lets programs (like PyTorch or TensorFlow) actually talk to the GPU hardware and tell it what to do.

flowchart LR
    W[📦 Big Workload] --> S[✂️ CUDA splits it
    into small tasks]
    S --> P1[Core 1]
    S --> P2[Core 2]
    S --> P3[Core 3]
    S --> P4[Core ...]
    P1 & P2 & P3 & P4 --> R[✅ Combined Result]

Without something like CUDA, developers would have to write extremely low-level, painful code to control the GPU directly. CUDA is a big reason NVIDIA became so dominant — everyone builds their AI software on top of it.

7️⃣ GPU Utilization

This one is simpler than it sounds: GPU utilization is just a percentage that tells you how busy the GPU actually is.

💸 Why should you care? GPUs are incredibly expensive — some cost tens of thousands of dollars each — and companies rent time on them by the hour. If your GPU utilization is sitting at 20%, you’re paying for 100% of an expensive chip while only using a fifth of its power. Low utilization is basically money quietly burning.

8️⃣ Memory Bandwidth

We already talked about HBM (the memory itself). Memory bandwidth is about speed — specifically, how fast the GPU can read data from and write data to that memory.

• H2D (Host to Device): data moving from CPU memory to GPU memory
• D2H (Device to Host): data moving back from GPU memory to CPU memory

For a lot of AI workloads, especially ones with huge models, memory bandwidth — not raw compute power — is actually the bottleneck. The GPU could technically calculate faster, but it’s stuck waiting on memory to hand it numbers.

9️⃣ Batch Size

When a model processes information, it doesn’t have to do it one item at a time — it can group multiple inputs together and process them all at once. That group is called a “batch,” and the batch size is how many inputs are bundled together.

Batch Size	Pro	Con
📈 Bigger	Better GPU utilization, higher throughput	More memory needed, slower per-request response
📉 Smaller	Faster individual responses	Wastes GPU parallel power

It’s a classic engineering balancing act: batch too small and you waste GPU power; batch too large and you run out of memory or slow down individual responses.

---

🔌 Part 2: How GPUs Talk to Each Other

A single GPU, no matter how powerful, usually isn’t enough to train or run today’s largest AI models. So companies connect many GPUs together — and how they’re connected turns out to matter enormously.

flowchart TB
    subgraph Server["🖥️ One Server"]
        direction LR
        G1[GPU] <-->|NVLink| G2[GPU]
        G2 <-->|NVLink| G3[GPU]
        G3 <-->|NVLink| G4[GPU]
        G1 <-.NVSwitch fabric.-> G4
    end
    subgraph Cluster["🏢 Whole Cluster"]
        Server -->|InfiniBand| Server2["🖥️ Another Server"]
        Server2 -->|InfiniBand| Server3["🖥️ Another Server"]
    end

🔟 NVLink

NVLink is a high-speed direct connection between GPUs, used when multiple GPUs in the same system need to share data with each other constantly.

Regular PCIe connections are actually too slow for GPU-to-GPU communication during heavy AI training. NVLink is like building a private high-speed highway directly between GPUs, skipping the normal, more congested route.

1️⃣1️⃣ NVSwitch

If NVLink is a direct road between two GPUs, NVSwitch is a switchboard that connects many GPUs together at once, inside the same server.

Picture eight GPUs in one machine. Instead of wiring each GPU directly to every other GPU (which gets messy fast), they all plug into a central NVSwitch, which routes traffic between any of them at very high speed. This is what allows a single server to act like one giant, unified GPU system rather than eight separate ones.

1️⃣2️⃣ InfiniBand

NVSwitch connects GPUs within one server. But what happens when one server’s worth of GPUs isn’t enough, and you need to link together dozens or thousands of servers (called “nodes”)?

That’s where InfiniBand comes in — a super-fast networking technology that connects entire GPU servers to each other, so a “cluster” of many machines can behave like one enormous system. This is the backbone of the giant AI superclusters you hear about.

---

🏗️ Part 3: Scaling Up — Clusters and Parallelism

Once you have many GPUs wired together, you need strategies for actually splitting the AI work across them. This is where things get genuinely clever.

1️⃣3️⃣ GPU Cluster

A GPU cluster is simply a group of GPU servers working together as if they were one giant AI system.

🏢 Think of it like a company: individual employees (GPUs) work in teams (servers), and teams across different offices (nodes) coordinate through a communication network to complete one giant shared project.

1️⃣4️⃣ Distributed Training

This is the “why” behind everything in Part 2 and Part 3: distributed training means training one AI model across many GPUs at the same time, instead of just one.

Why bother?

1. The model is too large to fit on one GPU’s memory → it has to be split up
2. Even if it fits, training would take forever on one GPU → spreading the work out finishes it faster

flowchart TB
    G1[GPU] & G2[GPU] & G3[GPU] & G4[GPU] -->|gradients| R{🔄 Average
    (Reduce)}
    R -->|updated model| B[📢 Broadcast]
    B --> G1b[GPU] & G2b[GPU] & G3b[GPU] & G4b[GPU]

1️⃣5️⃣ Tensor Parallelism

Splits a single layer of the model itself across multiple GPUs. Imagine one giant multiplication problem — instead of one GPU doing it all, you slice the matrix into pieces, hand each piece to a different GPU, and combine the slices into the full correct result.

1️⃣6️⃣ Pipeline Parallelism

Splits the model’s layers across different GPUs, like stages on an assembly line.

flowchart LR
    GPU0["GPU 0
    Layer 0"] -->|forward| GPU1["GPU 1
    Layer 1"] -->|forward| GPU2["GPU 2
    Layer 2"] -->|forward| GPU3["GPU 3
    Layer 3"]
    GPU3 -.backward.-> GPU2 -.backward.-> GPU1 -.backward.-> GPU0

1️⃣7️⃣ Data Parallelism

The easiest one to grasp: copy the entire model onto every GPU, then give each GPU a different slice of the data.

Parallelism Type	What’s split	Best for
🧮 Tensor	A single layer’s math	Layers too big for one GPU
🏭 Pipeline	The model’s layers	Very deep models
📊 Data	The training data	Simple scaling / speed

In practice, large-scale training often combines all three, arranging GPUs into a 3D grid where different “directions” handle different types of splitting. That’s genuinely how the biggest AI models in the world get trained.

1️⃣8️⃣ MIG (Multi-Instance GPU)

MIG flips the conversation around: instead of combining multiple GPUs to act as one, MIG splits one single GPU into several smaller, fully isolated mini-GPUs — like renting out individual rooms in a warehouse instead of the whole building.

---

⚡ Part 4: Making AI Fast and Efficient (The Inference Toolkit)

Everything so far has mostly been about training models. But once a model is trained, it needs to run efficiently to actually respond to users like you and me.

1️⃣9️⃣ Quantization

AI models store numbers with a lot of precision by default — a format called FP32. Quantization converts those numbers into a lower-precision, simpler format like INT8.

flowchart LR
    A["FP32
    0.34, 3.75, 5.64 ..."] -->|"round down"| B["INT8
    64, 134, 217 ..."]

It’s like rounding “3.14159265” down to just “3.” You lose a little precision, but save a huge amount of memory and gain a lot of speed.

2️⃣0️⃣ KV Cache (Key-Value Cache)

One of the cleverest tricks in modern AI. Without a shortcut, the model would have to re-read the entire conversation from scratch every time it generates the next word.

📓 The KV cache is like taking notes as you read a book instead of rereading the whole book every time someone asks a question about it.

2️⃣1️⃣ TTFT (Time To First Token)

How long you wait after sending a prompt before the very first word appears. This is the “first impression” speed metric.

2️⃣2️⃣ TPOT (Time Per Output Token)

The speed during the response — the time between each generated token once the model has already started answering.

flowchart LR
    P[📝 Prompt] --> T[🔤 Tokenize] --> PF["⏳ Prefill
    (process prompt)"]
    PF -->|TTFT| Tok1[Token 1]
    Tok1 -->|TPOT| Tok2[Token 2]
    Tok2 -->|TPOT| Tok3[Token 3]
    Tok3 -->|TPOT| TokN[Token N ...]

TTFT = wait before the first word. TPOT = speed between each word after that.

2️⃣3️⃣ Throughput

The “total work done” metric — requests per second or tokens per second across the whole system, not just one user.

⚖️ The tension: TTFT and TPOT care about how fast one request feels. Throughput cares about how many total requests the system can handle. Optimizing purely for one can hurt the other.

2️⃣4️⃣ Scheduler

Decides which job runs on which GPU, and in what order — like an air traffic controller for a data center.

2️⃣5️⃣ Power and Cooling

GPU servers consume enormous electricity and generate huge amounts of heat. Without effective cooling, GPUs throttle themselves down to avoid damage — wasting expensive compute power.

🔋 At a certain point, the limiting factor on how much AI a company can build isn’t clever software — it’s literally how much power and cooling they can get their hands on.

---

🧵 Bringing It All Together

If you zoom out, these 25 concepts tell one continuous story:

flowchart TD
    A["🧩 1. Fast specialized hardware
    GPU · HBM · CPU · PCIe · Tensor Cores · CUDA"] --> B["📊 2. Measure & manage it well
    Utilization · Memory Bandwidth · Batch Size"]
    B --> C["🔌 3. Connect many GPUs together
    NVLink · NVSwitch · InfiniBand"]
    C --> D["🏗️ 4. Split the training work
    Clusters · Distributed Training · Tensor/Pipeline/Data Parallelism · MIG"]
    D --> E["⚡ 5. Serve it fast & cheap
    Quantization · KV Cache · TTFT · TPOT · Throughput · Scheduler"]
    E --> F["🔋 6. Keep the lights on
    Power & Cooling"]

None of this is really “magic” — it’s engineering, layer upon layer, each one solving a very specific bottleneck that the layer below it created. The next time you see a chatbot respond in half a second, you’ll know there’s an entire stadium of specialized chips, custom wiring, clever caching tricks, and industrial-scale cooling systems working together to make that half-second possible.

---

Cheers,

Sim