What Is AI Inference? The Run-the-Model Phase Explained

Training and inference are two phases of a model's life, and they stress hardware in opposite ways. Training builds the model: it repeatedly runs data through the network and adjusts millions or billions of parameters until the model's outputs are accurate. It is computationally enormous, typically runs on large multi-accelerator clusters, tolerates long runtimes measured in hours to weeks, and is done once or on a periodic schedule. Inference uses the finished model: it feeds in new input and reads out a result, with no parameter updates, and it runs continuously for the entire life of the deployment. Where training is a scheduled, capital-heavy effort, inference is an ongoing operating cost that recurs with every request.

That difference is why the hardware that trains a model is rarely the hardware that serves it. Training is throughput-bound and batch-friendly — you can wait for a long job to finish — so it rewards raw compute concentrated in big clusters. Inference is frequently latency-sensitive, because a user or a mission system is waiting for the answer, so it rewards fast, efficient response on a much smaller footprint. A model trained across dozens of accelerators may be served on a single card, on a CPU, or on a compact edge device. For the majority of buyers who consume pre-trained models rather than build them, it is this serving footprint — not the training cluster — that they actually budget for, deploy, and scale.

Inference is a forward pass through the model. The input is first encoded into numbers — tokens for text, tensors for images or audio — and those numbers flow through the network's layers a single time, producing an output at the end. No weights change. For generative and large-language models, the output is produced token by token, with each new token conditioned on the ones already generated, which is why a long response takes longer than a short one and why the time to the first token and the steady per-token rate are measured as separate figures.

Two levers dominate real-world inference performance: precision and batching. A model can be served at reduced numerical precision — FP16 or BF16, or quantized further to INT8 or INT4 — which shrinks its memory footprint and speeds up computation with limited accuracy loss, and it is a common way to fit a large model onto smaller hardware. Batching groups concurrent requests so the accelerator processes many at once, raising overall throughput at some cost to any single request's latency. Memory matters disproportionately in all of this: the model's parameters plus its working state — for language models, the key-value cache that grows as text is generated — must fit in the accelerator's memory, so memory capacity and bandwidth often bound inference more tightly than raw compute does.

Throughput is the first number to nail down: queries per second, images per second, or, for language models, tokens per second at a stated batch size. It sets how many concurrent users or how large a workload a single node can serve, and therefore how many nodes a target load will require. Latency is the second: the time to a single response, which should be watched at a tail percentile such as p95 or p99 rather than an average, because the slowest responses are the ones users and time-critical systems actually feel. Throughput and latency trade against each other through batching, so a sound specification states both — for example, a required tokens-per-second at a maximum acceptable p99 latency — instead of one in isolation.

The third currency is watts, and the heat those watts produce. Accelerators are dense, power-hungry parts, and a rack has finite power and cooling. "Watts per rack unit," and the related measure of performance per watt, is a genuine procurement constraint: a data center often runs out of amperage or cooling capacity before it runs out of physical rack space, and in edge, vehicle, or forward deployments the power and thermal envelope can be the hard limit that decides the design. Evaluating inference on throughput per watt, not throughput alone, is what keeps both the operating cost and the facility itself sustainable as a deployment scales from a pilot to production.

Inference runs across a spectrum of locations. Centralized inference in a data center or cloud concentrates accelerators where power, cooling, and scale are available, and it is the default for the largest models and for high aggregate request volume. Edge inference instead pushes the model close to where data is created — a factory floor, a camera, a vehicle, a clinic, a forward site — to cut network round-trip latency, keep sensitive data local for privacy or bandwidth reasons, and keep working when connectivity is intermittent. The same trained model can be deployed to either environment, with the serving hardware sized to the location and its constraints.

The processor choice follows the model and the target. GPUs and dedicated AI accelerators dominate demanding inference because their massively parallel design suits the matrix math involved, and they are the standard for large language and vision models. CPUs remain perfectly capable for smaller or lighter-traffic models, classical machine learning, and workloads where an accelerator would sit mostly idle — and every server already has one. Compact edge accelerators and integrated neural processing units cover low-power, ruggedized, and embedded deployments. There is no single right answer, which is exactly why the CPU-versus-GPU decision and the edge-AI deployment pattern each deserve their own detailed treatment rather than a blanket rule.

Start from the model you actually intend to serve and its memory footprint, because accelerator memory — both capacity and bandwidth — is usually the first thing that constrains inference. Confirm the model, at your chosen precision, fits in the accelerator's memory with headroom for the working state and your batch size; under-provisioning memory forces you into heavier quantization, a spill to slower memory, or a smaller model. From there, match the platform: a GPU-ready server with the power connectors, cooling, and physical slots for the accelerators you have chosen, enough system memory and fast storage to keep them fed, and the network bandwidth to move data in and results out. Data-center accelerators such as the NVIDIA L4 — a compact, low-power inference card — and the H100, a high-end data-center GPU, bracket the range, from efficient edge-friendly serving to the very largest models.

Practical sourcing realities matter as much as the specification sheet. Our working catalog carries a broad NVIDIA line — on the order of 7,800 SKUs spanning accelerators, networking, and the software and support that surround them — alongside board-partner cards such as PNY's roughly 96-SKU range, and the GPU-ready rack servers and workstations that host them. Be candid about lead times: accelerator-class hardware is in high demand, and the large majority of graphics-card listings in our catalog show a back-ordered or lead-time status rather than same-day stock, so plan procurement around that reality instead of assuming shelf availability. For federal and regulated buyers, sourcing can be screened for TAA and NDAA Section 889 considerations as part of the quote — a screening service we perform, not a certification we hold — and configurations can be validated before they ship. When a workload is defined, moving from a model's memory requirement to a costed, lead-time-honest bill of materials is the step that turns an inference plan into hardware you can actually deploy.