The Conversation, Not the Batch Job

Training and inference are often talked about as two flavors of the same workload, different points on a single AI compute curve. Operationally, they barely resemble each other.

Training is a closed conversation.

When a model is being trained, most of its network traffic never leaves the building, or the small cluster of buildings, it's running in. Thousands of GPUs are synchronizing parameters with each other continuously, exchanging enormous volumes of data as they jointly work through a training run. That traffic is intense, it's often described as the reason "the network is effectively the computer" inside a modern AI cluster, but it's internal. The model doesn't care where its users are, because during training it doesn't have users yet. It cares about how fast its own GPUs can talk to each other.

That's a solvable problem with a known shape: build the cluster, engineer the fabric inside it, and locate the whole thing wherever power, land and the economics make sense. It's also, not coincidentally, exactly the design problem the last three years of hyperscale site selection has been optimized to solve.

Inference is a conversation with someone else.

The moment a model moves from being trained to being used, its traffic profile inverts. Every prompt, every voice query, every agentic workflow that plans a task, calls a tool, retrieves data, and acts on it, is a live exchange between the model and a person, an application, or an enterprise system sitting somewhere else entirely. That traffic has to leave the building. It has to cross networks it doesn't control, reach clouds and enterprise systems it doesn't own, and come back fast enough that the person on the other end doesn't notice the round trip.

Distance is not a rounding error in that equation. Light in fiber moves fast, but it isn't instantaneous, and every additional network hop between a user and a model adds real, measurable delay on top of the physical distance. A single exchange might tolerate that. An agentic workflow that chains several calls together to plan, retrieve, generate, and verify before it responds compounds that delay at every step. One data center analyst covering Cisco Live 2026 put the multiplier bluntly: agentic AI alone is projected to push AI network traffic up by another order of magnitude on top of already-tripling volumes over the next few years, precisely because these workflows don't just answer once.

The traffic is also going somewhere different.

This isn't only about speed, it's about topology. Training traffic is largely contained, GPU to GPU, inside a facility or a small number of directly linked sites. Inference traffic runs the other way: out to the networks, cloud platforms, and enterprise systems that inference depends on to actually do anything useful. STL Partners estimates that each megawatt of AI compute now generates up to twice the interconnect throughput of a comparable megawatt of traditional cloud capacity, and describes that traffic as increasingly multi-directional and bursty rather than following the predictable, symmetrical patterns older data center interconnect networks were built around.

Put those two facts together, live and latency-sensitive, and external-facing rather than internal, and a facility's value for inference starts to depend on something training never much cared about: how well-connected it is to the outside world, not just how much power it can deliver. One industry assessment of 2026 network trends put it plainly: AI's inference traffic is exposing the limits of network architectures that were built for older, more linear, more predictable data flows.

What this actually changes

None of this makes training's infrastructure needs disappear, and it doesn't make raw compute unimportant. But it does mean the qualities that matter for a training-optimized facility, contiguous power, internal fabric bandwidth, room to scale a single cluster, are not the same qualities that matter for a facility trying to serve inference well. Inference rewards proximity to the people and systems generating the queries, and it rewards dense, diverse connectivity to the networks and clouds those queries have to pass through.

That's a different design brief. It's not a bigger version of the same building, it's a building optimized for a fundamentally different kind of conversation. What that means for where these facilities get sited, and what they need to look like once they're built, is where this series goes next.

‍