Who Decides Which Model Handles What

When a multi-model agent workflow sends a request to an LLM, something has to decide which model handles it. Plumbing, sure. But routing configuration is where you can see who actually controls what in an agent system, often more clearly than any architecture diagram will tell you.

In LangGraph, routing lives in application code. A developer instantiates a ChatAnthropic object for one node, a ChatOpenAI object for another, and the graph compiles with those bindings locked in. There's a configurable pattern that lets you pass model selection as a runtime parameter, but even that parameter is defined in code and set at invocation time. Once compiled, the graph is immutable.

For anyone responsible for the system in production, the implications are concrete. An operator who wants to know which model handled which step can check LangSmith traces after the fact. But there is no configuration surface outside the codebase where they can redirect traffic or even see the routing logic without reading the source. The developer who wrote the graph is the routing authority. Everyone downstream inherits those decisions, often without knowing they were made.

Microsoft's Agent Framework 1.0 opens this up in an interesting way. Every provider, from Azure OpenAI to Anthropic to Ollama for local deployment, implements a shared IChatClient interface. Swapping providers is a one-line change. The interesting signal, though, is in the language of Microsoft's official documentation:

The framework is genuinely open. The defaults have an opinion about where the safe path runs. That opinion is doing real work even when nobody configures anything, because most teams follow the path of least resistance, which is the path the defaults illuminate.

Routing looks different again when it moves from application code to infrastructure.

Databricks Unity AI Gateway sits between applications and model endpoints as a proxy layer. Operators configure fallback chains: if Claude returns a 429, route to GPT-4, then to Llama, walking down the sequence until something succeeds. Every request gets logged to Unity Catalog with cost attribution sliced by endpoint tag, team, and provider. Here the operator decides how failures cascade and who pays for what. A fundamentally different answer to where control lives.

A gateway governs traffic that passes through it, though. Within a multi-step agent workflow, the decisions about which step calls which endpoint happen before the request reaches any gateway. The gateway sees each individual call in isolation, without the sequence logic that produced it.

The gap is real. A pricing workflow might fan out across three models: one for extraction, one for classification, one for comparison. The routing choices compound across the sequence. And the place where those choices get made, whether anyone outside the application code can see or change them, tells you more about your actual governance posture than your policy documents will.