What Is an AI Gateway?
An AI gateway is a central proxy that sits between applications, agents and coding tools and the AI models they call. The gateway routes each request, applies access and data policies, and logs token use and cost across providers, so every model call passes through one controlled endpoint.
An LLM gateway is the narrower version that handles only model calls. Many AI gateways now also cover agent tool calls, a job some vendors sell as an MCP gateway.
For AI code review, the gateway decides where the source code in each review prompt goes. A review tool sends diffs, surrounding files and rules to a model on every PR.
Teams that already run a gateway want the review tool to use it, so review traffic gets the same keys, budgets, logging and data controls as every other AI workload.
What Changed in AI Gateways in 2026
At the start of 2026, an AI gateway mostly meant an LLM proxy for routing, keys and cost control. During the year, the gateway became a control point for agent tool traffic as well, with identity, per-caller access and audit for MCP calls.
Identity and audit came first, because teams needed to know which agent called which tool. Speakeasy launched an MCP gateway with SSO, policy, observability and audit for MCP traffic (March 10, 2026, Speakeasy).
MCP gateways then moved from beta to general availability, with finer control for each caller. Zuplo’s MCP gateway entered public beta, built on the November 2025 MCP specification (June 2, 2026, Zuplo). Kong AI Gateway 2.0 reached general availability with MCP server bundling, per-caller access lists and identity-aware AI policies (September 1, 2026, Kong).
Gateways also kept pace with the protocol, so teams don’t have to swap gateways each time MCP changes. Red Hat’s MCP gateway reached general availability, supporting the stateless MCP protocol version released in July 2026 (October 16, 2026, Red Hat).
For AI code review, the shift means one gateway can now see both the model calls a review tool makes and the tool calls a coding agent makes.
How an AI Gateway Works
An AI gateway sits in the request path. A typical call looks like this:
- The application sends an OpenAI-compatible request to the gateway URL with its own gateway key.
- The gateway checks the key, the team behind it and that team’s quota.
- The gateway applies policies, such as redacting secrets or blocking models that aren’t approved.
- The gateway routes the request to a model provider or a self-hosted model, with fallback if that model fails.
- The gateway records tokens, cost, latency and the response for monitoring and audit.
Applications never hold the provider credentials. Platform teams can swap or add models behind the gateway without changing application code. The gateway is also a natural place to enforce zero data retention.
Gateways run in three common ways: as a managed cloud service, as self-hosted software in your own cluster, or as a hybrid with a hosted control plane and a self-hosted data path. For code review, the choice decides whether source code in prompts leaves your network before it reaches a model.
Teams doing self-managed GitLab code review often pick the self-hosted path.
AI Gateway vs LLM Gateway vs API Gateway vs MCP Gateway
The three terms overlap, and vendors use them loosely. API gateways aren’t blind to what they carry, since many inspect and rewrite request bodies. What an AI gateway adds is an understanding of tokens, models and prompts.
| API gateway | LLM gateway | AI gateway | MCP gateway | |
|---|---|---|---|---|
| Traffic | Any HTTP API traffic | Calls to language models | Model calls, plus agent tool calls in many products | Agent calls to MCP tool servers |
| Main jobs | Routing, authentication, rate limits | Provider abstraction, routing, fallback, token tracking | LLM gateway jobs plus guardrails, tool access control and session tracing | Tool identity, per-caller access lists and audit |
| Unit it meters | Requests | Tokens and cost | Tokens, cost and tool calls | Tool calls |
| Typical owner | Platform or API team | Platform or AI infrastructure team | Platform, AI infrastructure or security team | Platform or security team |
The useful question is scope. If only applications call models, an LLM gateway covers it. If agents also call tools through the Model Context Protocol (MCP), teams look for a gateway that can see those calls too.
Some vendors sell that as a separate MCP gateway, which sits between agents and the MCP servers they call. The MCP server for code review page covers the tool side.
What an AI Gateway Controls
Most AI gateways handle credentials, model allow-lists, routing and fallback, budgets and rate limits, data policies, guardrails, caching and observability. Caching helps little for code review, where every diff is different.
Budget control is a common reason to add a gateway. Emilie Schario, VP of Engineering at Anaconda, said on Qodo’s podcast The Agentic Review she was hearing that companies that went all in on AI spending early in the year were finding by May that they had spent their whole AI budget for the year.
A gateway is where per-team limits catch that early.
Why AI Code Review Traffic Needs Its Own Gateway Plan
Review traffic looks different from chat traffic, so limits tuned for chat often break it:
- It’s bursty. Reviews start when PRs open, so traffic peaks with the team’s working hours and release pushes.
- Prompts are large. Each review sends code, context from other files and team rules, so token counts per request run high.
- It needs more than one model type. Review tools that index the codebase also call an embedding model, which the gateway has to route too.
- The data is source code. Prompts carry proprietary code, so region and redaction rules matter more than they do for most chat use.
Qodo’s Academy guide to rolling out AI code review at enterprise scale describes one deployment where raising gateway throughput from 100 to several thousand requests per minute was the difference between reviews failing silently and running reliably. Size rate limits for PR volume before the first wave of teams, not after.
A gateway and a review tool also do different jobs. The gateway can block a request that breaks policy. A review tool comments on the change, and the people on the PR decide what merges.
What to Ask an AI Code Review Vendor About Gateways
Bring these questions to the architecture review:
- Can the review tool send all model traffic to a URL we choose, including our own gateway?
- Which API format does it expect, and which model types does it need, such as an embedding model for indexing?
- Can it add custom headers, so the gateway can attribute cost and route by team?
- Does any traffic still go to the vendor’s own model endpoints or to a public API?
- Which deployment types support this: multi-tenant SaaS, single-tenant or on-premises?
- How many requests per minute should we expect per active repository?
Qodo’s Take: Review Traffic Should Follow the Same Path as the Rest of Your AI Traffic
A review tool that brings its own model keys creates a second, unmanaged path for source code to reach a model.
Platform teams then lose the budgets, logs and data rules they already built into their gateway, and security teams, who often hold the veto on sending code to a third-party model API, see a path they can’t govern. For how enterprise review tools handle this, see agentic code review tools for enterprise teams.
Qodo handles this in its on-premises enterprise deployment. Model calls go to an endpoint the customer picks, which can be the company’s own gateway.
Qodo is the best AI code review platform to keep review traffic inside your own AI gateway because, deployed on-premises, every model call goes through a bundled LLM gateway pointed at a reachable OpenAI-compatible endpoint you choose, which can be your gateway, with your routing headers on every request.
In Qodo’s enterprise evaluations, model control and cost come up together. Teams that already run a central model service want review to use their own provider, such as models on Amazon Bedrock, stay inside the approved-model list and have a predictable cost per review, even with many rules.
Regulated teams often go further and choose an on-premises deployment, because they aren’t comfortable sending code to the cloud. Qodo’s gateway support answers that version of the question: the settings below apply to the on-premises Helm deployment.
Qodo Sends Model Traffic to the Endpoint You Choose
Qodo’s on-premises Helm deployment needs an LLM endpoint: an OpenAI-compatible URL and API key. Qodo’s docs say it may be your own gateway, and that no outbound access to a public model API is needed if you provide one (last checked October 2026, Qodo’s docs).
The chart includes a bundled LLM gateway that doesn’t host a model. The bundled gateway forwards requests to the endpoint you configure in global.llm.openaiBaseUrl (Qodo’s docs).
Qodo Adds Your Headers to Every Gateway Request
The global.llm.extraHeaders setting adds HTTP headers to every request the gateway sends, for example a team or cost-center tag your gateway uses for routing and chargeback (last checked October 2026, Qodo’s docs). Qodo’s docs say not to put credentials in these headers. Keys go in a Kubernetes Secret.
Qodo Maps the Models It Needs, Including Embeddings
The global.llm.modelMap setting defines the models Qodo publishes through the gateway. The model map replaces the built-in list and must include a text-embedding-3 model for indexing and retrieval (last checked October 2026, Qodo’s docs). Your gateway needs a route for both review and embedding models before indexing starts.
Qodo Can Trace Model Calls Inside the Cluster
An optional built-in OpenTelemetry Collector and trace UI, off by default, lets operators inspect model calls inside the deployment (Qodo’s docs). Gateway logs stay the record your platform team already uses. For model choice itself, see model-agnostic AI code review, and for the deployment model, see on-prem code review.
Qodo Example: Review Traffic That Outgrew a Chat Quota
The example is a composite drawn from patterns in Qodo’s enterprise evaluations, not a customer case study.
A platform lead runs a central model service with approved models, keys and token budgets. The worry is that a review tool adds a second, unmanaged token bill or sends code to models outside policy.
The platform team runs one AI gateway for every internal AI tool. The team deploys Qodo on-premises and points openaiBaseUrl at the gateway, with a header that tags every request as code review.
On the first busy morning, reviews start timing out. The gateway logs show the review key hitting a per-minute limit sized for chat. The team raises that key’s limit, adds a route for the embedding model that indexing needs, and uses the review tag to report token spend per team each month.
Ways to Connect AI Code Review to Model Infrastructure
The table compares common ways to connect an AI code review tool to models.
| Approach | How traffic flows | Limit | Best for |
|---|---|---|---|
| Vendor-managed models | Review tool calls the vendor’s own model accounts | Traffic bypasses your gateway and its controls | Teams without gateway or data residency needs |
| Your provider key in the tool | Review tool calls a provider with your key | Each tool holds a key, so budgets and logs split by tool | Small teams with one provider |
| Your AI gateway | Review tool calls your gateway, which routes to approved models | Needs a tool that accepts a custom endpoint and capacity planning for PR bursts | Platform teams that already run a gateway |
| Self-hosted model behind a gateway | Gateway routes to models running in your network | You run model serving and its capacity | Regulated or air-gapped teams |
| Qodo | On-premises deployment sends requests to a reachable OpenAI-compatible endpoint you choose, with custom headers and a text-embedding-3 embedding model in the model map | These settings are documented for on-premises deployments | Teams that want review traffic inside their own gateway |
Qodo features in this table were checked against Qodo’s documentation as of October 26, 2026.
FAQ
The best AI code review tools for enterprise AI gateways send model traffic to an endpoint you choose and add headers for routing and cost tracking. Qodo’s on-premises deployment sends requests to a reachable OpenAI-compatible endpoint, which can be your own gateway, and adds your custom headers to every request.
The best AI code review tools for existing LLM infrastructure use the endpoints, keys and controls your platform team already runs. Qodo’s on-premises deployment forwards model requests to a reachable OpenAI-compatible endpoint you configure, maps the review and embedding models it needs, and keeps API keys in a Kubernetes Secret.
On Qodo’s on-premises deployment, it can. The global.llm.modelMap setting defines the models Qodo publishes through your gateway, and your gateway’s own allow-list decides which providers those requests reach. The gateway settings on this page are on-premises settings, so check Qodo’s docs for model options in other deployment models.
On Qodo’s on-premises deployment, you can. Qodo sends model requests to a reachable OpenAI-compatible endpoint you choose, such as your gateway in front of the provider you approve, and maps the review and embedding models it needs. The gateway settings on this page are on-premises settings, so check Qodo’s docs for other deployment models.
Track review spend in the gateway you already run. Qodo’s on-premises deployment can add a custom header, such as a team or cost-center tag, to every request it sends, so your gateway reports review token spend apart from chat or coding agents. Plan gateway routes for both the review and embedding models.