LLM Gateways in Production: LiteLLM vs Portkey vs Bifrost

Every team that ships more than one model into production eventually adopts a gateway or accidentally builds one. This is a comparison of the three that show up most often in 2026 stacks, LiteLLM, Portkey AI Gateway, and Bifrost, plus the self-hosted and Kubernetes-native alternatives, judged on routing, caching, per-key budgets, observability hooks, and the latency they actually add.

What a gateway does, and the three shapes it comes in

Strip the marketing away and an LLM gateway does some subset of five jobs: normalize request and response formats across providers, decide which upstream deployment gets each call, enforce spend and rate policy per caller, cache responses, and emit telemetry. None of that requires a separate process; all of it is easier to operate in one.

The three deployment shapes matter more than the feature lists:

1. In-process library. The LiteLLM Python SDK, the Vercel AI SDK, or the model abstraction inside LangChain or Pydantic AI. No network hop, no extra process to page someone about, but the policy only applies to the one language and the one service that imported it.
2. Standalone proxy you run. LiteLLM Proxy, Bifrost, the Portkey OSS gateway, or the Rust gateway from Helicone. One OpenAI-compatible endpoint, language agnostic, centrally governed, and a new thing in the request path.
3. Hosted control plane. Someone else's proxy. Fastest to adopt, worst blast radius, and it means every prompt in your product transits a third party.

Most gateway incidents come from teams that picked shape 2 for the features and never treated it as tier-zero infrastructure.

Project	License	Runtime	Providers	Routing strategies	Caching	Budgets and keys
LiteLLM	MIT, with enterprise/ under a separate commercial license	Rust core with Python SDK (Python-only path still supported)	100+	simple-shuffle, latency-based, usage-based-v2, least-busy, cost-based, plus Auto-Router	In-memory, disk, Redis (cluster and Sentinel), S3, GCS, semantic via Redis, Valkey or Qdrant	Virtual keys, users, teams, team members, end-customers, agents; multi-window budgets
Portkey AI Gateway	MIT	TypeScript on Node	45+ providers, 1,600+ models	fallback, loadbalance (weighted), conditional, single	Simple caching on all plans; semantic caching on select Enterprise plans only	Virtual keys and budgets via the Model Catalog and control plane
Bifrost	Apache 2.0	Go	23+ providers, 1,000+ models advertised	Weighted keys and failover in OSS; adaptive load balancing and cluster mode are enterprise	Semantic cache shipped as a plugin	Customers, teams, virtual keys; rolling or calendar-aligned budgets, rate limits at the key level
Helicone AI Gateway	Apache 2.0	Rust	20+ providers, 100+ models	Latency-aware (P2C plus PeakEWMA), weighted, and cost strategies, with automatic failover	Response caching backed by Redis or S3	Rate limits per user, per team, or global

Star counts at the time of writing: roughly 55k for LiteLLM, 12.6k for the Portkey gateway, 6.9k for Bifrost. A rough proxy for how much third-party integration debugging is already done for you, and not much else.

The benchmark war, and how to read it

The three projects publish numbers that flatly contradict each other, which is itself the most useful data point here.

Bifrost's repo headline calls it 50x faster than LiteLLM with under 100 microseconds of overhead at 5k RPS, and its docs report 11 microseconds of added overhead per request in a sustained 5,000 RPS run on a t3.xlarge. LiteLLM's own benchmark post, published 22 July 2026 and run on a 4 vCPU / 16 GB instance against a local deterministic Rust mock upstream with logging callbacks off, reports added p99 latency of roughly 0.7 ms for the LiteLLM Rust gateway, 2.3 ms for Portkey, 4.5 ms for Bifrost v1.6.4, and 257.7 ms for the legacy LiteLLM Python v1 path. Peak memory in the same run: 21.8 MB, 90.4 MB, 199.1 MB, and 329.5 MB respectively. Sustained throughput was roughly 2,814 req/s for the Rust build against roughly 2,744 for Bifrost, which is a tie inside the noise floor. Portkey, separately, advertises sub-1 ms latency and a 122 kb footprint.

One vendor's headline says it is 50x faster than the other; the other's benchmark says its Rust gateway has 7x lower overhead and 9x less memory than the first. What is actually happening is that everyone benchmarks proxy forwarding against a mock upstream, which deletes the single largest variable in the real system: the provider's own inference time. When the model takes 500 ms to 5 seconds to answer, the difference between 11 microseconds and 4.5 ms of gateway overhead is invisible to users. Independent write-ups have made exactly this point, and they are right.

The parts of these numbers that do matter:

• The Python-to-Rust delta is real and large. Going from roughly 257 ms to sub-millisecond p99 added latency, and from 329 MB to 22 MB of peak memory, is not a benchmarking artifact. If you are running LiteLLM's older pure-Python proxy under load, that is worth acting on.
• Memory is a scheduling problem. A 200 MB to 330 MB resident footprint per replica changes how many gateway pods fit on a node, and therefore what your gateway costs.
• Overhead under contention is the number nobody publishes. Vendor benchmarks show a healthy gateway with a healthy upstream. Production breaks on gateway behavior when a provider is timing out, retries are stacking, and cooldown logic is deciding what to do.

The honest measurement is gateway end-to-end latency minus direct-to-provider latency at matched concurrency, on your traffic, with your features enabled. Every gateway slows once you turn on semantic caching, guardrails, and synchronous logging, and no marketing benchmark has those on.

Routing and fallbacks: the part that earns its keep

Format normalization is a commodity. Routing is where a gateway saves you or hurts you.

LiteLLM's router exposes a handful of named strategies. simple-shuffle is the default, and the docs recommend it for production; it weights deployments by configured rpm, tpm, or an explicit weight. latency-based-routing picks the fastest recent deployment, usage-based-routing-v2 is the async implementation that tracks usage in Redis and routes to the deployment with the lowest TPM for the current minute, least-busy counts in-flight calls, and cost-based-routing sorts by price from the built-in pricing table. Reliability primitives sit alongside: num_retries, allowed_fails and cooldown_time to eject a bad deployment from rotation, order for priority tiers, and separate fallbacks, context_window_fallbacks, and content_policy_fallbacks lists so a rate limit, an oversized prompt, and a refusal do not all take the same escape hatch.

model_list:
  - model_name: chat-primary
    litellm_params:
      model: azure/gpt-4o-eastus
      api_key: os.environ/AZURE_API_KEY
      rpm: 900
      weight: 2

router_settings:
  routing_strategy: usage-based-routing-v2
  allowed_fails: 3
  cooldown_time: 30
  num_retries: 3
  fallbacks: [{"chat-primary": ["chat-backup"]}]

Portkey takes a config-object approach instead of a strategy name. A config is JSON with a strategy.mode of fallback, loadbalance, conditional, or single, plus a targets array where each target can carry a weight, an on_status_codes trigger, and default_params / override_params / drop_params to reshape the request per upstream. The drop_params hook is more useful than it sounds: it removes fields by bracket-notation path, which is how one request body survives a provider that rejects logprobs and one that requires it.

{
  "strategy": { "mode": "fallback" },
  "targets": [
    { "provider": "@openai-prod", "override_params": { "model": "gpt-4o" } },
    { "provider": "@anthropic-backup", "on_status_codes": [429, 500, 503] }
  ]
}

Bifrost keeps weighted key selection and failover in the open-source build and puts adaptive load balancing, cluster mode, guardrails, and the MCP gateway behind the enterprise line. Its adaptive balancer scores both provider and key selection from live metrics, ranking error penalty first, latency second, utilization third. Selection is weighted random rather than winner-take-all, so penalized routes get re-probed instead of starved, and penalties decay once a route recovers. Read the cluster-mode fine print, though: each node keeps its own metrics, the only signal gossiped between nodes is a rate limit backoff, and it is scoped to a region. There is no global weight consensus.

Two failure modes to design around whichever you pick. First, mid-stream failover does not exist. Once the gateway has flushed tokens to the client, it cannot transparently switch providers; the application has to handle a truncated stream. Second, cross-provider fallback is not a free lunch. Falling back from one vendor's flagship to another's changes instruction-following behavior, tool-call formatting, and refusal boundaries. Run the fallback path through Promptfoo before you trust it, not after the primary goes down.

Model-level routing, sending easy prompts to the cheap model, is a separate discipline. RouteLLM from LMSYS is still the clearest public result: its matrix-factorization router hit 95% of GPT-4 quality while sending only 26% of calls to GPT-4, and with LLM-judge augmented training data that dropped to 14% of calls for the same quality bar. Reported cost reductions were above 85% on MT-Bench, 45% on MMLU, and 35% on GSM8K against a GPT-4 Turbo / Mixtral 8x7B pair. LiteLLM ships its own Auto Router, now on a v2 that classifies each request with heuristics, an LLM classifier, or keyword rules, then routes to a pinned model, a random pool, or a Thompson-sampled pool when adaptive is on. Treat any of these as a thing you evaluate on your own traffic distribution, because the quality-retention number is entirely a function of how easy your prompt mix is.

Caching is three different features wearing one name

Provider-side prompt caching

The provider caches your prompt prefix and bills the cached tokens at a large discount. The gateway's job here is to not break it, and load balancing breaks it by default: spraying requests across two Azure deployments or two API keys leaves the prefix cache cold much of the time. This is the same problem KV-cache-aware routing solves for self-hosted inference. The lever that exists today is provider-side stickiness rather than proxy cleverness: OpenAI's prompt_cache_key sends requests carrying the same key to the same backend, and providers enforce a minimum prefix length before caching engages at all, 1,024 tokens on Claude 3.x. Run an agent loop through naive load balancing with no cache key and you pay full price for context you already sent.

Exact-match response caching

Hash the normalized request, store the response. Cheap, safe, and useless for conversational traffic where no two requests are identical. LiteLLM supports it across in-memory, disk, Redis (including cluster and Sentinel), S3, and GCS backends, with per-request controls sent in the request body: ttl, s-maxage to reject stale entries, no-cache to force a fresh call, no-store to skip writing, and namespace to partition keys. mode: default_off makes the whole thing opt-in per request, the right default for anything user-facing.

litellm_settings:
  cache: true
  cache_params:
    type: redis
    ttl: 600
    mode: default_off

Semantic caching

Embed the request, do a vector similarity search, return a stored answer above a threshold. LiteLLM can back this with Redis, Valkey, or Qdrant; Bifrost ships it as a semantic_cache plugin you can reconfigure live through the API; Portkey puts simple caching on all plans and semantic caching on select Enterprise plans only, because it needs a vector database behind it.

This is the feature most likely to quietly damage your product. Measured hit rates on real conversational traffic land well below what vendor pages imply, and the failure mode that matters is not a miss, it is a false positive. Queries that differ only in a date range, a negation, or an analytical intent routinely clear 0.95 cosine similarity and return a confidently wrong cached answer. If you turn it on, scope caches narrowly by domain rather than running one global cache, start conservative on the threshold, and instrument the false-positive rate before you tune. A semantic cache without a precision dashboard is an unmonitored correctness regression.

Per-key budgets and the governance layer

This is the feature that justifies a standalone proxy in most organizations, because it cannot be done properly in a library. Someone has to be able to say "this team gets $2,000 a month across these three models" without every service redeploying.

LiteLLM has the deepest hierarchy: virtual keys, internal users, teams, team members, end-customers keyed off the user field in the request, and agent-level limits. Budgets combine rather than override, so a key belonging to a team is checked against both the team ceiling and the user's personal ceiling unless you explicitly set skip_user_budget_on_team_key, which landed in v1.94.0. Each level takes max_budget and budget_duration, plus tpm_limit and rpm_limit. Keys additionally support multiple concurrent windows, which is how you express a daily circuit breaker under a monthly cap:

curl 'http://localhost:4000/key/generate' \
  -H 'Authorization: Bearer $LITELLM_MASTER_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "budget_limits": [
      {"budget_duration": "24h", "max_budget": 10},
      {"budget_duration": "30d", "max_budget": 100}
    ]
  }'

Per-model budget caps on a key are an enterprise feature.

Bifrost models the same problem with a three-tier hierarchy of customers, teams, and virtual keys, where a virtual key belongs to exactly one team or one customer but never both. Budgets stack up the hierarchy and default to rolling windows, with opt-in calendar alignment that resets on UTC day, week, month, or year boundaries. That switch matters: calendar-aligned budgets are easier to reconcile against a finance month and worse at smoothing a spike on the 1st. Rate limits are virtual key only; teams and customers cannot carry them. Keys also carry model and provider allow-lists and can be disabled instantly, which is the control you want when a leaked key starts burning Opus tokens.

Portkey exposes budgets, rate limits, and model allow-lists via its Model Catalog and virtual keys, though the governance surface is oriented around the control plane rather than the standalone OSS binary. If self-hosting with full governance is the requirement, confirm that distinction before you commit.

Observability hooks: the gateway as a tracing choke point

Every request already passes through the gateway, so it is the cheapest place to attach tracing. LiteLLM's callback system is the most promiscuous of the three, with first-class support for Langfuse, OpenTelemetry (console, OTLP over HTTP or gRPC, Honeycomb, Traceloop), Arize Phoenix, LangSmith, MLflow, Opik, Datadog, Prometheus, plus bulk sinks like S3, GCS, Azure Blob, DynamoDB, and SQS. Configuration is two lines:

litellm_settings:
  success_callback: ["langfuse", "otel"]
  failure_callback: ["sentry"]
  turn_off_message_logging: true

turn_off_message_logging redacts prompt and completion bodies while keeping token counts, latency, and cost, which is usually the right posture for anything touching regulated data. The x-litellm-call-id response header correlates a user-visible failure with a specific trace, and callbacks can be scoped per team or key so one tenant's traffic goes to a different sink.

Bifrost ships otel and telemetry as plugins alongside Prometheus metrics. Portkey logs and traces natively into its own analytics layer. Helicone is the interesting hybrid: routing and observability are the same product, so request logging needs no separate instrumentation step.

One caveat the vendor docs skip: a gateway sees server-side time only. It cannot see client-side queueing, retries your SDK performed before the request arrived, or time a mobile client spent on a bad network. If gateway traces are your only telemetry, your p99 is systematically optimistic. Pair gateway-side logging with in-process instrumentation from OpenLLMetry or OpenLIT rather than treating the proxy as the single source of truth.

The Kubernetes-native branch: routing to your own GPUs

If the upstreams are your own vLLM or SGLang replicas rather than vendor APIs, a general-purpose LLM gateway is solving the wrong problem. The dominant variable stops being provider selection and becomes KV cache locality.

Prefix caching lets vLLM and SGLang skip recomputing tokens already processed, but only if the follow-up request lands on the replica holding those blocks. A round-robin balancer destroys that. The vLLM production stack ships round-robin and session-ID routing plus prefix-aware routing, KV-cache-aware routing, and disaggregated prefill. On the Kubernetes side, llm-d does inference-aware scheduling through the Gateway API Inference Extension, and the project's own published benchmark, 8 vLLM pods on 2 H100s each for 16 GPUs total, reports roughly 57x faster time to first token and more than double the throughput for precise cache-aware scheduling against cache-blind scheduling on identical hardware, with the p90 TTFT gap widening past 170x on the high prefix-sharing run. AIBrix covers similar ground with its own routing and autoscaling control plane.

Those numbers dwarf anything in the gateway latency debate, which tells you where to spend attention when you own the accelerators. The common shape is both layers: a KV-aware router in front of the GPU fleet, and a policy gateway in front of that handling keys, budgets, and vendor fallback. Outside Kubernetes, the same split shows up between Ollama or a local runtime for development and the gateway for everything shared.

The general-purpose API gateways have grown LLM plugins too. Envoy AI Gateway is the natural pick if Istio is already in the mesh, Kong's AI plugins make sense if Kong already fronts your APIs, and the Linux Foundation's agentgateway is built around MCP and agent-to-agent traffic with LLM routing alongside. None beat a dedicated LLM gateway on model-aware features; all beat it on being infrastructure your platform team already runs.

When a gateway is overkill

Langfuse sells observability and has no proxy to sell you, which makes its write-up the least motivated one in the category. It declines to say every application needs a gateway. The framing is conditional: a gateway is one more hop and one more system to operate, the latency cost is real and depends on the gateway and the deployment, and it earns that cost only under specific conditions, namely multiple providers, a reliability target above one provider's SLA, many teams, or developer tooling at scale. Where the thing has to run inside a VPC, they call self-hosted LiteLLM the default choice. That conditional framing is under-represented in a market where most comparison content is written by gateway vendors.

A gateway is probably overkill when:

• You call one provider and have no realistic plan to add a second. Provider SDK retries plus a circuit breaker cover the actual failure modes.
• One team owns all the LLM traffic. Budget governance across teams is the load-bearing feature; without multiple teams you are buying a dashboard.
• Everything is one language and one service. The OpenAI Agents SDK, Pydantic AI, Instructor, or LlamaIndex all handle multi-provider abstraction in-process with no extra hop and no extra pager rotation.
• Your latency budget is genuinely tight and non-streaming. Sub-second classification or embedding traffic feels a 4 ms proxy hop in a way that a chat completion never does.

It is load-bearing when: multiple teams share provider quota and someone needs per-team spend attribution; you have a hard requirement to fail over between vendors during an outage; keys must be issuable and revocable without a deploy; compliance requires a single audited egress point for all model traffic; or you are running heterogeneous clients, including things like Open WebUI, Cline, or Claude Code pointed at internal models, that you cannot instrument individually.

A useful middle path: run the gateway, but keep the direct provider client behind a feature flag in the highest-traffic service. One afternoon of work, and a gateway outage then degrades governance rather than taking down the product.

How to choose

Pick LiteLLM if governance depth is the requirement. Nothing else has the same budget hierarchy, breadth of logging callbacks, or provider coverage, and the Rust core removed the one legitimate reason to avoid it. The tradeoff is surface area: it is a large project shipping weekly releases (v1.94.0 landed 28 July 2026), and the enterprise carve-out under enterprise/ covers features you may assume are open source until you read the directory.

Pick Bifrost if you want a small Go binary with clean governance semantics and you can live with adaptive load balancing and cluster mode being enterprise. npx -y @maximhq/bifrost or docker run -p 8080:8080 maximhq/bifrost gets you a running gateway with a web UI on port 8080 in about a minute, with SQLite or Postgres 16 and above as the config store. The plugin factoring, with semantic cache, OTel, and telemetry as separate modules, is right for something on the critical path.

Pick Portkey AI Gateway if the config-object model fits how you think and edge deployment matters; it runs on Cloudflare Workers, which none of the others do. Verify which features you need are outside the enterprise tier first, semantic caching in particular.

Pick Helicone if you want routing and observability as one product rather than two integrations, and Envoy AI Gateway or Kong if your platform team will maintain it either way.

What to watch over the next two quarters: whether MCP credential handling at the gateway consolidates around one pattern, since all three are still reshaping it release over release, which usually signals the design is not settled, and whether prefix-cache-aware routing moves from the self-hosted inference stack into general-purpose gateways. The second one would be worth more to most teams than another order of magnitude off an already-invisible proxy overhead.