“Helicone | Managed / self-hosting options | Observability, request logging, caching, gateway capabilities | Monitoring and managing LLM usage”
“Combines LLM observability with a gateway that supports multiple models and providers.”
“Portkey | Managed / self-hosted options | Routing, retries, fallbacks, observability, governance | Production applications and enterprise teams”
“Configurable routing.”
“Automatic retries and fallbacks.”
“Request tracing and observability.”
“Rate limits and access controls.”
“Enterprise governance.”
“OpenRouter | Managed | Single API, large model catalog, easy model switching | Fastest way to access many models”
“No gateway infrastructure to deploy.”
“One API key and API interface.”
“Access to a large catalog of models.”
“Ability to change models by changing the model identifier.”
“Routing and fallback options.”
“The trade-off is that you introduce another service into your inference path, with its own pricing, data-handling arrangements, and availability considerations.”
“LiteLLM | Open source Self-hosted or managed | OpenAI-compatible API, 100+ providers, fallbacks, budgets, routing | Maximum flexibility and infrastructure control”
“One OpenAI-compatible API for multiple providers.”
“Support for OpenAI, Anthropic, Gemini, Mistral, Bedrock, Vertex AI, Ollama, and many others.”
“Automatic retries and fallback models.”
“Load balancing across deployments.”
“Centralized API keys, budgets, and rate limits.”
“Self-hosting in your own Kubernetes cluster or infrastructure.”
“Vercel AI Gateway | Managed | Unified model access, routing, provider integrations | Applications built with Vercel AI SDK”
“This is worth evaluating if you're building with Node.js or TypeScript.”
“Works with the Vercel AI SDK and existing OpenAI SDK integrations.”
“Supports hundreds of models.”
“Offers automatic provider routing and fallback.”
“Provides unified billing and observability.”
“Can often be adopted by changing the API base URL or model identifier.”
“MLflow AI Gateway | Self-hosted / platform-managed | Unified interface, traffic splitting, failover | ML platform teams”
“Unified interface, traffic splitting, failover”
“Good if you're already using MLflow for your machine-learning infrastructure.”
“Azure API Management AI Gateway | Managed | Centralized policies, security, multiple model backends | Organizations already using Azure”
“Centralized policies, security, multiple model backends”