LLM Inference Gateway
OpenAI-compatible gateway with provider routing, streaming, auth, rate limits, and usage accounting.
Problem
Apps talking to many model providers need one API, failover, and cost visibility without rewriting every client.
Overview
A gateway that presents a stable chat/completions API while routing to configured providers, tracking usage, and failing over on timeouts/errors.
Solution
Terminate auth and rate limits at the edge, stream tokens to clients, and record usage for cost attribution.
Architecture
Client → gateway (API key) → rate limit → route policy → provider adapter → stream back → async usage write.
- Client
- Gateway
- Auth + limits
- Router
- Provider A
- Provider B
- Usage DB
Implementation
- OpenAI-compatible request/response shapes.
- Adapter interface per provider.
- Timeouts, retries, and fallback provider list.
- Usage rows: model, tokens in/out, estimated cost, key id.
Data model
api_keys, usage_events(model, prompt_tokens, completion_tokens, cost_estimate, latency_ms, status).
API design
POST /v1/chat/completions (stream optional). Admin routes for keys and routing policies.
Tradeoffs
- Passthrough streaming simplicity vs deep response inspection.
- Central gateway control vs per-service direct provider SDKs.
Challenges
- Correct SSE framing across providers.
- Fair rate limiting under bursty traffic.
- Accurate cost estimates as pricing changes.
Reliability
- Circuit breakers / failover on provider errors.
- Hard timeouts to protect callers.
Security
- API keys hashed at rest.
- No logging of full prompts in default config.
Performance
p50/p99 latency should be measured against each provider under a fixed prompt suite — not fabricated.
What I learned
- A gateway is valuable when you need policy, accounting, and failover in one place.
- Streaming bugs show up only under real token bursts.
Highlights
- Provider routing + fallback
- Token & cost accounting
- Per-key rate limits
- Streaming passthrough