API Observability Lite
Ingest request traces/logs, index for search, and surface latency/error dashboards for service owners.
Problem
Without a shared view of latency and errors, teams debug from guesswork and scattered logs.
Overview
A compact observability design: accept OTLP or structured logs, store hot data for search, and expose service-level latency/error views.
Solution
Separate ingest from query. Keep recent high-cardinality data queryable; downsample or expire aggressively.
Architecture
Instrumented services → OTLP collector/ingest → write path → indexed store → query API → UI dashboards and alerts.
- Services
- OTLP
- Ingest API
- Hot store
- Dashboards
Implementation
- Ingest authentication and payload size limits.
- Normalize spans/logs to a common schema.
- Service overview: p50/p95, error rate, top routes.
- Trace detail view by trace id.
Data model
services, spans/events partitions by time, indexes on service+route+status.
API design
POST /v1/traces, GET /v1/services, GET /v1/traces/:id, GET /v1/metrics/summary.
Tradeoffs
- Full Elasticsearch vs cheaper columnar store.
- Cardinality limits vs developer freedom.
Challenges
- Hot cardinality explosions from unbounded labels.
- Keeping ingest cheap under burst.
Reliability
- Buffering on ingest failure.
- Backpressure when store lags.
Security
- Tenant isolation on ingest tokens.
- PII scrubbing hooks before index.
Performance
Query SLOs should be set after indexing real traffic samples — not guessed.
What I learned
- Observability products fail first on cost and cardinality, not on charts.
- A few good default dashboards beat infinite customization.
Highlights
- OTLP ingest
- Service health views
- Error grouping
- Retention policies