LLM observability is where SRE was in 2015 — every vendor is racing, every team has cobbled together an internal solution, and there is no canonical reference architecture. OpenTelemetry ships GenAI semantic conventions. Datadog, Honeycomb, and New Relic all have LLM observability products. But almost nothing exists on how to pick between them, how to sample high-token workloads without blowing your storage bill, or how to link traces to eval results.
This hub curates skills for the SRE, platform engineer, or applied AI engineer who now owns an LLM feature and needs to see what it is doing. Rolling your own OTel-based agent tracing when you cannot lock in a vendor, sampling strategies that keep interesting traces without paying for boring ones, cost attribution per user and per prompt, and the linking pattern that lets you jump from a bad production trace directly to an eval regression suite. Framework-agnostic and vendor-honest.
Who this hub is for
SREs, platform engineers, and applied AI engineers responsible for LLM feature reliability and cost.
LLM observability tracks the same request/latency/error metrics as regular APM, plus LLM-specific dimensions: prompt content, completion content, token counts, model name, temperature, tool calls, and eval scores. A regular APM span tells you a request took 800ms. An LLM observability span tells you the request took 800ms across 3 tool calls and used 4,200 input tokens plus 1,100 output tokens on gpt-4o.
Use OTel if you care about vendor portability, want to send the same trace to multiple destinations, or already run an OTel Collector. Use a vendor SDK (Langfuse, Braintrust, LangSmith) if you want turnkey tracing with prompt playback and pairwise-diff UI without setup. Sophisticated teams do both — instrument with OTel semantic conventions, ship to a vendor via OTLP.
Tail-based sampling on the OTel Collector: capture 100% of traces where duration > p95 or where an error occurred or where an eval score is below threshold; capture 5-10% of everything else. This costs a fraction of full retention while preserving every trace worth investigating. LangSmith and Braintrust have similar sampling primitives built in.
Rule of thumb: 2-5% of inference spend goes to observability. On a $50K/month inference bill, $1-2.5K/month for tracing plus storage is a defensible budget. If you are over 10%, your sampling strategy is wrong. If you are under 1%, you are probably flying blind on regressions.
Yes, this is the modern win. Tag every production trace with a stable prompt-template ID. When your eval suite regresses on that template, jump directly from the eval failure to the production traces that used the same template. LangSmith and Braintrust both support this natively. On OTel, add a custom attribute `genai.prompt.template.id` and index it in your backend.
Every LLM trace stores the prompt and completion, which almost certainly contains PII. Redact before writing: use a redaction proxy on the OTel Collector, or a vendor with built-in redaction (Langfuse, Arize both support it). At minimum redact email addresses, phone numbers, and any known PII field names before persist. HIPAA and GDPR both apply — a trace store is a data store.