Six distinct postings on HN "Who is hiring? July 2026" name-check "evals" as a core competency. Anthropic, OpenAI applied, and Scale AI hire evaluations engineers as a dedicated role at $200-350K. And yet the tutorial ecosystem for LLM evaluation is stuck on RAGAS hello-worlds and promptfoo YAML snippets from 2024. Nothing serious on judge-model calibration against human labels, pairwise preference eval design for agents, regression suites for tool-calling, or dataset curation from production traces.
If you ship any LLM feature to real users, you have already been burned by vibes-based QA — a prompt tweak that fixed one case and silently broke five others, an upstream model swap that regressed on your critical path, a subtle drift you only noticed when a customer complained. The skills below curate the actual production-grade patterns: how to build a judge model that agrees with human labels 90%+ of the time, how to run a regression suite in CI without spending $500 per pull request, how to sample the right traces from prod for offline eval, and when to use Braintrust vs Langfuse vs LangSmith vs your own harness. This is the reading list for the applied AI engineer who has decided "evals as first-class artifacts" is not optional anymore.
Who this hub is for
Applied AI engineers, ML engineers, and staff engineers shipping LLM features at Fortune 500 companies and AI-native scale-ups.
LLM evals are structured tests that measure the quality, safety, and consistency of your language model outputs. Unlike unit tests that check exact values, evals score fuzzy properties: is this response helpful, is it factually accurate, does it follow the system prompt. Without evals you cannot ship an LLM feature confidently — every prompt tweak or model upgrade risks silent regression. Evals turn "vibes-based" QA into a real engineering practice.
Yes, with calibration. A GPT-4-class judge model agrees with human labelers 85-92% of the time on well-defined tasks (factuality, helpfulness, format compliance) when the rubric is precise and the judge sees pairwise comparisons rather than absolute scores. You calibrate by labeling 200-500 examples yourself, running the judge on the same set, computing inter-annotator agreement, and iterating the rubric until judge-human agreement exceeds 85%.
LangSmith wins for LangChain / LangGraph users with tight integration and best trace UI. Braintrust wins for teams that want a dedicated evaluation-first product with excellent regression suite tooling. Langfuse wins for self-hosted / open-source-preferred stacks and cost-sensitive teams. All three are viable. If your team is not committed, start with Braintrust for greenfield evals, Langfuse for cost, LangSmith for LangChain-native.
A reasonable engineering norm: 5-10% of total inference spend goes to evals. If you spend $50K/month on production LLM calls, budget $2.5-5K/month for eval infrastructure and judge model calls. This buys you continuous regression suites in CI, weekly pairwise preference evals against production traces, and the ability to safely upgrade underlying models.
Record 200-500 real production traces where the agent behaved correctly. For each, extract the input state and the sequence of tool calls the agent made. In CI, replay each trace against your current agent, compare tool-call sequences (order, arguments, count), and flag deltas. This catches the two biggest agent regressions: calling the wrong tool, and calling the right tool with wrong arguments.
Yes — even more so. When you own the model, you can retrain to fix a bug; when you rent the model, silent upstream changes can regress your product overnight without warning. GPT-4o and Claude both quietly ship new model revisions monthly. Evals are your only defense — a CI check that fires when your prompt-plus-model combination degrades below your baseline.