evaluation infrastructure · agent reliability · 2026

EvalMedic: Diagnosing Evaluation Regressions After the Red Metric

A diagnosis-and-decision layer above existing LLM and agent evaluation stacks. EvalMedic asks what changed, why the metric moved, whether the measurement is still trustworthy, and what evidence to collect next.

github repo system design validation plan

V0.1 design complete · implementation in progress

The problem after evaluation

Modern evaluation tools are increasingly good at running suites, logging traces, scoring outputs, comparing experiments, and showing that a metric changed. The difficult work often begins after that dashboard turns red.

Did the model regress, or did the prompt, tool, runtime, data, harness, rubric, or grader change the measurement?

An evaluation engineer must usually answer this through manual trace inspection and ad hoc reruns. EvalMedic turns that second-order debugging work into versioned measurement metadata, controlled replay experiments, explicit evidence, and calibrated abstention.

Positioning: EvalMedic is not a new general-purpose eval runner and does not replace Inspect AI, LangSmith, Braintrust, OpenAI Evals, or internal infrastructure. It consumes their artifacts and focuses on the decisions they leave to engineers.

Four maintenance questions

01 · plan

Adaptive Eval Planner

What is the smallest informative subset to run for this release, and when is more evidence required?

02 · diagnose

Counterfactual Regression Triage

Does the failure persist when one component is frozen or swapped? Rank supported root-cause hypotheses and abstain when interactions prevent identification.

03 · calibrate

Evaluator Drift Calibration

When a judge, rubric, or grader prompt changes, are historical and current scores still comparable across important slices?

04 · maintain

Benchmark Maintenance

Should a production failure be promoted, merged, retained as a diagnostic, or retired without allowing the suite to bloat?

Architecture: an overlay, not a replacement

Existing eval stackruns · traces · scores · production failures

Versioned measurement manifestmodel · prompt · tools · runtime · data · evaluator · cost

EvalMedic evidenceranked causes · drift warnings · next tests · human escalation

The key design requirement is that every run carries a measurement manifest. Without explicit versions for the model, prompt/scaffold, tools, runtime, data, harness, and evaluator, root-cause attribution is often impossible.

V0.1: controlled diagnosis before broad claims

The first release deliberately narrows the system to the parts with clear ground truth: a canonical schema, controlled perturbations, counterfactual regression triage, and grader-drift checks. The initial claim is not that EvalMedic solves causal debugging in general. It is:

Given a known controlled perturbation, can EvalMedic identify the responsible component—or abstain when the evidence is ambiguous?

30deterministic tasks across tool calling, SQL, and file/code

5controlled release conditions, R0 through R4

≈300planned trajectories with two repeats

Design targets above are planned validation scope, not reported experimental results.

Condition	Controlled change	Known cause	Diagnostic question
R0	Baseline configuration	None	What does stable measurement look like?
R1	Swap model/checkpoint only	Model	Does the failure persist with the old surrounding stack?
R2	Alter system prompt/scaffold only	Prompt	Does the baseline model now reproduce the regression?
R3	Change tool schema or inject runtime errors	Tool/runtime	Do recorded old outputs recover performance?
R4	Change rubric/grader and re-score frozen traces	Evaluator	Is apparent movement measurement drift?

What success and failure look like

Root-cause quality: top-1 accuracy, top-k recall, and regression recall on known perturbations.
Honest uncertainty: abstention rate and calibration when component interactions make causes non-identifiable.
Engineering value: diagnosis latency and number of replay experiments compared with manual investigation.
Falsification: narrow or stop the project if it does not reduce effort beyond competent analysis of existing traces.

Counterfactual triage

EvalMedic freezes most of the measurement system, swaps one component at a time, and records whether the regression recovers. The output is an evidence profile rather than a falsely precise causal probability.

candidate model + old prompt/tools/runtime/grader
  → failure persists? model change becomes more plausible

baseline model + new prompt
  → baseline now fails? prompt/scaffold becomes more plausible

same model/prompt + recorded old tool outputs
  → performance recovers? tool/runtime contribution becomes more plausible

same frozen traces + old versus new grader
  → scores recover? evaluator drift becomes more plausible

interactions remain unresolved
  → abstain and request the next discriminating replay

One-at-a-time replay is not guaranteed causal identification. External state, nondeterminism, and component interactions can make faithful reconstruction impossible. The system must expose those limitations rather than hide them behind a root-cause score.

Developer surface

The first public repository focuses on typed measurement schemas, reproducible controlled-release fixtures, a minimal triage engine, tests, and a transparent roadmap. The intended interface is small:

evalmedic diagnose runs/release-v12.json
evalmedic calibrate-grader --old J3 --new J4 --anchors frozen.json
evalmedic plan --baseline v11 --candidate v12

Inspect AI is the first planned adapter. LangSmith, Braintrust, custom adapters, CI integration, and an evidence UI follow only after the core diagnosis algorithms prove useful.

browse the public repository

Build sequence

Release	Scope	Claim to earn
V0.1	Canonical schema, controlled perturbations, triage, grader drift	Known changes can be diagnosed or safely abstained on.
V0.2	Adaptive subset selection and sequential stopping	Regression recall can be preserved with lower evaluation cost.
V0.3	Failure clustering and benchmark promotion/merge/retire	Production failures improve coverage without uncontrolled suite growth.
V1.0	Additional adapters, CI integration, evidence UI	The layer is useful beyond a controlled research prototype.

What this project is testing

The broader research idea is that an evaluation stack is itself a measurement instrument. It must be versioned, calibrated, diagnosed, and maintained—not merely executed.

EvalMedic will be presented first as an engineering system. Claims of academic novelty will wait for a dedicated related-work review and empirical results. The public roadmap keeps proposals, implemented functionality, and measured findings visibly separate.

← back to selected research