Which model ran which bench, and what is still missing

The coverage picture

Nominally the matrix is nearly full: 45 of 48 model × bench cells have a run. Usably, only 3 of the 8 benches have a complete, apples-to-apples set of all 6 models. The blockers are not missing runs, they are config drift: the grader model changes between benches, and epochs changes between models on the same bench.

Six models were evaluated: GPT-5.5, Claude Opus 4.7, DeepSeek-V4-Pro, PLaMo-3.0-Prime, MedGemma-27B-text-it and MedGemma-4B-it. Eight benches: HealthBench full, consensus and hard, plus HealthBench Professional whole-set and its four use-case slices (consult, writing, research, red-teaming). 68 logs across 10 HuggingFace spaces.

The good news for a comprehensive story: only 5 runs are strictly required to close every real gap, and 3 of those 5 are grader-only passes that reuse cached model responses, so they are cheap. One further decision (what to do about a half-finished gpt-4.1 re-grade) is worth 8 more runs or a deletion, your call.

Model × bench matrix

Scores are the length-adjusted HealthBench score × 100, which is the headline metric OpenAI reports. The last four columns are the Professional use-case slices. Each column header carries its judge model (4.1 = openai/gpt-4.1, 4o-mini = openai/gpt-4o-mini, 5.4 = openai/gpt-5.4) and sample count: columns are internally consistent but not comparable to each other.

in the comparable set for that column   present but off-config   no run exists
* ran at epochs=1 where the rest of the column ran epochs=8  ·  c score comes from a full cache replay (no model generation)  ·  p mostly cached (<200 candidate tokens per sample)

Reading the cells

The c and p markers matter less than they look. Caching a model's response and re-grading it is legitimate reuse: the response does not depend on the judge. What the markers tell you is that the number cannot be independently reproduced from the published log alone, because the generation happened in some earlier run. 27 of 68 logs are full replays; only 23 generated fresh candidate responses.

The * marker is the one that breaks a table. DeepSeek's Professional runs used epochs=1 (525 samples) while every other model used epochs=8 (4200 samples). More epochs shrinks per-sample judge noise, so DeepSeek's 31.0 carries visibly more error than GPT-5.5's 47.8 next to it.

What is already complete

Three benches have all six models under one judge and one epoch setting. These you can publish today without caveats beyond the usual judge disclosure.

Model rankings are stable across every bench, which is the reassuring part: GPT-5.5 and Opus 4.7 lead and are within noise of each other, DeepSeek is third, then MedGemma-27B and PLaMo trade places depending on the bench, and MedGemma-4B is last everywhere. Whatever you fix in the gaps below is very unlikely to reorder the leaderboard.

What to run next

5 required runs

These five close every genuine hole in the matrix. Ordered by how much they buy you.

inspect eval healthbench_professional.py \
  --model openrouter/deepseek/deepseek-v4-pro \
  --epochs 8 \
  -T judge_model=openai/gpt-5.4 -T judge_reasoning_effort=low

for uc in writing research red_teaming; do
  inspect eval healthbench_professional.py@healthbench_professional_$uc \
    --model vllm/google/medgemma-4b-it --epochs 8 \
    -T judge_model=openai/gpt-5.4 -T judge_reasoning_effort=low
done

The gpt-4.1 decision

On 2026-08-05 the two MedGemma models were re-run on hard and consensus with the judge switched from gpt-4o-mini to gpt-4.1. Nothing else was re-run. That leaves a half-populated second column on both benches: 2 of 6 models have a gpt-4.1 number, 6 of 6 have a gpt-4o-mini number.

Two clean options, and you should pick one explicitly rather than let both sit in the folder:

The candidate responses for all 8 of those are already cached from July, so option 2 needs no model generation at all, only judge spend.

Slices with no clean data at all

Beyond the four use-case slices, HealthBench Professional carries three more breakdown axes: difficulty (2 levels), source slice (3), and specialty (36). None of these has a usable number for any model, because they exist only as in-log metrics and those metrics are computed wrongly.

The in-log subset metrics (use_case_*_score, type_*_score, difficulty_*_score, source_slice_*_score, specialty_*_score) discard the length adjustment and clip each sample to [0, 1] before averaging. Comparing them against the standalone slice runs in hs-prof-subsets shows how far off that is:

Every error is positive, and the worst cases are the slices where models score near zero, which is exactly where per-sample clipping does the most damage. The four use-case slices are safe only because Ajay built hs-prof-subsets to recompute them as standalone task runs. Difficulty, source slice and specialty have no equivalent, so if you need those breakdowns they must be re-aggregated from per-sample scores, or run as standalone filtered tasks the way the use-case slices were.

Why cells disagree

Three sources of drift, in descending order of how much they move a number.

                          gpt-4o-mini   gpt-4.1     delta
consensus, MedGemma-27B      0.7758     0.9105     +13.5
full, GPT-5.5                0.4873     0.5687      +8.1
full, Opus-4.7               0.4762     0.5344      +5.8
consensus, MedGemma-4B       0.7137     0.7583      +4.5
hard, MedGemma-27B           0.2109     0.1415      -6.9
hard, MedGemma-4B            0.1060    -0.0347     -14.1

How big does a gap have to be to matter

Bootstrap standard errors from the logs, so you can tell a real difference from resampling noise before commissioning a re-run:

This is why GPT-5.5 at 47.8 and Opus 4.7 at 48.0 on Professional should be reported as tied, and why the judge-swap effects above (up to 14 points) are unambiguously real rather than noise.

One footnote: the Aug-05 MedGemma-4B hard runs report a standard error of exactly 0.0. That is a clipping artifact, not precision. The bootstrap clamps the mean to [0, 1], and since that run's mean is negative every resample clamps to 0, collapsing the spread. Treat the std error as unknown for any run with a negative mean.

Where everything lives

All ten HealthBench spaces are mirrored locally, 68 logs. healthbench-v1 is a symlink into the inspect_ai sandbox where it already lived.

Per-run detail with judge, epochs, token counts and a provenance classification is in hb-spaces/MANIFEST.csv, one row per log. hb-spaces/INDEX.md carries the trap list. On HuggingFace, healthbench-profes and healthbench-prof-part1 exist but hold no logs, so there is nothing to mirror.

Questions this raises

If I only have budget for one thing, what is it?

The DeepSeek Professional epochs=8 run. It is the single cell that breaks the Professional table, and Professional is the bench with the physician baseline, so it is the one you would actually put in front of someone.

Can I merge the four use-case slices back into a whole-set Professional score?

No. 236 + 142 + 147 + 191 = 716 against 525 total samples, so samples carry multiple use-case tags and the slices overlap. Use the whole-set run for the headline and the slices only as breakdowns.

Are the two MedGemma hard spaces duplicates?

Effectively yes. medgemma-healthbench-hard and healthbench-hard-medgemma-v2 are the same experiment run two hours apart, differing only in whether the model was addressed as openai-api/vllm/google/… or vllm/google/…. The 0.8-point spread between them is vLLM sampling noise, which incidentally gives you a free repeatability estimate for the local models.

Was gpt-5-nano ever evaluated?

One attempt on 2026-07-09, which died on a 401. It is not part of the model set.

Why do so many runs show zero token usage?

cache=true was set on every run, and Inspect does not count cache hits toward stats.model_usage. An empty model_usage means every call was served from cache, which is the fastest way to spot a replay masquerading as a re-run.