HealthBench config check: our spaces vs OpenAI's published numbers

We have a real anchor for this check: OpenAI's GPT-5.6 system card publishes gpt-5.5 scores for all four HealthBench variants, and gpt-5.5 is one of the models we ran. Lining our numbers up against theirs gives a per-variant verdict rather than a guess.

The result splits cleanly. The two spaces where we used the correct grader reproduce OpenAI within a point. The full HealthBench run lands at 55.8 against their 56.5, and our HealthBench Professional physician baseline lands at 43.87 against their published 43.7. That is close enough to say the dataset, prompting, rubric scoring, length adjustment and aggregation are all wired up right.

The two spaces where we used the wrong grader are badly off. HealthBench Hard is 5.5 points low and HealthBench Consensus is 13.6 points low. Both were graded by gpt-4o-mini instead of GPT-4.1, not by choice but because the inspect_evals task wrappers for those two variants do not expose a judge_model argument and silently fall back to the package default. Consensus is the loudest signal: every frontier model in OpenAI's table sits in a 94 to 96 band, and ours sits at 82.

A separate, smaller gap shows up on the inference side. Our gpt-5.5 Professional score is 4 points below OpenAI's, even though the grader is correct there. The physician baseline matching to 0.2 points rules out the grading pipeline, which points the finger at reasoning effort: we pass none, while OpenAI evaluates "at the highest reasoning effort option available via each model's API." Claude was run with extended thinking off entirely.

v2 update: the DeepSeek Professional re-run

Ajay re-ran HealthBench Professional for DeepSeek on 2026-08-06 and published it as its own space, ajay-citadel/healthbench-professional-deepseek-v4-pro. It is the first run that addresses anything on the what to change list, so this version of the report folds it in and corrects the numbers it supersedes. Everything else in v1 stands unchanged.

The re-run is a single combined task over all 525 examples rather than five separate subset tasks, which is what OpenAI's reference implementation does. That removes the manual sample-weighting v1 had to apply, and it gives DeepSeek a genuine consult run for the first time.

The v1 sample-weighted arithmetic checks out against the native number: (236×31.83 + 142×6.70 + 147×53.10) / 525 = 30.99, exactly the re-run's own bootstrap_score. So the stitching method v1 used was sound; the 3.3-point move comes entirely from the consult subset being re-generated rather than replayed. DeepSeek does not change rank — it stays third behind claude-opus-4-7 (48.02) and gpt-5.5 (47.81).

What the re-run fixed, and what it did not

reported bootstrap_score        0.3099   # length-adjusted
recomputed mean(adjusted)       0.3099   # match
recomputed mean(unadjusted)     0.3781   # matches the log's own mean metric
bootstrap 1000x, own resample   0.3104 +/- 0.0283   # log: 0.3099 +/- 0.0279
criteria_met_rate               0.5921   # match; 1135 criteria over 525 samples

length adjustment: adj = raw - 0.0147 * ((chars - 2000) / 500)
  mismatches over 525 samples   0        # arithmetic verified per sample

served model      deepseek/deepseek-v4-pro     # matches requested, no silent substitution
judge served      gpt-5.4-2026-03-05           # 1135 calls, one per rubric criterion
judge config      {system_message: "You are a helpful assistant.", temperature: 0.0,
                   reasoning_effort: "low"}    # matches OpenAI's internal reference
candidate config  {cache: true}                # no reasoning effort passed

rubric coverage   1135 graded / 1135 in dataset metadata   # no criteria dropped
unique prompt ids 525 / 525
empty completions 0
mean chars        4320   median 4053   p99 11620   max 17982
frac. samples with length-adjusted score < 0   0.303

The anchor check

All numbers are the length-adjusted score on the 0 to 100 scale, which is what OpenAI reports as the primary metric. Their figures come from the GPT-5.6 system card Table 6, given as length-adjusted (unadjusted, mean chars). Ours are recomputed from the downloaded .eval logs.

Our Professional overall is a sample-weighted mean of the three use-case subsets (consult 236, writing 142, research 147, total 525), since we ran them as separate tasks rather than one benchmark. Red teaming is a cross-cutting slice of those same 525 examples, not a fourth use case, so it is excluded from the weighting.

Two things worth noticing beyond the scores. First, mean response lengths track closely (2175 vs 2313, 2158 vs 2289, 2143 vs 2259, 3748 vs 3818), consistently about 5% shorter but never structurally different. Since length is the one output property that would move if our prompting or harness diverged, this is good evidence the request side is faithful. Second, the two correct grader rows and the two wrong grader rows separate perfectly, which is what makes the diagnosis confident rather than speculative.

The physician baseline is the cleanest check we have

HealthBench Professional ships a set of 525 physician-written responses. Scoring them involves no model inference at all: the responses are fixed text, and the only moving part is the grader plus the aggregation. So if our number matches OpenAI's, the entire scoring half of the pipeline is verified, and any remaining discrepancy on real models has to live on the inference side.

Every cell is within a point. Note that the per-use-case rows here are recomputed from per-sample scores, not read off the log's own subset metrics, which are inflated for a reason covered in issue 4.

Config matrix

What we set against what OpenAI's reference implementation sets, per variant.

v2: the re-run column is DeepSeek only. It closes the task-shape and explicit-grader gaps and leaves the two that matter most for score comparability — samples per example and reasoning effort — exactly where they were.

The length adjustment is worth calling out as correct-by-verification, not just correct-by-declaration: we confirmed the arithmetic per sample against score − penalty × ((len − center) / 500), OpenAI's calculate_length_adjusted_score. On one Hard sample: raw 0.6102, completion 13,776 chars, giving 0.6102 − 0.0392 × 23.552 = −0.3131, which is exactly the stored value.

Issue 1: Hard and Consensus were graded by gpt-4o-mini

This is the largest error and it was not a deliberate choice. The inspect_evals task wrappers for these two variants accept only the length-adjustment arguments and pass nothing else through, so judge_model silently keeps the package default of openai/gpt-4o-mini:

def healthbench(
    ...
    judge_model: str | Model = "openai/gpt-4o-mini",   # package default
    ...
)

@task
def healthbench_hard(
    length_adjustment_center: float | None = None,
    length_adjustment_penalty_per_500_chars: float | None = None,
) -> Task:
    return healthbench(                      # judge_model never forwarded
        subset="hard",
        length_adjustment_center=length_adjustment_center,
        length_adjustment_penalty_per_500_chars=length_adjustment_penalty_per_500_chars,
    )

OpenAI never grades any variant with a mini-tier model. In simple_evals.py the same healthbench_grading_sampler is handed to healthbench, healthbench_hard and healthbench_consensus alike, so all three get GPT-4.1, or all three get GPT-5.4 low when the --healthbench-use-gpt-5-4-low-grader flag is set.

The HealthBench paper measured what a weaker grader costs. Its meta-evaluation ranks candidate graders by agreement with physicians (Macro-F1): GPT-4.1 0.709, o4-mini 0.692, o3 0.681, GPT-4.1 mini 0.661, GPT-4.1 nano 0.580, and calls the smaller variants "substantially worse". gpt-4o-mini is not in that table but sits below GPT-4.1 mini in capability, so 0.661 is the optimistic bound.

Issue 2: reasoning effort is never set, and the models diverge because of it

The HealthBench Professional paper states that models are "evaluated at the highest reasoning effort option available via each model's API (e.g., xhigh for GPT-5.4)", and measures a 5.6 to 7.3 point gain moving from low to xhigh. None of our runs pass a reasoning effort, so each provider's default applies, and the defaults are not comparable to each other:

Claude ran with extended thinking off on every sample of every log, while gpt-5.5 and DeepSeek reasoned by default. Whatever the intended comparison was, this is not it: the reasoning models get their default budget and Claude gets none. Claude is the most disadvantaged model in the set for a reason that has nothing to do with Claude.

This also explains the one gap the grader cannot account for. On Professional, where our grader is correct and the physician baseline matches to 0.2 points, gpt-5.5 still lands 4 points low. A default-effort versus highest-effort difference of that size is consistent with the 5.6 to 7.3 point low-to-xhigh delta OpenAI reports.

Issue 3: three spaces show the unadjusted score as the headline

OpenAI reports "length-adjusted score (unadjusted, mean response length in characters)", so the adjusted number is the primary one. In our listing.json the primary_metric for hs-non-professional, hs-hard and hs-consensus is the unadjusted bootstrap_score. Only hs-prof-subsets carries use_length_adjusted: true.

This is not cosmetic. Length adjustment reorders the Hard leaderboard almost completely, because our models differ enormously in verbosity:

Claude gains because it is the only model averaging under the 2000-character centre. medgemma-27b loses 16 points. Anyone reading the space's default metric is reading a different ranking from the one OpenAI's methodology produces.

Issue 4: the Professional per-use-case subscores are inflated

The custom healthbench_professional.py emits use_case_*_score, type_*_score and difficulty_*_score metrics alongside the headline. These do not use the same aggregation as the headline and should not be quoted.

The mean(clip(raw)) column matches the reported column exactly, to six decimals, on all ten subset metrics the log emits — there is no bootstrap noise in these numbers at all. Red teaming is the clearest illustration of the damage: the log presents it as 24.94 when the correctly aggregated value is −5.19, a 30-point swing that flips the slice from mediocre to negative.

The same error was present in the v1 spaces; it was simply harder to characterise there, because the physician-baseline responses are short enough that the adjusted and unadjusted scores nearly coincide.

The physician baseline makes the error measurable, because OpenAI publishes the ground truth for exactly these three cells:

overall mean raw                    = 44.29   # matches reported 44.29
overall mean adj                    = 43.87   # matches reported 43.87, and OpenAI's 43.7
overall mean adj, clipped per-sample= 49.71
frac. of samples with adj < 0       = 0.164

by use case          n      mean_adj   clipped_per_sample
  consult          1888      42.54          48.09
  research         1176      56.60          57.47
  writing          1136      32.91          44.39

The headline bootstrap_score is unaffected and remains correct. Only the subset breakdowns are wrong, which matters because the by-use-case split is the most quoted view of Professional results.

Issue 5: five Professional scores are floored at zero

OpenAI clips the aggregate mean to [0, 1], and our implementation follows suit, so this is faithful behaviour rather than a bug. But it means five reported numbers are all displayed as 0.00 while their true values differ by 7 points, which hides real ranking information:

Worth reporting the unclipped value alongside the clipped one for these, or at least noting that a zero means "at or below zero" rather than "scored nothing".

Run hygiene

What to change

Concretely, for the two broken variants, stop using the wrapper tasks and call the parent task instead:

# broken: judge_model silently defaults to gpt-4o-mini
inspect eval inspect_evals/healthbench_hard \
  -T length_adjustment_center=2000 \
  -T length_adjustment_penalty_per_500_chars=0.0392

# works: call healthbench() directly with subset=
inspect eval inspect_evals/healthbench \
  -T subset=hard \
  -T judge_model=openai/gpt-4.1 \
  -T length_adjustment_center=2000 \
  -T length_adjustment_penalty_per_500_chars=0.0392

Re-running with the correct grader and highest reasoning effort should move Hard up by roughly 5 points and Consensus up by roughly 13 toward the published band. If it does not, that is the signal something else is wrong. If it does, the config is settled.

Full results

Main variants, all six models

Judge: gpt-4.1 for full, gpt-4o-mini for hard and consensus. Epochs 1 throughout. All runs completed with zero errors. Only the full column is grader-comparable to the published row.

Professional subsets, length-adjusted

Red teaming values are shown unclipped so the ordering is visible; the logs report these as 0.00. Weighted overall uses consult 236, writing 142, research 147. Only gpt-5.5 and claude-opus-4-7 beat the physician baseline overall, and both do so narrowly.

v2: the DeepSeek row now comes from the 2026-08-06 single-task re-run, whose 30.99 is a native overall rather than a weighted stitch. Its subset figures are recomputed from per-sample data, not read off the log's own subset metrics, which are inflated — see issue 4. The superseded v1 row is kept for traceability: its consult and writing figures came from the 2026-07-25 cache replay, and its research and red-teaming figures from the 2026-07-24 epochs=8 runs. Every other model's row is still epochs=8 and unchanged, so DeepSeek is the only single-sample row in this table.

How this was checked

The four HuggingFace spaces were cloned and their .eval logs pulled as real LFS blobs (the plain clone yields 133-byte pointers). Headers gave the declared config; per-sample model events gave the config actually sent, including which model served every grading call; per-sample scores were re-aggregated independently to confirm the reported metrics and to recompute the subset breakdowns.

Local copies at /Users/kazuki/Sandbox/hb-spaces/. The hs-original space was excluded, its link is broken. The v2 space's .eval was fetched over the HuggingFace resolve/main endpoint rather than git lfs pull, since git-lfs is not installed on this machine; the file is a valid zstd-compressed Inspect log and reads cleanly. Reference sources: HealthBench (arXiv:2505.08775), HealthBench Professional (arXiv:2604.27470), the GPT-5.6 Preview system card Table 6, and openai/simple-evals.