HealthBench eval logs: 68 runs, 6 models, 8 benches

Start here

Every HealthBench run we have, as Inspect .eval logs, in one place. Six models (GPT-5.5, Claude Opus 4.7, DeepSeek-V4-Pro, PLaMo-3.0-Prime, MedGemma-27B, MedGemma-4B) across HealthBench full, consensus, hard, and Professional with its four use-case slices.

To pull a single log straight down, use the .eval button in the tables below, or fetch it directly:

Filenames carry the config, so professional__gpt-5.5__2026-07-24.eval needs no lookup. A -replay or -cached suffix means the model responses came from Inspect's cache rather than a fresh generation; -FAILED means the run errored and is kept only for provenance.

All 68 logs

Scores are ×100. raw is the HealthBench score, adj is the length-adjusted one. Grouped by bench, sorted by model.

HealthBench full 10 logs

5000 samples, the whole open-ended set

HealthBench consensus 8 logs

3671 samples, criteria physicians agreed on

HealthBench hard 13 logs

1000 samples, the hardest slice

HealthBench Professional 7 logs

525 samples, physician-written, has a human baseline

Professional: consult 6 logs

236 samples

Professional: writing 6 logs

142 samples

Professional: research 6 logs

147 samples

Professional: red-teaming 6 logs

191 samples

Professional: physician baseline 6 logs

the human reference, model-independent

fresh model responses generated in this run   replay every call served from cache   cached under 200 candidate tokens per sample   baseline human responses, no generation by design   failed errored or cancelled
? judge not recorded in task_args, inferred from the scorer default and confirmed against stats.model_usage

Analysis

Before you quote a number

Four things will bite you if you take a score straight out of a log.

How this was assembled

The runs were executed by Ajay between 2026-07-09 and 2026-08-06 and originally published as ten separate HuggingFace Spaces under ajay-citadel. This Space consolidates all of them into one viewer, renames the logs so the config is legible from the filename, and adds the provenance classification that the raw logs do not carry.

data/log_mapping.csv maps every renamed file back to its original space and filename, so nothing here is a dead end. data/MANIFEST.csv is the full per-run header dump: judge, epochs, token counts, package versions. data/INDEX.md is the short version of the traps list.

The original spaces remain the upstream source. If a number here disagrees with one there, the logs are byte-identical, so the difference is in which run you are reading, not in the data.