Comparing three extraction configurations

We ran three configurations of the Claude Opus 5.5 pipeline on the same 12 papers: 7 research articles, 2 case reports (one in Spanish), 2 reviews and 1 brief report, totaling 463k characters. The papers were unseen by the pipeline, and each configuration's rules and settings were fixed before its run.

These papers have no expert gold standard. We compare cost, output coverage and status agreement; accuracy still needs manual evaluation.

The 12 benchmark papers

Result

Cost and time per run of 12 papers

Bars show totals including repairs; the detailed table also reports first-request costs. V1 ran before two parser fixes and needed extra repairs, so totals include that implementation difference.

What each variant finds

Agreement with the baseline

Supported relations per paper

For each paper, one bar per variant: the number of gene–disease relations that variant judged supported. All bars share one scale. Hover a bar for its uncertain and explicit-negative counts too.

Full table

Per-variant comparison with PubTator3