BioRED gene–disease extraction results
Results by BioRED's original splits
We used BioRED's original Train / Dev / Test splits, without training or prompt tuning on them. We have not checked whether the foundation model saw BioRED during pretraining. The confidence intervals overlap; equivalence across splits remains untested.
Strict setting; whiskers are bootstrap 95% intervals. Dev and Test have about 70 abstracts each, so their intervals are wider.
Head to head with PubTator3 (50-abstract test sample)
PubTator3's public abstract annotations for the frozen 50-abstract relation-positive test sample, aligned to the same text and scored by the same script and rules. PubTator3 uses BioRED in training; the exact training split of the deployed models was not verified. Both systems' numbers in this section are for these same 50 abstracts.
Metric rows share a 0–1 scale; count rows are scaled to their larger value. For our method, recall and F1 are shown strict and with pairs we flagged uncertain counted as predictions.
The categories include non-supported judgments and differences in which entities count as diseases, such as symptoms and broad terms ("tumor", "infection"). Each disagreement still needs review.
Precision, recall and F1
Whiskers: bootstrap 95% intervals (2,000 resamples of abstracts). Strict: every gold relation counts, only supported pairs are predictions. Scope-adjusted: gold relations whose "disease" our rules record as a symptom/phenotype are set aside. Supported + uncertain: pairs we flagged uncertain also count as predictions.
Matched, missed and additional relations
Missed and additional relations by category
This study vs background: agreement with BioRED's novelty label
BioRED labels most gene–disease relations Association; our rules use association / correlation / causal. Not scored as a metric.
Selected examples
These examples were selected by hand to show different outcomes. They are not a random sample.