Gene2Disease-LLM
How request design, input scope, and evaluation choices change what we conclude about LLM judgments of gene–disease relations.请求设计、输入范围和评估选择，怎样改变我们对大语言模型判断基因–疾病关系的结论。
Zeqian Tong · November 2026 · empirical study with two local models (Qwen3.5-9B, Gemma-4-12B) · manuscript in preparationZeqian Tong · 2026 年 11 月 · 用两个本地模型（Qwen3.5-9B、Gemma-4-12B）做的实证研究 · 稿件准备中
Four questions at a glance四个问题的回答
Prior work compared one-pair and many-pair requests on abstracts, with free triple generation or schema-enforced answers. This study asks whether the trade-offs hold on paragraph-level full-text passages when omissions and format failures are kept and reported.已有研究只在摘要上比较过"每次一个配对"和"每次多个配对"，而且用的是自由生成或强制完整回答的协议。本研究问的是：在段落级的全文上、保留并报告遗漏和格式失败时，这些取舍是否依然成立。
On BioRED abstracts, pairwise requests had higher recall and joint requests higher precision in both models. On BioTriplex full-text passages the recall difference was inconclusive (0.039, with room for a loss of up to about 0.10).BioRED 摘要上，两个模型都是 pairwise 召回更高、joint 精确率更高。BioTriplex 全文段落上，召回差不确定（0.039，最多可能约 0.10）。
Full text raised recall from 0.42–0.45 to 0.97. Abstract passages mention together only 73 of 154 gold pairs, so expanded coverage enabled most of the gain; this is a property of the input, not an LLM-specific advantage.读全文让召回从 0.42–0.45 升到 0.97。摘要段落只覆盖 154 个金标配对中的 73 个，所以增益主要来自覆盖范围；这是输入的性质，不是 LLM 特有的优势。
The reading of model output and the output format can change which request form looks better: under a line-based format, the Gemma-4-12B pairwise-versus-joint point estimate reversed on development data. A conclusion about request form is a conclusion about a protocol.对模型输出的读法和输出格式，会改变"哪种请求形式更好"：换成 lines 格式后，Gemma-4-12B 在开发集上 pairwise 对 joint 的点估计反转了。关于请求形式的结论，其实是关于整个协议的结论。
Every retained citation resolved, and 92–99% of gold-matched BioTriplex predictions cited a sentence that overlaps the gold relation trigger. Still, among passage-supported items the cited sentences did not support the relation in 10.7% (BioRED) and 14.9% (BioTriplex).保留的引用全部能解析，BioTriplex 命中金标的预测有 92–99% 引用的句子与金标的关系触发词重叠。但在段落确实支持的条目里，仍有 10.7%（BioRED）和 14.9%（BioTriplex）的引用句本身不支持这个关系。
Volume, agreement with the gold standard, citation validity, and textual support are reported separately throughout; none of them alone establishes factual correctness.全文把抽取数量、与金标的一致性、引用有效性和文本支持分开报告；单看其中任何一项，都不能说明事实上的正确性。
Design设计
The three request forms judge the same candidate pairs over the same paragraph-level passages, with the same instruction, JSON output, and output cap. Only the number of pairs per request differs.三种请求形式在同样的段落上判断同样的候选配对，指令、JSON 输出格式和输出上限都相同，只有每次请求列出的配对数不同。
Paragraph-level blocks; given genes and diseases; pairs that co-occur in a passage.段落级的块；实体给定；只判断在同一段落里共现的配对。
Pairwise (1 pair), chunked (≤5), joint (all pairs of the passage).pairwise（1 个）、chunked（≤5 个）、joint（段落里全部配对）。
Every assertion cites sentence IDs of its passage; truncations, parse failures, and omissions count as misses, with no repair.每条断言都引用本段落的句子编号；截断、解析失败和遗漏都算没有预测，不修复。
Paper-level micro recall (and precision on BioRED); paired bootstrap over papers; δ = 0.05.论文层面的 micro 召回（BioRED 加精确率）；按论文的配对 bootstrap；δ = 0.05。
Development, then one frozen test run先开发集，再冻结后测试一次
- All settings (output cap 1000 + 150 per pair, thinking off, prompt cache off, JSON) were chosen on BioRED Dev and BioTriplex train.所有设置（输出上限 1000 + 150×配对数、关闭思考、关闭缓存、JSON）都在 BioRED Dev 和 BioTriplex train 上选定。
- The protocol was frozen on 8 November 2026 and run once on BioRED Test (100 abstracts) and BioTriplex validation and test (30 articles): 12 runs, 8 comparisons.协议在 2026 年 11 月 8 日冻结，然后在 BioRED Test（100 篇摘要）和 BioTriplex val+test（30 篇全文）上只跑一次：12 个运行、8 个比较。
- Six primary contrasts (pairwise − joint recall on both corpora and precision on BioRED, per model) use Bonferroni-adjusted 99.17% intervals.6 个主要对比（每个模型：两个语料上的 pairwise − joint 召回，加 BioRED 精确率）用 Bonferroni 校正后的 99.17% 区间。
Frozen test results冻结后的测试集结果
Primary family: pairwise minus joint, canonical reading, 99.17% intervals. No run had a server failure, a context refusal, or a provider block.主要家族：pairwise − joint，规范化读法，99.17% 区间。所有运行都没有服务器故障、上下文拒绝或服务方拦截。
| Corpus语料 | Model模型 | Metric指标 | Pairwise / joint | Difference [99.17%]差 [99.17%] | Reading读法 |
|---|---|---|---|---|---|
| BioRED Test | Qwen3.5-9B | Recall召回 | 0.881 / 0.668 | +0.214 [+0.100, +0.323] | pairwise meaningfully higherpairwise 明显更高 |
| Precision精确率 | 0.306 / 0.407 | −0.102 [−0.229, −0.012] | joint higher, magnitude unresolvedjoint 更高，幅度不确定 | ||
| Gemma-4-12B | Recall召回 | 0.908 / 0.786 | +0.122 [+0.030, +0.254] | pairwise higher, magnitude unresolvedpairwise 更高，幅度不确定 | |
| Precision精确率 | 0.321 / 0.528 | −0.207 [−0.332, −0.108] | joint meaningfully higherjoint 明显更高 | ||
| BioTriplex | Qwen3.5-9B | Recall召回 | 0.968 / 0.929 | +0.039 [−0.021, +0.107] | inconclusive不确定 |
| val + test | Gemma-4-12B | Recall召回 | 0.968 / 0.929 | +0.039 [+0.000, +0.092] | inconclusive不确定 |
BioRED precision is measured against the corpus annotation; BioTriplex is scored for recall only because its relation annotation is not assumed to be exhaustive. Magnitudes differed between development and test data, so the two models are not ranked.BioRED 的精确率按语料标注计算；BioTriplex 不假设标注完整，只报召回。开发集和测试集上的幅度不一致，所以不给两个模型排高低。
Failures concentrate in large joint requests失败集中在大的 joint 请求里
Gemma-4-12B joint requests failed in 11 of 133 BioRED calls and 9 of 363 BioTriplex calls, all malformed JSON. Passages with more than 40 candidate pairs held 18.7% of BioRED gold exposures but only 2.45% on BioTriplex; joint requests listed 10.1 pairs on average on BioRED and 4.9 on BioTriplex.Gemma-4-12B 的 joint 请求在 BioRED 上 133 次调用失败 11 次，在 BioTriplex 上 363 次失败 9 次，全是 JSON 写坏了。超过 40 个配对的段落在 BioRED 上占金标暴露的 18.7%，在 BioTriplex 上只占 2.45%；joint 请求平均列出的配对数，BioRED 是 10.1，BioTriplex 是 4.9。
Tokens are not compute省 token 不等于省算力
Joint requests cut prompt tokens by 89% (BioRED) and 81% (BioTriplex), but local compute time only by 35–40% and 25–30%. Chunked requests in Qwen3.5-9B were equivalent to pairwise in recall on BioTriplex and had lower recall on BioRED.joint 请求让提示 token 减少了 89%（BioRED）和 81%（BioTriplex），本地计算时间却只减少 35–40% 和 25–30%。Qwen3.5-9B 的 chunked 请求在 BioTriplex 上召回和 pairwise 等价，在 BioRED 上召回更低。
Evidence assessment证据核验
One annotator (the author) judged 76 sampled edges with the model, request form, and stratum hidden: first whether the displayed passage supports the relation, then, after locking those labels, whether each set of cited sentences does. Criteria follow the BioRED annotation guideline.一名标注者（作者本人）在看不到模型、请求形式和分层的情况下，判断 76 条抽样的边：先判断展示的段落是否支持这个关系，锁定这些答案后，再判断每组引用句是否支持。判断口径按 BioRED 标注指南。
| Corpus语料 | Stratum分层 | N | n | Supported支持 | Proportion [95%]比例 [95%] |
|---|---|---|---|---|---|
| BioRED Test | S1 · outside gold, pairwise onlyS1 · 金标外，只有 pairwise 报出 | 326 | 18 | 2 | 0.111 [0.032, 0.321] |
| S2 · outside gold, jointS2 · 金标外，joint 报出 | 380 | 16 | 7 | 0.438 [0.234, 0.664] | |
| G · gold-matchedG · 命中金标 | 280 | 6 | 5 | 0.833 [0.440, 0.970] | |
| S1 − S2 | −0.326 [−0.567, −0.034] | ||||
| BioTriplex test | S1 | 82 | 16 | 11 | 0.688 [0.467, 0.847] |
| S2 | 186 | 14 | 7 | 0.500 [0.275, 0.725] | |
| G | 83 | 6 | 5 | 0.833 [0.448, 0.969] | |
| S1 − S2 | +0.188 [−0.128, +0.464] |
On BioRED, the 16 unsupported S1 items were co-mentions (11), genes reported only as prognostic factors (3), and relations only to a symptom (2). On BioTriplex, predictions outside the gold set were often supported, consistent with an incomplete gold standard. The assessment measures support in one displayed passage under BioRED criteria; it is not a measure of factual correctness, and it has one annotator who knows the hypotheses.BioRED 上 S1 的 16 条不支持：只是共现 11 条，只作预后因子 3 条，只与症状有关 2 条。BioTriplex 上金标外的边常常得到支持，这和金标不完整相符。这项核验测的是一个展示段落里、按 BioRED 口径的支持，不是事实正确性；而且只有一名知道研究假设的标注者。
Input scope输入范围
Same 30 BioTriplex articles, same entities, same gold pairs, pairwise requests; only the passages the model reads change.同样的 30 篇 BioTriplex 论文、同样的实体和金标配对、同样用 pairwise 请求，只改变模型读的段落。
Recall on BioTriplex validation and test. Abstract passages cover 73 of 154 gold pairs; the full-text runs recovered 149. Any extractor limited to abstracts and the same co-occurrence rule faces the same ceiling.BioTriplex val+test 上的召回。摘要段落覆盖 154 个金标配对中的 73 个，全文运行找回了 149 个。任何只读摘要、用同样共现规则的抽取器，都受同一个上限约束。
Evaluation choices评估选择
Reading the output输出的读法
The canonical reading accepts predefined equivalent spellings (bracketed sentence IDs, entity aliases); the strict reading does not. The readings agreed on all six primary test contrasts but changed one secondary test classification and two development classifications.规范化读法接受预先规定的等价写法（带方括号的句子编号、实体别名），严格读法不接受。两种读法在 6 个主要测试对比上结论一致，但改变了 1 个次要测试比较和 2 个开发集比较的分类。
Output format (exploratory)输出格式（探索性）
With an untuned line-based format on development data, Gemma-4-12B joint failed calls fell from 24 to 1, yet many more assertions were rejected and recall dropped; the pairwise − joint point estimate reversed. These are single development runs, not an interaction test.在开发集上换成未调过的 lines 格式后，Gemma-4-12B joint 的失败调用从 24 次降到 1 次，但被拒收的断言大幅增加，召回下降，pairwise − joint 的点估计也反转了。这只是开发集上的单次运行，不是交互作用检验。
Guideline rules (exploratory)指南规则（探索性）
Adding nine relation rules from the BioRED annotation guideline to a minimal instruction kept recall, precision, and F1 within δ in both models on development data, at about 35% more prompt tokens in Qwen3.5-9B.在最简指令上加入 BioRED 标注指南里的 9 条关系规则，两个模型在开发集上的召回、精确率和 F1 都在 δ 之内，但 Qwen3.5-9B 的提示 token 多了约 35%。
Failure accounting失败的统计口径
Failed and omitted judgments count as misses in every primary result. Every run, including the exploratory ones, is listed with its failed calls, truncations, rejected assertions, and omitted pairs.失败和遗漏的判断在所有主要结果里都算没有预测。每个运行（包括探索性的）都列出了失败调用、截断、被拒收的断言和遗漏的配对。
Audit trail审计记录
- Test splits can only be read through a frozen manifest committed before any test run; it records digests of the code, configurations, data files, weights, and execution scripts, and each test run is journaled.测试集只能通过冻结清单读取，清单在任何测试运行之前提交；它记录了代码、配置、数据文件、模型权重和执行脚本的哈希，每次测试运行都写日志。
- An append-only provenance note records the freeze, the launch checks, and when results were first opened.一份只追加的审计记录写明了冻结、启动检查，以及结果第一次被打开的时间。
- The evidence assessment followed a written protocol whose dated revision log lists every deviation: first-phase labels reviewed before locking with self-reported times, and estimation code written after labeling (alternatives reported).证据核验按书面协议进行，协议的修订记录逐条写明了每一处偏差：第一阶段答案在锁定前回看过、用时为自报，估计代码在标注之后才写（同时报告了替代做法）。
- This public repository starts from a clean history; the freeze (commit 5ac1447) was made in the private development repository.这个公开仓库是从干净的历史开始的；冻结（提交 5ac1447）是在私有的开发仓库里做的。
Limitations局限
- Two local 9–12B models; one run per condition; intervals cover document sampling, not run-to-run variation.只有两个 9–12B 的本地模型；每个条件只跑一次；区间只反映论文抽样，不包括运行之间的差异。
- Entities are given and candidate pairs are limited to passage co-occurrence; 13 BioRED Test gold pairs fall outside this rule.实体是给定的，候选配对限于段落内共现；BioRED Test 有 13 个金标配对不满足这个规则。
- BioTriplex: 30 articles, recall only.BioTriplex 只有 30 篇，只报召回。
- Both public test splits were used in earlier work on this project; the protocol was frozen before the test runs of this study.两个公开测试集在本项目以前的工作里用过；本研究的协议在测试运行之前冻结。
- One hypothesis-informed annotator, small strata, passage-level support.只有一名知道研究假设的标注者，各层样本小，判断的是段落层面的支持。
Repository and documents仓库和文档
Code and data代码和数据
- github.com/Suzuka2788/Gene2Disease-LLM
- Pipeline, tests, configurations, aggregate results; standard-library Python.流程代码、测试、配置和汇总结果；只用 Python 标准库。
- Not included: manuscript (in preparation), corpus texts, gold annotations, per-paper outputs, annotation sheets.不包括：稿件（准备中）、语料原文、金标、逐篇输出、标注表。
Results documents (Chinese)结果文档（中文）
- E4_RESULTS.md · frozen test evaluation冻结后的测试集评测
- EVIDENCE_RESULTS.md · evidence assessment证据核验
- EF_DEV_RESULTS.md · development results开发集结果
- E4_PROVENANCE.md · audit record审计记录
- EVIDENCE_PROTOCOL.md · assessment protocol核验协议