GeneDiseaseTrace-LLM
Extracting gene–disease relations from full papers with an LLM. Each reported relation includes the source sentences used as evidence.用 LLM 从论文全文中抽取基因–疾病关系，并为每条关系保留作为证据的原文句子。
The task任务定义
The task covers 100 biomedical papers in PMC JATS XML format.任务包含 100 篇生物医学论文，输入为 PMC JATS XML 全文。
Task 1 · Gene and disease extraction任务 1 · 基因与疾病实体抽取
Extract all gene and disease mentions from each paper and output the identified entity lists, if any.抽取每篇论文中提及的所有基因与疾病名称，并输出识别出的实体列表（如有）。
Task 2 · Gene–disease relationship analysis任务 2 · 基因–疾病关系分析
For paper i with Ni genes and Mi diseases, evaluate all Ni × Mi candidate pairs. Link a pair if the paper provides evidence of an association, correlation, or causal relationship, yielding a gene–disease bipartite graph per paper.对含 Ni 个基因与 Mi 个疾病的第 i 篇论文，评估全部 Ni × Mi 个候选配对；若原文提供关联、相关或因果关系的证据则建立连边，最终构建每篇论文的基因–疾病二部图。
We also record whether a relation is a finding of the paper or background knowledge. Pairs with weak evidence are marked uncertain and kept separate from the supported links.在原任务之外，我们还标注关系来自本文发现还是背景知识。证据不足的配对记为 uncertain，与 supported 连边分开。
How the rules were developed规则是怎样确定的
We used PubTator3 as a baseline. Early blind tests included Claude Opus 5.5, GPT-6.1 Sol and GPT-6 Astra.对比基线为 PubTator3。早期盲测使用了 Claude Opus 5.5、GPT-6.1 Sol 和 GPT-6 Astra。
Write the extraction steps and decision rules in skill.md.在 skill.md 中写明抽取步骤和判定规则。
Refine the rules on pilot papers, then fix the evaluation sets before comparing configurations.用试点论文修改规则，在比较配置前固定评估集。
Run the papers in fresh sessions using only the written rules.在新会话中运行论文，只提供书面规则。
Check quotes, character offsets and output tables with local code.用本地代码检查引文、字符偏移和输出表格。
Compare model outputs and rule versions for each gene–disease pair.逐对比较模型输出和规则版本。
From full rules to a decision card从完整规则到判定卡
- We reviewed existing systems and chose PubTator3 for comparison.调研现有系统后，我们选择 PubTator3 做对比。
- We developed the rules on 5 pilot papers, from Skill 1.0 to 1.3, and collected 35 boundary cases to check consistency.规则在 5 篇试点论文上从 Skill 1.0 修改到 1.3，另整理了 35 个边界案例检查判定是否一致。
- We shortened the rules into a decision card to reduce input tokens and interaction turns. It agreed with the full rules on 110/112 boundary-case decisions, with the same result across 3 runs.为减少输入 token 和交互轮次，我们把完整规则压缩成判定卡。在边界案例测试中，110/112 个判定与完整规则相同，3 次运行结果一致。
- We compared the rule and reasoning configurations on a fixed set of 12 papers.随后在固定的 12 篇论文上比较规则和推理配置。
- We used the selected V1 configuration for all 100 papers, then evaluated it on BioRED without BioRED-specific training or prompt tuning.选定 V1 后，运行全部 100 篇论文，再用 BioRED 评估；没有针对 BioRED 做训练或提示词调优。
Cost and runtime成本与耗时
These runs used Claude Opus 5.5. Costs are the CLI-reported list prices and include repair requests.以下运行使用 Claude Opus 5.5，成本按 CLI 报告的官方标价统计，包含修复请求。
The paper sets differ between rows. The 100-paper average includes long papers and all repair requests.各行使用的论文集不同。100 篇均值包含长论文和全部修复请求。
Reducing model calls减少模型调用
- We checked model behavior on 5 baseline papers before and after each change.每次修改前后，都在 5 篇基准论文上检查模型行为。
- Local code locates text spans and calculates character offsets.文本定位和字符偏移计算交给本地代码。
- Each paper gets one initial request and, if validation fails, at most one repair request.每篇论文先请求一次；校验失败时，至多再请求一次修复。
- The shorter rules reduced input from 20k to 9k tokens and are reused as a cached system prompt.压缩规则后，输入从 20k 降到 9k tokens，并通过缓存系统提示词复用。
Shorter model responses缩短模型输出
- The response contains entities, evidence references and gene–disease pairs.模型返回实体、证据引用和基因–疾病配对。
- The model returns sentence numbers. Local code then retrieves the original quotes and their character offsets.模型只返回句子编号，本地代码再取出原文并计算字符偏移。
- Very long documents, such as conference proceedings, are split into chunks.会议论文集等超长文献分块处理。
Extraction steps抽取步骤
Put the rules, numbered paragraphs and sentences, and response schema in one prompt.将规则、编号后的段落和句子、输出格式放入同一条提示词。
Return structured judgments without tool calls, file reads or shell commands.模型直接返回结构化判定，不调用工具、读文件或执行命令。
Locate mentions, calculate offsets, fill the gene × disease matrix and check the output format.本地代码定位实体提及、计算偏移、补全基因 × 疾病矩阵，并检查格式。
Send validation errors, such as a mismatched citation, back to the model for one repair attempt.如有引文不匹配等校验错误，将错误发回模型，尝试修复一次。
A/B test: rule compression and thinking intensityA/B 测试：规则压缩与推理强度
12 papers frozen prior to testing: 7 research articles (2 short, 3 medium, 2 long), 2 case reports (one in Spanish), 2 reviews, and 1 brief report. All variants used Claude Opus 5.5, with one initial request and at most one repair per paper. Relation counts below are from final outputs.测试前冻结的 12 篇论文：7 篇研究论文（2 短、3 中、2 长）、2 篇病例报告（含 1 篇西班牙语文献）、2 篇综述与 1 篇简报。所有配置均使用 Claude Opus 5.5，每篇一次初始请求、至多一次修复；下列关系数来自最终输出。
| V1 · card, mediumV1 · 判定卡，中等推理 | V2 · full rules, mediumV2 · 完整规则，中等推理 | V3 · card, lowV3 · 判定卡，低推理 | |
|---|---|---|---|
| First-request cost, 12 papers首次请求成本（12 篇） | $3.55 | $4.01 | $2.71 |
| Total cost, including repairs含修复的总成本 | $4.61 | $4.55 | $3.12 |
| Total model time, including repairs含修复的模型总耗时 | 837 s | 789 s | 334 s |
| Repair requests修复请求数 | 4 | 2 | 2 |
| Supported relationsSupported 关系数 | 64 | 54 | 56 |
| Uncertain / negative pairsUncertain / negative 配对 | 28 / 1 | 20 / 11 | 15 / 1 |
| Pair-status agreement with V1与 V1 的配对状态一致率 | — | 0.976 | 0.985 |
- V1 and V2 agreed on 97.6% of pair statuses. The card reduced first-request cost by 11%, but total cost after repairs was similar ($4.61 vs $4.55). Most of V2’s 11 negative labels came from variant comparisons in one paper; those cases still need manual review.V1 和 V2 的配对状态一致率为 97.6%。判定卡使首次请求成本降低 11%，加上修复后总成本相近（$4.61 vs $4.55）。V2 的 11 个 negative 标签主要来自一篇论文中的变异比较，这些案例还需要人工复核。
- Compared with V1, low reasoning effort used about 60% less time and 32% less money, including repairs. Status agreement was 98.5%; supported labels fell from 64 to 56, and uncertain labels from 28 to 15.与 V1 相比，低推理强度的总耗时减少约 60%，总成本减少约 32%，均包含修复。状态一致率为 98.5%；supported 标签从 64 个降至 56 个，uncertain 从 28 个降至 15 个。
- We used V1 for the 100-paper run. It returned the most supported labels in this comparison. Without expert labels for these papers, status agreement and output counts cannot tell us which configuration is more accurate or has higher recall.我们选用 V1 运行 100 篇论文，因为它在这次比较中返回的 supported 标签最多。这批论文还没有专家金标准，状态一致率和输出数量不足以判断哪种配置更准确、召回更高。
Comparison with PubTator3 (same 12 papers)与 PubTator3 的对比（同一组 12 篇论文）
We compared our full-text outputs with PubTator3 annotations retrieved by PMID for the same 12 papers, aligning entity mentions to the local text. PubTator3 annotates entities in full text, but its relations mainly come from abstracts. This comparison describes the two output sets; neither is a gold standard.我们按 PMID 获取同一组 12 篇论文的 PubTator3 预计算标注，将实体提及对齐到本地文本，再与全文抽取结果比较。PubTator3 的实体标注覆盖全文，关系主要来自摘要。这里比较两个系统的输出，双方都不作为金标准。
Strengths优势
- Our output contains 2.8 times as many relation pairs (64 vs 23), each with source sentences. These counts alone do not measure accuracy or recall.本方法报告的关系对数量为基线的 2.8 倍（64 vs 23），每条附原文句子。仅凭这些数量，无法判断准确率或召回率。
- 81% draw on full-text sections beyond the abstract: body text, tables, and captions.81% 的关系证据来自摘要以外的全文内容：正文、表格与图注。
- Relations are labeled as study findings or background knowledge; 28 pairs are marked uncertain. On matched BioRED relations, source-label agreement is 75% (526/703).关系附有本文发现或背景知识标签，另有 28 对标为 uncertain。在匹配的 BioRED 关系上，来源标签一致率为 75%（526/703）。
- Distinguishes clinical symptoms from formal disease entities.将临床症状与正式的疾病实体区分开。
- The method returned relations from a Spanish paper for which PubTator3 returned no genes.在一篇西班牙语论文中，本方法抽取到了关系，PubTator3 未返回基因。
Limitations局限性
- For these 12 papers, the mean was $0.38 and 70 s per paper including repairs ($0.30 for the first request). PubTator3 annotations are free to query.这 12 篇论文平均每篇花费 $0.38、耗时 70 秒，包含修复；仅首次请求为 $0.30。PubTator3 的预计算标注可免费查询。
- Our supported set excludes 12 pairs reported by PubTator3: 3 involve entities our rules classify as symptoms, 4 received non-supported labels, and 5 involve an entity we did not capture. The disagreements await independent review.我们的 supported 集合未包含 PubTator3 报告的 12 对：3 对涉及本规则归为症状的实体，4 对未判为 supported，5 对涉及未捕获的实体。这些分歧尚未经过独立复核。
- Does not normalize entities to database identifiers (NCBI Gene / MeSH).暂未将实体标准化映射至数据库标识符（NCBI Gene / MeSH）。
Evaluation across all 100 papers全部 100 篇论文的评估结果
Parser changes解析器修改
The parser now accepts common harmless syntax variations that previously triggered extra repair calls.解析器现在兼容一些不影响含义的常见语法变化，减少由此触发的额外修复。
Papers with no supported output未返回 supported 关系的论文
The pipeline returned no supported relations for 25 papers. We grouped them by their captured entities; confirming them as true negatives will require independent full-text review.有 25 篇论文未返回 supported 关系。按捕获的实体可分为以下几类，是否属于真阴性还需要独立阅读全文确认。
- 8 have no gene entities captured by the pipeline (e.g., dialysis timing, adolescent gambling, nanoparticle chemistry).8 篇未捕获基因实体（如透析时机选择、青少年博彩行为、纳米颗粒化学）。
- 8 have genes but no disease entities captured under our policy (e.g., fly chromatin, worm oocytes, plant transcriptomes).8 篇捕获基因但未捕获本规则范围内的疾病实体（如果蝇染色质、线虫卵母细胞、植物转录组研究）。
- 9 have both entity types but no pair labeled supported (e.g., assay markers or speculative discussion).9 篇捕获两类实体，但没有配对被标记为 supported（如实验标志物或推测性讨论）。
PubTator3 reported one relation across these 25 papers. A relation missed by both systems would remain undetected in this comparison.PubTator3 在这 25 篇中报告了 1 条关系。这种比较仍会遗漏两个系统都未检出的关系。
Evaluation on BioREDBioRED 评估
BioRED has 600 abstracts. We used the 419 with at least one gold gene–disease relation (1,633 relations in total). We ran the frozen V1 configuration without BioRED-specific training or prompt tuning, at $0.056 per abstract. The other 181 abstracts were excluded.BioRED 共 600 篇摘要。我们使用其中至少有一条金标准基因–疾病关系的 419 篇，共 1,633 条关系。运行时使用冻结的 V1 配置，未针对 BioRED 训练或调整提示词，平均成本为 $0.056/篇。其余 181 篇未纳入评估。
We scored gene–disease pairs within each abstract by matching overlapping mention spans, without requiring the relation type to agree. These are scores for the relation-positive subset, not official full-task BioRED scores. Quote and offset checks confirm the cited source spans; semantic support still needs review.我们按摘要内的基因–疾病对评分，通过提及跨度重叠匹配，不要求关系类型相同。这些分数适用于关系阳性子集，不是 BioRED 官方全任务分数。引文和偏移检查能确认引用位置，句子是否支持关系仍需复核。
| Precision精确率 | Recall召回率 | F1 | |
|---|---|---|---|
| Strict (supported only)严格口径（仅计 supported） | 0.89 | 0.43 | 0.58 |
| Supported + uncertain宽口径（supported + uncertain） | 0.81 | 0.58 | 0.68 |
- Train / Dev / Test F1 scores were 0.58 / 0.61 / 0.54. The 95% confidence intervals overlap, so we have not established equivalent performance across splits.Train / Dev / Test 的 F1 分别为 0.58 / 0.61 / 0.54，95% 置信区间重叠，目前还不能认定各子集表现等价。
- On the same 50 abstracts, our precision was 0.84 vs PubTator3’s 0.83, recall 0.41 vs 0.65, and F1 0.55 vs 0.73. The precision estimates are close; equivalence was not established. PubTator3 uses BioRED in training, although we did not verify the deployed models’ exact training split.在同一批 50 篇摘要上，本方法与 PubTator3 的精确率为 0.84 vs 0.83，召回率为 0.41 vs 0.65，F1 为 0.55 vs 0.73。精确率估计相近，但尚未证明等价。PubTator3 训练使用了 BioRED，我们未核实部署模型的具体训练划分。
- Of 930 missed gold relations, 547 (59%) had both entities captured: 295 were labeled not_found, 240 uncertain and 12 explicit_negative. Another 131 involved symptom/phenotype classification, 159 an uncaptured disease and 93 an uncaptured gene. Manual review is needed to distinguish rule differences from extraction errors.930 条漏检金标准关系中，547 条（59%）已捕获两侧实体：295 条标为 not_found，240 条 uncertain，12 条 explicit_negative。另有 131 条涉及症状/表型分类，159 条未捕获疾病，93 条未捕获基因。规则差异和抽取错误各占多少，还需要人工复核。
- Source labels agreed with BioRED on 526/703 matched relations (75%), with "both" mapped to Novel. This measures agreement among matched relations, not end-to-end novelty detection.在匹配的 703 条关系中，526 条来源标签与 BioRED 一致（75%）；“两者兼有”映射为 Novel。这衡量的是匹配关系上的一致性，不是端到端的新颖性识别表现。
Gene–disease bipartite graphs基因–疾病二部图
Each paper has a bipartite graph with genes on one side and diseases on the other. Edges represent supported relations and retain the source sentences and study-finding/background labels.每篇论文对应一张二部图，两侧分别为基因和疾病。连边表示 supported 关系，保留原文句子及本文发现/背景知识标签。
Next experiments接下来做什么
First, review full-text relations and missed pairs independently. Then test how much the rules, evidence requirements and repair requests each contribute. Broader evaluation should include relation-negative abstracts, one-to-one matching and relation-type scoring.先独立复核全文关系和漏检配对，再分别测试规则、证据要求和修复请求的作用。后续评估还需要纳入关系阴性摘要，采用一对一匹配，并对关系类型评分。
Compare models比较不同模型
Run GPT-6.1 Sol, GPT-6 Astra and other models on the same benchmark to compare accuracy and cost.在同一基准上运行 GPT-6.1 Sol、GPT-6 Astra 和其他模型，比较准确性和成本。
Try a local model尝试本地模型
Test a local open-weight model for initial extraction, with a commercial model checking selected results.测试用本地开放权重模型初步抽取，再由商业模型复核部分结果的方案。
Detailed results详细结果
注：以下交互结果页内容为英文（二部图页面支持中英切换）。