使用 Browser Harness 通过 NCBI E-utilities 高效抓取 PubMed 文献数据【免费下载链接】browser-harnessBrowser Harness | Self-healing harness that enables LLMs to complete any task.项目地址: https://gitcode.com/gh_mirrors/br/browser-harness导读本文介绍在 Browser Harness 项目中如何完全不依赖浏览器、仅用http_get调用 NCBI E-utilities REST API 完成 PubMed收录 3700 万 生物医学文献记录的检索与数据抽取。你将掌握 ESearch → ESummary 两调用快速管线、EFetch XML 深度解析、usehistoryWebEnv大结果集分页、EInfo/ELink 跨库能力以及一系列经过真实输出验证的字段解析与限流避坑要点可直接用于文献综述、元数据分析、实体提取等自动化任务。为什么 PubMed 应该走 API 而不是浏览器在 Browser Harness 的领域技能Domain Skills体系中每个站点都有一份独立的scraping.md作为 Agent 的首选操作手册。项目根目录 SKILL.md 明确规定了边界A basic fetch of public information needs no browser. If a plain HTTP request can read it — a public page, an API, docs — usecurlor your fetch tool, and leave the browser alone.PubMed 正是典型场景https://pubmed.ncbi.nlm.nih.gov的所有数据都可以通过 NCBI E-utilities REST API 用纯 HTTP 获取不需要浏览器渲染、不需要登录、不需要 API Key。相比之下浏览器方案需要启动 CDP 守护进程、导航页面、等待渲染、解析 DOM代价高且毫无必要。同时pubmed.ncbi.nlm.nih.gov的搜索结果页是动态渲染的直接在浏览器里抓反而更脆弱。在 Browser Harness 中执行这一管线的关键是http_get帮助函数——它是浏览器内核 helpers 提供的纯 HTTP 请求封装与浏览器完全解耦def http_get(url, headersNone, timeout20.0): Pure HTTP — no browser. Use for static pages / APIs. Wrap in ThreadPoolExecutor for bulk. When BROWSER_USE_API_KEY is set, routes through the fetch-use proxy (handles bot detection, residential proxies, retries). Falls back to local urllib otherwise.从源码看http_get具备两点对 PubMed 抓取很重要的行为无 Key 时走本地urllib自带User-Agent: Mozilla/5.0与 gzip 解压处理helpers.pyPubMed 返回的 gzip 响应会被自动解压为文本设置了BROWSER_USE_API_KEY时自动路由到 fetch-use 代理处理 bot 检测、住宅代理、重试批量抓取时天然具备更高的容错性。因此Agent 在启用BH_DOMAIN_SKILLS1并访问pubmed.ncbi.nlm.nih.gov时goto_url会自动返回本目录下scraping.md的技能提示见 helpers.py 的 domain-skills 探测逻辑从而引导 Agent 走上用http_get E-utilities的正轨而不是贸然打开浏览器。关键事实NCBI E-utilities 无需 API Key免费申请 Key 后速率限制从 3 req/s 提升到 10 req/s。所有端点均为公开 REST 接口返回 JSON 或 XML。最快管线ESearch → ESummary两调用、全 JSON、无 XML 解析对于大多数任务检索 → 获取元数据ESearch → ESummary 是最快路径两次调用、两个 JSON 响应、全程不碰 XML。import json from helpers import http_get # Step 1: search → get PMIDs search json.loads(http_get( https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi ?dbpubmedtermdeeplearningradiologyretmax10retmodejson )) pmids search[esearchresult][idlist] # e.g. [41999029, 41998456, ...] count search[esearchresult][count] # total hits across all pages # Step 2: fetch lightweight metadata for all PMIDs in one call summary json.loads(http_get( fhttps://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi f?dbpubmedid{,.join(pmids)}retmodejson )) result summary[result] for uid in result[uids]: art result[uid] print(uid, art[pubdate], art[source]) print( , art[title][:80]) print( authors:, [a[name] for a in art[authors][:3]]) # Confirmed output (2026-04-18): # 41999029 2026 Apr 18 Med Sci Monit # Use of Deep Learning Models in the Diagnosis of Proptosis Through Orbi # authors: [Kesimal U, Akkaya HE, Polat Ö] # 41998456 2026 Apr 17 Sci Rep # ...何时升级到 EFetch XML当需要完整摘要正文、MeSH 主题词、完整作者名非 Last I 缩写形式、结构化摘要小节标签、或从文献记录内提取 DOI 时使用 EFetch XML。常见工作流搜索 PubMedESearchESearch 的核心职责是关键词 → PMID 列表并返回总命中数与查询翻译import json from helpers import http_get data json.loads(http_get( https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi ?dbpubmed termlargelanguagemodelsclinical retmax5 retmodejson sortpubdate # newest first; default is relevance datetypepdat # filter by publication date mindate2024/01/01maxdate2024/12/31 # YYYY/MM/DD format )) result data[esearchresult] print(Total hits:, result[count]) # 24160 — note: string, not int print(PMIDs:, result[idlist]) print(Query translation:, result[querytranslation]) # Confirmed output (2026-04-18): # Total hits: 24160 # PMIDs: [41996895, 41996722, 41996006, 41995888, 41995759] # Query translation: large language models[MeSH Terms] OR ...querytranslation是 NCBI 对查询的 MeSH 扩展解释务必检查它以确认搜索意图被正确理解详见下文 Gotchas。ESearch 字段标签Field Tags追加到 term标签写法含义machine learning[MeSH Terms]MeSH 受控词表Hinton GE[Author]作者姓 首字母attention is all you need[Title]标题词Nature[Journal]期刊名2024[pdat]出版年份布尔运算符AND、OR、NOT。短语搜索exact phrase[Title]英文引号包裹精确短语并加字段标签。排序选项sort值效果省略相关度默认pubdate最新出版优先Author第一作者字母序JournalName期刊字母序轻量元数据 —— ESummaryJSON无 XMLESummary 用逗号分隔的 PMID 列表批量返回轻量字段一次调用可覆盖全部检索结果import json from helpers import http_get data json.loads(http_get( https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi ?dbpubmedid41999029,41998456,41997837retmodejson )) result data[result] for uid in result[uids]: art result[uid] # Key fields available: title art[title] # full title string source art[source] # abbreviated journal name fulljournalname art[fulljournalname] pubdate art[pubdate] # e.g. 2026 Apr 18 epubdate art[epubdate] # e-pub ahead of print date (may be empty) authors art[authors] # list of {name: Last I, authtype: ...} volume art[volume] issue art[issue] pages art[pages] pubtype art[pubtype] # list: [Journal Article, Review, ...] # Extract DOI from elocationid or articleids: doi_field art[elocationid] # e.g. doi: 10.12659/MSM.951157 article_ids {x[idtype]: x[value] for x in art[articleids]} doi article_ids.get(doi) pmc_id article_ids.get(pmc) # PMC ID if open access print(uid, pubdate, source) print( , title[:70]) print( doi:, doi, | pmc:, pmc_id) # Confirmed output (2026-04-18): # 41999029 2026 Apr 18 Med Sci Monit # Use of Deep Learning Models in the Diagnosis of Proptosis Through Orbi # doi: 10.12659/MSM.951157 | pmc: None注意articleids是一个[{idtype: ..., value: ...}]列表转成字典后即可同时取出doi与pmc开放获取文章才有 PMC ID。完整文献元数据 —— EFetch XML当需要完整摘要正文、完整作者名、MeSH 主题词、结构化摘要小节时使用 EFetch XML。由于 EFetch 返回的是 XML不是 JSON需要配合 Python 标准库xml.etree.ElementTree解析import json, xml.etree.ElementTree as ET from helpers import http_get raw http_get( https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi ?dbpubmedid41999029,36328784retmodexmlrettypeabstract ) root ET.fromstring(raw) for art in root.findall(.//PubmedArticle): mc art.find(MedlineCitation) pmid mc.find(PMID).text article mc.find(Article) # Title — use itertext() to handle embedded tags like i, sub title .join(article.find(ArticleTitle).itertext()).strip() # Abstract — plain or structured (BACKGROUND / METHODS / RESULTS / CONCLUSION) abstract_el article.find(Abstract) if abstract_el is not None: sections [] for t in abstract_el.findall(AbstractText): label t.get(Label, ) # e.g. BACKGROUND, METHODS text .join(t.itertext()).strip() sections.append(f[{label}] {text} if label else text) abstract .join(sections) else: abstract # ~15% of articles have no abstract # Journal year journal article.find(Journal) j_title journal.find(Title).text if journal is not None else pub_date journal.find(.//PubDate) if journal is not None else None if pub_date is not None: year_el pub_date.find(Year) medline_el pub_date.find(MedlineDate) # fallback for old/seasonal dates season_el pub_date.find(Season) # e.g. Jul-Aug, Oct-Dec year (year_el.text if year_el is not None else medline_el.text[:4] if medline_el is not None else ) # DOI doi_el next( (e for e in article.findall(ELocationID) if e.get(EIdType) doi), None ) doi doi_el.text if doi_el is not None else # Authors — handle CollectiveName (consortium/group authors) author_list article.find(AuthorList) authors [] if author_list is not None: for a in author_list.findall(Author): collective a.find(CollectiveName) last a.find(LastName) fore a.find(ForeName) initials a.find(Initials) if collective is not None: authors.append(collective.text) elif last is not None: full last.text if fore is not None: full f, {fore.text} authors.append(full) # MeSH controlled vocabulary terms mesh_list mc.find(MeshHeadingList) mesh_terms [] if mesh_list is not None: mesh_terms [ mh.find(DescriptorName).text for mh in mesh_list.findall(MeshHeading) if mh.find(DescriptorName) is not None ] print(fPMID{pmid} ({year}) {j_title}) print(f Title: {title[:70]}) print(f Authors: {authors[:3]}) print(f DOI: {doi}) print(f MeSH: {mesh_terms[:4]}) print(f Abstract: {abstract[:120]}) # Confirmed output (2026-04-18): # PMID41999029 (2026) Medical science monitor : international medical... # Title: Use of Deep Learning Models in the Diagnosis of Proptosis Thro # Authors: [Kesimal, Uğur, Akkaya, Habip Eser, Polat, Önder] # DOI: 10.12659/MSM.951157 # MeSH: [Humans, Deep Learning, Exophthalmos, Magnetic Resonance Imaging] # Abstract: BACKGROUND Proptosis is a common manifestation of orbital disease... # PMID36328784 (...) # Abstract: [OBJECTIVES] Physical inactivity and sedentary behaviour... ← structured这段代码示范了四个重要的解析模式itertext()处理混入i/sub的标题、AbstractText的Label属性还原结构化摘要BACKGROUND / METHODS / RESULTS / CONCLUSION、ELocationID的EIdTypedoi属性提取 DOI、CollectiveName兜底处理团体作者详见 Gotchas。大结果集 —— usehistory WebEnv当命中数超过retmax上限 10 000时不要对每一页重新执行 ESearch。用usehistoryy让 NCBI 在服务端保存结果集然后通过WebEnvquery_key直接分页 EFetchimport json, xml.etree.ElementTree as ET from helpers import http_get # Step 1: ESearch with usehistoryy — NCBI holds result set on server search json.loads(http_get( https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi ?dbpubmedtermCRISPRgeneeditingretmax0retmodejsonusehistoryy )) webenv search[esearchresult][webenv] # server-side session token query_key search[esearchresult][querykey] # result set ID within session total int(search[esearchresult][count]) print(fTotal: {total}, WebEnv: {webenv[:30]}..., query_key: {query_key}) # Confirmed output (2026-04-18): # Total: 24160, WebEnv: MCID_69e4203757db89391008d6f1..., query_key: 1 # Step 2: EFetch pages using WebEnv (no re-searching) batch_size 200 for start in range(0, min(total, 1000), batch_size): # cap at 1000 for demo raw http_get( fhttps://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi f?dbpubmedquery_key{query_key}WebEnv{webenv} fretstart{start}retmax{batch_size}retmodexmlrettypeabstract ) root ET.fromstring(raw) articles root.findall(.//PubmedArticle) print(f Fetched {len(articles)} articles (start{start})) # process articles here...要点retmax0的 ESearch 只返回计数、不返回 ID配合usehistoryy可计数 存结果一步完成后续每页只需改retstart偏移。EInfo —— 列出 NCBI 数据库与库元数据EInfo 无需任何参数即可返回全部 39 个 NCBI 数据库2026-04-18 实测import json from helpers import http_get data json.loads(http_get( https://eutils.ncbi.nlm.nih.gov/entrez/eutils/einfo.fcgi?retmodejson )) dbs data[einforesult][dblist] print(fTotal databases: {len(dbs)}) # Confirmed: 39 (2026-04-18) print(dbs[:10]) # [pubmed, protein, nuccore, ipg, nucleotide, structure, # genome, annotinfo, assembly, bioproject]指定dbpubmed则可拿到 PubMed 专属元数据——字段列表、记录总数、可用链接类型import json from helpers import http_get data json.loads(http_get( https://eutils.ncbi.nlm.nih.gov/entrez/eutils/einfo.fcgi?dbpubmedretmodejson )) db_info data[einforesult][dbinfo][0] print(DB name:, db_info[dbname]) print(Record count:, db_info[count]) # total PubMed records link_names [l[name] for l in db_info.get(linklist, [])] print(fLink types ({len(link_names)}):, link_names[:5]) # Confirmed (2026-04-18): # DB name: pubmed # Record count: 37620453 # Link types (48): [pubmed_assembly, pubmed_bioproject, ...]ELink —— 跨数据库关联ELink 把 PubMed 记录关联到其他 NCBI 数据库。注意pubmed_pubmedrelated articles 相似文献依赖的相似度服务器间歇性不可用会返回Couldnt resolve #exLinkSrv2, the address table is empty.。优先使用下述非相似度类 linknameimport json from helpers import http_get # Link a PMID to its free full-text in PMC (if open access) # linknamepubmed_pmc — may also hit the server outage; check error field data json.loads(http_get( https://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi ?dbfrompubmedid38325330linknamepubmed_pmcretmodejson )) error data.get(ERROR, ) if error: print(ELink error:, error) # Couldnt resolve #exLinkSrv2... — NCBI server issue else: for ls in data.get(linksets, []): for lsdb in ls.get(linksetdbs, []): print(lsdb[linkname], →, lsdb[links][:5])PubMed 可用的 ELink linkname共 48 个中常用者如下linkname目标pubmed_pmcPMC 中的免费全文pubmed_pubmed_citedin引用本文的文献pubmed_pubmed_refs本文引用的参考文献pubmed_gene相关 Gene 记录pubmed_clinvar与文献相关的临床变异pubmed_gds相关 GEO 数据集实用替代方案如果 ELink 挂掉直接从 EFetch/ESummary 提取 DOI用https://doi.org/{doi}直达全文页面。URL 与参数速查E-utilities 基础 URLhttps://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi # search → PMIDs https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi # PMIDs → JSON summary https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi # PMIDs → full XML https://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi # cross-db links https://eutils.ncbi.nlm.nih.gov/entrez/eutils/einfo.fcgi # DB metadataESearch 参数参数取值说明dbpubmed检索 PubMed 恒为pubmedterm查询字符串支持[Author]、[Title]、[MeSH Terms]等字段标签retmax整数上限 10000单次返回结果数retmodejsonJSON 输出sortpubdate、Author、JournalName默认相关度datetypepdat出版、edatentrez 收录、mdat修改日期过滤类型mindate、maxdateYYYY/MM/DD或YYYY需配合datetypeusehistoryy服务端保存结果返回webenvquerykeyEFetch 参数参数取值说明dbpubmedid38000000,37999999逗号分隔 PMID单次建议不超过约 200 个query_keyWebEnv来自usehistoryy的 ESearch大结果集时替代idretstart整数WebEnv 分页偏移retmax整数上限 10000批大小retmodexmlEFetch 必须用 XML完整记录无 JSON 输出rettypeabstract返回摘要 核心元数据PubMed 文章 URL 构造pmid 41999029 pubmed_url fhttps://pubmed.ncbi.nlm.nih.gov/{pmid}/ doi 10.12659/MSM.951157 doi_url fhttps://doi.org/{doi} # resolves to publisher page pmc_id PMC9876543 # from ESummary articleids pmc_url fhttps://www.ncbi.nlm.nih.gov/pmc/articles/{pmc_id}/Gotchas —— 实战避坑清单以下是本技能文档经过真实请求验证的坑点Agent 与开发者都应逐条对照count是字符串而非整数。search[esearchresult][count]返回24160而不是24160。任何算术运算前必须int()转换。EFetch 完整记录retmode必须为xml。与 ESearch、ESummary 不同EFetch 带retmodejson返回的是扁平文本MEDLINE 引文文本格式不是结构化 JSON。EFetch 响应一律用xml.etree.ElementTree解析。ArticleTitle可能内嵌 XML 标签。含斜体iStaphylococcus aureus/i或数学sub2/sub的标题是混合内容节点。必须用.join(el.itertext())而不是el.text——后者会静默丢弃第一个子标签之后的所有内容。约 15% 的文章没有摘要。短通讯、社论、信函及老记录中article.find(Abstract)返回None。必须用if abstract_el is not None守卫。作者名结构多变务必处理CollectiveName。联盟/团体论文把组名如GeKeR Study Group、Breast Cancer Association Consortium放在CollectiveName下而不是LastName/ForeName。个体作者才有LastName及可选的ForeName与Initials。应先检查CollectiveName直接落到LastName而不检查会产生None错误。已确认的真实案例2026-04-18PMID 37586835GeKeR Study Group、PMID 36328784Breast Cancer Association Consortium。PubDate 有三种结构。多数文章有Year 可选Month 可选Day季刊用Season如Jul-Aug、Oct-Dec代替Month少数老记录用MedlineDate如1995 Fall且没有Year。安全提取模式pub_date journal.find(.//PubDate) year_el pub_date.find(Year) if pub_date is not None else None medline_el pub_date.find(MedlineDate) if pub_date is not None else None year (year_el.text if year_el is not None else medline_el.text[:4] if medline_el is not None else )EFetch 批量 ID 控制在约 200 个/次。API 接受id逗号分隔列表但超大批次500偶尔会超时或返回截断的 XML。超过 200 篇时按块迭代或改用usehistoryWebEnv。ELinkpubmed_pubmed相似文献间歇性故障。NCBI 相似度服务器返回Couldnt resolve #exLinkSrv2, the address table is empty.——这是截至 2026-04-18 持续存在的服务端问题不是限流错误。pubmed_gene、pubmed_pmc、pubmed_clinvar等 linkname 也可能报同样错误。此时用 DOI 作为跳转出版商全文的兜底。速率限制无 Key 3 req/s免费 Key 10 req/s。无 Key 时超过 3 req/s 返回 HTTP 429。连续请求之间插入time.sleep(0.34)。在 NCBI 账户页面申请免费 Key 后在所有 URL 追加api_keyYOUR_KEY。ESearchretmax上限为 10 000。要取回超过 10 000 个 PMID用usehistoryy并通过 EFetch 的retstart偏移分页。EFetch 自身的retmax也接受最高 10 000。retmax0时 ESearch 只返回计数、不返回 ID——适合纯计数场景。配合usehistoryy可在不预取 ID 的情况下存储结果供后续分页search json.loads(http_get( https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi ?dbpubmedtermcancerretmax0retmodejsonusehistoryy )) total int(search[esearchresult][count]) # e.g. 4800000 webenv search[esearchresult][webenv]ESummaryauthors字段是缩写名Last I不是完整名。需要完整名如Kesimal, Uğurvs ESummary 的Kesimal U时用 EFetch XML 取ForeName。批量任务不需要完整名时ESummary 更快。querytranslation显示 NCBI 如何解释你的查询。ESearch 响应中的esearchresult.querytranslation是查询的 MeSH 扩展版本。检查它可以确认检索结果是否符合你的本意例如拼写是否被当成 MeSH 词、运算符是否按预期组合。在 Browser Harness 中的落地方式1. 作为领域技能被自动加载本文件位于agent-workspace/domain-skills/pubmed/目录。当BH_DOMAIN_SKILLS1开启领域技能时Agent 导航到pubmed.ncbi.nlm.nih.govgoto_url 会自动探测该目录并返回最多 10 个技能文件名指引 Agent 先读本文件再动手——这正是本技能文档先读再做设计意图的实现机制。任务特定代码应写入 agent-workspace/agent_helpers.py而本文的from helpers import http_get中的helpers即指 Browser Harness 内核 helperssrc/browser_harness/helpers.py与 agent_helpers 合并后的命名空间。2. 用browser-harnessCLI 直接运行所有示例代码都可以原样放进browser-harness的 heredoc 中执行——CLI 会自动预导入全部 helpersrun.py先调用ensure_daemon()再exec且_install_helper_trace()会对每个 helper 调用做耗时记录browser-harness PY import json from helpers import http_get search json.loads(http_get( https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi ?dbpubmedtermdeeplearningradiologyretmax5retmodejson )) print(search[esearchresult][count], search[esearchresult][idlist]) PY由于纯 HTTP 抓取不涉及浏览器这个过程不会启动 CDP 守护进程的浏览器导航也不消耗浏览器资源。3. 批量抓取的扩展建议并发http_get的 docstring 明确建议Wrap in ThreadPoolExecutor for bulk。对 PubMed 这类有硬性速率限制的 API并发度应受速率限制约束无 Key 时 3 req/s即并发窗口内总请求数 ÷ 时间 ≤ 3而不是盲目拉满线程数容错http_get设置了 20 秒默认超时timeout20.0可针对大响应调整设置BROWSER_USE_API_KEY后会自动启用 fetch-use 代理的 bot 检测规避与重试helpers.py组合管线检索类任务本技能与全文抓取类任务如同一目录体系中的 arxiv/scraping.md 示范的citation_*meta 解析思路可以串联成检索 → 元数据 → 全文的完整流水线全程无需浏览器。结语PubMed 是典型的API 优先数据源3700 万 条文献记录、无鉴权、JSON/XML 双格式、服务端分页全部通过 NCBI E-utilities 的五个 REST 端点ESearch / ESummary / EFetch / ELink / EInfo可达。配合 Browser Harness 的http_get纯 HTTP helper任何 Agent 都能以不碰浏览器的方式完成从关键词检索、批量元数据获取到全文摘要抽取的完整链路。唯一需要牢记的是尊重 3/10 req/s 的速率限制、用int()处理字符串计数、用itertext()处理混合内容节点、始终守卫CollectiveName与缺失摘要——本技能文档中的每一个 Gotcha 都来自真实请求的确认输出可直接作为生产代码的校验依据。【免费下载链接】browser-harnessBrowser Harness | Self-healing harness that enables LLMs to complete any task.项目地址: https://gitcode.com/gh_mirrors/br/browser-harness创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考