后端【免费下载链接】defuddleGet the main content of any page as Markdown.项目地址https://gitcode.com/gh_mirrors/de/defuddle点击查看免费下载在网页正文提取场景中「Logo」是最常见也最容易误判的页面噪声几乎所有站点都会把站标放在header里而部分站点又习惯用logo作为 class 命名的一部分。当正文容器的 class 恰好也包含logo子串时例如a-content--logo-end一个粗糙的子串匹配规则会把整篇正文误删。本文以 defuddle 仓库中的测试样本 selectors--logo-substring-wrapper.html 及其期望输出 selectors--logo-substring-wrapper.md 为切入点逐行拆解 defuddle 的选择器清理机制精确选择器与部分选择器如何分工、logo词边界正则如何做到该删的删、该留的留以及这一行为如何被 fixture 测试体系固化下来。读完本文你将掌握 defuddle 选择器规则的判定边界也能学会如何为自己的站点排查正文被误删这类问题。测试样本要验证什么一个名字里藏着 logo的正文容器先看这个 fixture 对应的原始 HTMLtests/fixtures/selectors--logo-substring-wrapper.html它的结构非常典型!DOCTYPE html html head titleCity Approves Transit Expansion - Example Daily/title meta nameauthor contentJordan Blake /head body header classsite-header div classlogoa href/img src/img/logo.svg altExample Daily/a/div /header main classpage-main article classarticle-body h1City Approves Transit Expansion/h1 div classkey-points pThe council approved three new transit lines .../p !-- 若干关键要点段落 -- /div div classa-content a-content--logo-end div classbody-copy pThe vote clears the way for construction to begin .../p !-- 正文主体段落 -- /div /div /article /main /body /html页面上存在两个敏感目标header classsite-header内的div classlogo——这是真正的站点 Logo 容器必须被剔除div classa-content a-content--logo-end——这是正文的外层包装容器它的 class 里含有logo子串a-content--logo-end中的logo但它承载的是文章主体绝不能因为子串匹配被误删。测试文件名selectors--logo-substring-wrapper正是对这一场景的精确概括logo-substringlogo 子串与 wrapper正文包装层。期望输出 selectors--logo-substring-wrapper.md 证明了 defuddle 的正确行为Logo 容器消失而正文完好无损地保留下来{ title: City Approves Transit Expansion, author: Jordan Blake, site: Example Daily, published: }The council approved three new transit lines on a nine-to-four vote after a hearing that ran past midnight, ending a debate that has occupied the chamber in one form or another for the better part of two decades without ever reaching a binding decision.Bonds issued by the transit authority will cover roughly two thirds of the projected cost, with the remainder drawn from a state matching fund that lawmakers renewed earlier this year over the objections of the rural caucus, which wanted the money spent on highway resurfacing instead.Two members who voted against the measure have asked for an independent audit of the ridership projections before any bonds are issued, and a third said she would support the audit even though she voted in favor of the expansion on the merits of the eastern alignment.Construction on the first line is scheduled to begin in March and to reach the eastern terminus in four years, though the authority has cautioned that the schedule assumes it can acquire the last right-of-way parcels without going to court over the price of two industrial lots.The county commission still has to ratify the cost-sharing agreement, a step the mayors office describes as a formality but that at least two commissioners have said they intend to use as leverage for a station stop that the current alignment leaves out entirely.The vote clears the way for construction to begin next spring on lines that will connect the eastern suburbs to the downtown core for the first time since the original system opened, when the eastern edge of the built-up area stopped several miles short of where it stands today.Officials estimate that the first of the three lines will carry passengers within four years of groundbreaking, assuming the environmental review now underway concludes on schedule and the authority is able to acquire the last of the right-of-way parcels without resorting to condemnation proceedings.Opponents argued that the projections rest on optimistic assumptions about population growth in corridors that have seen very little new housing built since the last comprehensive plan, and that the authority has historically overestimated demand on precisely the kind of suburban extension now proposed.Transit advocates who packed the chamber called the vote long overdue. Earlier proposals collapsed over disputes about which neighborhoods would host maintenance yards and how the costs would be shared between the city and the surrounding county, whose commission has yet to schedule its own vote on the funding agreement.The authoritys chief engineer told the council that early utility relocation work could begin as soon as the environmental review concludes, and that the agency has already begun the procurement process for the rolling stock it expects to need by the time the eastern line opens for revenue service.A spokesperson for the mayor said the administration considers the funding agreement settled and does not expect the county commission to reopen the cost-sharing formula, which was negotiated over eighteen months and signed by both executives before the council took up the measure this spring.Construction is expected to begin in March, pending final environmental review.对比 HTML 与期望输出可以发现三件事站点 Logodiv.logo被删除key-points与a-content a-content--logo-end两层包装被解除解包而非保留正文段落以纯 Markdown 段落形式输出标题、作者、站点名被正确提取到元数据中。选择器清理在 defuddle 处理管线中的位置在深入规则细节之前先明确这段逻辑所处的处理阶段。defuddle 的主流程位于 src/defuddle.ts其中removeBySelector是内容清理管线中的确定性清除步骤removeHiddenElements按计算样式剔除隐藏元素removeEyebrowLabel剔除标题上方的分类标签removeBySelector按 class/id/属性选择器确定性剔除已知非内容元素导航、页脚、侧栏、Logo 等在 src/defuddle.ts 中调用入口函数定义于 src/removals/selectors.tsremoveLowScoring基于链接密度等启发式打分剔除低分块运行在removeBySelector之后因此选择器阶段清理得越干净打分阶段看到的 DOM 越纯净。removeBySelector的调用还受两个开关控制removeExactSelectors与removePartialSelectors默认均为 true定义见 src/types.ts分别对应下面要讲的精确匹配与部分匹配两条路径。第一道防线精确选择器直接命中 Logo 容器defuddle 将特征非常明确的元素写死在EXACT_SELECTORS数组中src/constants.ts运行前用join(,)拼成一个巨型选择器字符串EXACT_SELECTORS_JOINED在 src/removals/selectors.ts 中通过一次querySelectorAll全部取出。与 Logo 相关的精确规则有三条src/constants.ts// logos [classlogo i], #logo, #Logo,注意第一条用的是完全相等的属性选择器[classlogo i]即只有当 class 属性值恰好是logo不区分大小写时才命中。本 fixture 中的div classlogo正是这种情况因此它在精确匹配阶段就被标记删除与后面的部分匹配逻辑无关。而a-content--logo-end的 class 值不是logo不会被这条规则命中。精确匹配阶段还有若干保护性跳过逻辑避免误伤正文跳过代码块内的元素el.closest(pre, code)防止语法高亮 span 被误删跳过带响应式显示类如 Tailwind 的hidden sm:flex的隐藏元素跳过mainContent的祖先元素防止正文容器本身被拔除。第二道防线部分选择器按词边界匹配 logo仅仅处理classlogo显然不够现实中大量站点使用site-logo、logo-wrapper、header-logo这类组合类名。这些靠PARTIAL_SELECTORS数组中的子串模式处理其中 logo 的规则是整条数组里最精密的一条src/constants.ts// Matches logo only at a class-token edge (site-logo, logo-wrapper). (?:(?![\\w-])logo|logo(?![\\w-])),这是一个词边界式正则含义是logo要么出现在 class token 的开头后面不能跟\w或-即后面是 token 结束要么出现在 token 的结尾前面不能是\w或-即前面是 token 开始。用这个规则验证本 fixture 中的两个候选class 值是否匹配原因logo已在精确阶段删除—精确选择器[classlogo i]命中site-logo匹配logo在 token 末尾前面是-lookaheadlogo(?![\\w-])成功logo-wrapper匹配logo在 token 开头lookbehind(?![\\w-])logo成功a-content--logo-end不匹配logo前后都是-两个方向的边界断言均失败关键就在a-content--logo-endlogo前面是-、后面也是-既不是 token 开头也不是 token 结尾于是两条分支(?![\\w-])logo与logo(?![\\w-])全部落空正文容器得以存活。这正是该 fixture 名称中 substring 一词的深意——规则并非简单的包含 logo 就删而是logo 作为一个独立词元出现才删从而在logo作为其他单词组成部分如catalog、dialogue以及本场景的 BEM 风格--logo-end修饰符时保持克制。如果改成朴素子串匹配logo那么a-content--logo-end会被命中并整体删除整篇正文随之丢失——本 fixture 就是用来永久防止这类回归的。部分匹配的完整判定流程removeBySelector的部分匹配阶段src/removals/selectors.ts并不只检查 class而是把多个测试属性拼接后统一做正则测试。测试属性定义在 src/constants.tsexport const TEST_ATTRIBUTES [ class, id, data-component, data-test, data-testid, data-test-id, data-qa, data-cy ];运行时会拼成[class],[id],[data-component],...这样的属性选择器TEST_ATTRIBUTES_SELECTORsrc/constants.ts一次性取出所有候选元素。对每个候选元素跳过已标记删除的元素、[data-defuddle]提取器输出内的元素以及 code/pre 及其内部元素——语法高亮、内联代码的类名如language-name、meta-很容易命中噪声词必须先排除非标题元素把class >赞分享后端【免费下载链接】defuddleGet the main content of any page as Markdown.项目地址https://gitcode.com/gh_mirrors/de/defuddle点击查看免费下载相关推荐Defuddle 社交互动计数器移除机制解析如何让 “13 Likes” 不污染提取的正文Defuddle 社交互动计数器移除机制解析如何让 “13 Likes” 不污染提取的正文 本文围绕 Defuddle 仓库中的 fixture 测试用例 c后端defuddle 内容模式清理实战代码块样板文本与文末冗余区块的自动剔除defuddle 内容模式清理实战代码块样板文本与文末冗余区块的自动剔除 本文以 defuddle 仓库中的测试样例 content patterns cod后端Sherlock 移除站点清单解析用户名探测的设计边界与站点剔除机制Sherlock 移除站点清单解析用户名探测的设计边界与站点剔除机制 Sherlock 是一个通过用户名并发探测各大网站个人主页的开源工具其站点数据以 JS网络安全CLI渗透测试上一篇5分钟永久备份QQ空间青春回忆GetQzonehistory使用指南下一篇GetQzonehistory3分钟永久保存你的QQ空间青春回忆创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考