拼多多反爬对抗实战：Scrapy 中间件化采集架构解析

1. 选型依据PDD 公开数据分布在移动端 APImobile.yangkeduo.com与 H5mobile.pinduoduo.com。当采集规模上升手写requests 线程池在三个方面迅速失效调度限流、重试、去重需自行实现易误伤出口 IP对抗耦合签名、UA、Cookie、代理切换逻辑与业务混杂难以演进可观测缺乏统一日志、统计与失败队列。Scrapy 以「中间件 调度器 Pipeline」的声明式结构内建上述能力。本框架的核心设计原则是将反爬对抗逻辑全部收敛至 Downloader/Spider Middleware业务 Spider 仅负责请求编排与解析。轻量一次性抓取用httpx parsel即可需限流、代理池与长期运行的采集Scrapy 为更优解。2. 反爬机制与应对矩阵维度表现应对请求签名移动端强制anti_content设备指纹 时间戳 请求体哈希签名算法中间件化或接入合规签名 SDK频率限制同 IP/设备高频触发 460/限流代理 IP 池 AUTOTHROTTLE设备指纹client_info/device_id行为关联多套设备参数轮换行为验证风险后返回滑块/验证码降速、换出口 IP、降并发登录态部分接口需有效 Cookie账号池 Cookie 持久化anti_content是移动端第一道校验门。社区已有开源还原实现但随版本迭代维护成本高生产环境建议将其封装为独立签名服务由中间件调用隔离算法失效风险。3. 框架结构scrapy-pinduoduo/ ├── scrapy_pinduoduo/ │ ├── settings.py │ ├── items.py # dataclass 数据模型 │ ├── middlewares/ │ │ ├── proxy_middleware.py # 隧道代理亿牛云 │ │ ├── sign_middleware.py # anti_content 注入 │ │ └── ua_middleware.py # UA/设备指纹轮换 │ ├── spiders/ │ │ ├── goods_search.py │ │ └── goods_detail.py │ └── pipelines/ │ └── save_pipeline.py ├── tests/ ├── pyproject.toml └── scrapy.cfg依赖隔离使用uv对抗逻辑与业务 Spider 解耦更换签名方案或代理供应商时仅改对应 middleware。4. 初始化uv venvsource.venv/bin/activate uvaddscrapy pydantic httpx scrapy genspider goods_search mobile.yangkeduo.compyproject.toml[project] name scrapy-pinduoduo version 0.1.0 requires-python 3.10 dependencies [scrapy2.11, pydantic2.0, httpx0.27]5. 商品搜索 Spiderscrapy_pinduoduo/spiders/goods_search.pyfrom__future__importannotationsimportscrapyfromscrapy_pinduoduo.itemsimportGoodsItemclassGoodsSearchSpider(scrapy.Spider):拼多多商品搜索爬虫移动端 H5 接口分页采集。namegoods_searchallowed_domains[mobile.yangkeduo.com]api_templatehttps://mobile.yangkeduo.com/proxy/api/api/aristotle/search?q{keyword}page{page}custom_settings{DOWNLOAD_DELAY:1.5,CONCURRENT_REQUESTS:4,AUTOTHROTTLE_ENABLED:True,}def__init__(self,keyword:str手机壳,max_page:int10,*args,**kwargs):super().__init__(*args,**kwargs)self.keywordkeyword self.max_pageint(max_page)defstart_requests(self)-scrapy.Request:forpageinrange(1,self.max_page1):urlself.api_template.format(keywordself.keyword,pagepage)yieldscrapy.Request(url,callbackself.parse,meta{need_sign:True,page:page},errbackself.on_error,)defparse(self,response:scrapy.http.Response):dataresponse.json()itemsdata.get(goods_list,[])ordata.get(items,[])forrowinitems:yieldGoodsItem(goods_idrow.get(goods_id),titlerow.get(goods_name)orrow.get(title),pricerow.get(price)/100ifrow.get(price)elseNone,salesrow.get(sales_num)orrow.get(cnt),shop_idrow.get(mall_id),pageresponse.meta.get(page),)defon_error(self,failure):self.logger.error(请求失败: %s | %s,failure.request.url,failure.value)items.pyfrom__future__importannotationsfromdataclassesimportdataclassdataclassclassGoodsItem:goods_id:int|Nonetitle:str|Noneprice:float|Nonesales:int|Noneshop_id:int|Nonepage:int|None6. 代理中间件PDD 对出口 IP 敏感规模化采集依赖代理池打散 IP 维度。本例采用 亿牛云代理 隧道代理——服务端按请求轮询出口 IP无需自维护 IP 池契合 Scrapy 每请求换 IP 的高并发模型。scrapy_pinduoduo/middlewares/proxy_middleware.pyfrom__future__importannotationsimportbase64fromscrapyimportsignalsfromscrapy.httpimportRequest,ResponseclassYiniuProxyMiddleware:亿牛云隧道代理中间件固定入口 Basic 鉴权按请求自动换 IP。PROXY_URLhttp://tunnel.16yun.cn:10000PROXY_AUTH用户名:密码# 替换为亿牛云后台鉴权信息classmethoddeffrom_crawler(cls,crawler):mcls()crawler.signals.connect(m.spider_opened,signalsignals.spider_opened)returnmdefprocess_request(self,request:Request,spider)-None:ifrequest.meta.get(use_proxy,True):request.meta[proxy]self.PROXY_URL tokenbase64.b64encode(self.PROXY_AUTH.encode()).decode()request.headers[Proxy-Authorization]fBasic{token}defprocess_response(self,request:Request,response:Response,spider):ifresponse.statusin(429,460,503):spider.logger.warning(代理限流 %s触发重试,response.status)request.dont_filterTruereturnrequestreturnresponsedefspider_opened(self,spider)-None:spider.logger.info(亿牛云代理中间件已启用)settings.py中间件注册数值为执行优先级DOWNLOADER_MIDDLEWARES{scrapy_pinduoduo.middlewares.ua_middleware.UAMiddleware:300,scrapy_pinduoduo.middlewares.proxy_middleware.YiniuProxyMiddleware:350,scrapy_pinduoduo.middlewares.sign_middleware.SignMiddleware:400,}隧道代理相较私密 IP 池的优势在于内置轮询规避因轮换策略不当导致的 IP 复用封禁中小规模采集性价比最高。7. 签名中间件scrapy_pinduoduo/middlewares/sign_middleware.pyfrom__future__importannotationsfromscrapyimportRequestclassSignMiddleware:anti_content 注入。签名实现下沉至独立服务/SDK本类仅做调用。defprocess_request(self,request:Request,spider)-None:ifnotrequest.meta.get(need_sign):returnbodyrequest.body.decode()ifrequest.bodyelserequest.headers[anti-content]self.generate_anti_content(request.url,body)staticmethoddefgenerate_anti_content(url:str,body:str)-str:raiseNotImplementedError(接入 anti_content 签名实现开源还原或合规 SDK)8. 落盘 Pipelinescrapy_pinduoduo/pipelines/save_pipeline.pyfrom__future__importannotationsimportcsvimportjsonfrompathlibimportPathfromscrapy_pinduoduo.itemsimportGoodsItemclassSavePipeline:双格式落盘JSON Lines流式入库 CSV运营分析。defopen_spider(self,spider):self.out_dirPath(output)self.out_dir.mkdir(exist_okTrue)self.jsonlopen(self.out_dir/f{spider.name}.jsonl,w,encodingutf-8)self.csv_fileopen(self.out_dir/f{spider.name}.csv,w,encodingutf-8-sig,newline)self.writercsv.DictWriter(self.csv_file,fieldnames[goods_id,title,price,sales,shop_id,page],)self.writer.writeheader()defclose_spider(self,spider):self.jsonl.close()self.csv_file.close()defprocess_item(self,item:GoodsItem,spider)-GoodsItem:rowitem.__dict__ self.jsonl.write(json.dumps(row,ensure_asciiFalse)\n)self.writer.writerow(row)returnitem9. 限速与重试settings.pyCONCURRENT_REQUESTS8DOWNLOAD_DELAY1.0RANDOMIZE_DOWNLOAD_DELAYTrue# 抖动延迟规避固定节奏AUTOTHROTTLE_ENABLEDTrueAUTOTHROTTLE_START_DELAY1.0AUTOTHROTTLE_MAX_DELAY10.0AUTOTHROTTLE_TARGET_CONCURRENCY4.0RETRY_ENABLEDTrueRETRY_TIMES3RETRY_HTTP_CODES[429,460,500,502,503,504]DUPEFILTER_CLASSscrapy.dupefilters.RFPDupeFilter10. 分布式扩展规模超出单机时scrapy-redisstart_urls与去重指纹下沉至 Redis多节点协同消费能力下沉签名服务、代理网关独立为 HTTP 微服务Spider 无状态水平扩容。注意PDD 对「同设备指纹的多机并发」同样聚类风控设备参数池须随节点数线性扩张否则扩机器反而加速封禁。