https://txtmix.com/tags/%E7%BD%91%E9%A1%B5%E6%8A%93%E5%8F%96/Recent content in 网页抓取 on Text MatrixHugozh-cnSat, 23 May 2026 08:55:34 +0800https://txtmix.com/posts/tech/scrapling-adaptive-web-scraping-framework-guide/Sun, 12 Apr 2026 02:31:39 +0800https://txtmix.com/posts/tech/scrapling-adaptive-web-scraping-framework-guide/<h1 id="scrapling现代网页爬虫框架自适应反反爬并发爬取">Scrapling：现代网页爬虫框架——自适应、反反爬、并发爬取</h1> <h2 id="一项目概述">一、项目概述</h2> <h3 id="11-scrapling-是什么">1.1 Scrapling 是什么</h3> <p><strong>Scrapling</strong> 是 D4Vinci 开发的<strong>自适应网页爬虫框架</strong>，能够处理从单次请求到大规模爬取的各类场景。其核心特点是：</p> <ul> <li><strong>自适应解析</strong>：网站结构变化时自动定位元素</li> <li><strong>反反爬绕过</strong>：开箱即用绕过 Cloudflare Turnstile 等反爬机制</li> <li><strong>并发爬取</strong>：支持 Scrapy 风格的 Spider 框架</li> <li><strong>AI 集成</strong>：内置 MCP Server，支持 AI 辅助爬取</li> </ul> <h3 id="12-核心数据">1.2 核心数据</h3> <table> <thead> <tr> <th>指标</th> <th>数值</th> </tr> </thead> <tbody> <tr> <td>Stars</td> <td>36.1k ⭐</td> </tr> <tr> <td>Forks</td> <td>3.1k</td> </tr> <tr> <td>最新版本</td> <td>v0.4.5 (2026-04-07)</td> </tr> <tr> <td>许可证</td> <td>BSD-3-Clause</td> </tr> <tr> <td>语言</td> <td>Python 99.9%</td> </tr> <tr> <td>贡献者</td> <td>17</td> </tr> </tbody> </table> <h3 id="13-为什么选择-scrapling">1.3 为什么选择 Scrapling</h3> <table> <thead> <tr> <th>特点</th> <th>说明</th> </tr> </thead> <tbody> <tr> <td>🤖 自适应解析</td> <td>网站改版后自动重新定位元素</td> </tr> <tr> <td>🛡️ 反反爬</td> <td>绕过 Cloudflare Turnstile</td> </tr> <tr> <td>⚡ 性能卓越</td> <td>文本提取 2.02ms（比 MechanicalSoup 快 767 倍）</td> </tr> <tr> <td>🕷️ Spider 框架</td> <td>Scrapy 风格，支持并发、暂停/恢复</td> </tr> <tr> <td>🔌 AI 集成</td> <td>MCP Server，AI 辅助数据提取</td> </tr> <tr> <td>🐳 Docker 支持</td> <td>一键部署，含所有浏览器</td> </tr> </tbody> </table> <hr> <h2 id="二技术架构">二、技术架构</h2> <h3 id="21-整体架构">2.1 整体架构</h3> <div class="code-block code-line-numbers open" style="counter-reset: code-block 0"> <div class="code-header language-"> <span class="code-title"><i class="arrow fas fa-angle-right" aria-hidden="true"></i></span> <span class="ellipses"><i class="fas fa-ellipsis-h" aria-hidden="true"></i></span> <span class="copy" title="复制到剪贴板"><i class="far fa-copy" aria-hidden="true"></i></span> </div><pre tabindex="0"><code>┌─────────────────────────────────────────────────────────────┐ │ Scrapling │ ├─────────────────────────────────────────────────────────────┤ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Spiders │ │ Fetchers │ │ Parser │ │ │ │ (爬取框架) │ │ (请求引擎) │ │ (解析引擎) │ │ │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ └────────────────┼────────────────┘ │ │ ▼ │ │ ┌─────────────────────────────────────────────────┐ │ │ │ Core Engine │ │ │ │ • Session Management │ │ │ │ • Proxy Rotation │ │ │ │ • Adaptive Element Tracking │ │ │ │ • Checkpoint/Pause &amp; Resume │ │ │ └─────────────────────────────────────────────────┘ │ │ ▼ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Fetcher │ │StealthyFetcher│ │DynamicFetcher│ │ │ │ (HTTP) │ │ (反反爬) │ │ (浏览器) │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ └─────────────────────────────────────────────────────────────┘</code></pre></div> <h3 id="22-核心组件">2.2 核心组件</h3> <table> <thead> <tr> <th>组件</th> <th>功能</th> <th>适用场景</th> </tr> </thead> <tbody> <tr> <td><strong>Fetcher</strong></td> <td>HTTP 请求，TLS 指纹伪装</td> <td>静态页面</td> </tr> <tr> <td><strong>StealthyFetcher</strong></td> <td>高级反反爬，Cloudflare 绕过</td> <td>反爬网站</td> </tr> <tr> <td><strong>DynamicFetcher</strong></td> <td>浏览器自动化，JS 渲染</td> <td>SPA/动态加载</td> </tr> <tr> <td><strong>Spider</strong></td> <td>并发爬取框架</td> <td>大规模爬取</td> </tr> <tr> <td><strong>Selector</strong></td> <td>CSS/XPath/文本解析</td> <td>数据提取</td> </tr> <tr> <td><strong>MCP Server</strong></td> <td>AI 辅助爬取</td> <td>智能数据提取</td> </tr> </tbody> </table> <hr> <h2 id="三fetcher-详解">三、Fetcher 详解</h2> <h3 id="31-三种-fetcher-对比">3.1 三种 Fetcher 对比</h3> <table> <thead> <tr> <th>Fetcher</th> <th>速度</th> <th>反反爬</th> <th>JS 支持</th> <th>适用场景</th> </tr> </thead> <tbody> <tr> <td><strong>Fetcher</strong></td> <td>⚡⚡⚡</td> <td>❌</td> <td>❌</td> <td>静态页面，高速请求</td> </tr> <tr> <td><strong>StealthyFetcher</strong></td> <td>⚡⚡</td> <td>✅ Cloudflare</td> <td>❌</td> <td>反爬网站，无需 JS</td> </tr> <tr> <td><strong>DynamicFetcher</strong></td> <td>⚡</td> <td>✅</td> <td>✅</td> <td>SPA，动态内容</td> </tr> </tbody> </table> <h3 id="32-http-请求fetcher">3.2 HTTP 请求（Fetcher）</h3> <div class="code-block code-line-numbers open" style="counter-reset: code-block 0"> <div class="code-header language-python"> <span class="code-title"><i class="arrow fas fa-angle-right" aria-hidden="true"></i></span> <span class="ellipses"><i class="fas fa-ellipsis-h" aria-hidden="true"></i></span> <span class="copy" title="复制到剪贴板"><i class="far fa-copy" aria-hidden="true"></i></span> </div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">scrapling.fetchers</span> <span class="kn">import</span> <span class="n">Fetcher</span><span class="p">,</span> <span class="n">FetcherSession</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># 单次请求</span> </span></span><span class="line"><span class="cl"><span class="n">page</span> <span class="o">=</span> <span class="n">Fetcher</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;https://quotes.toscrape.com/&#39;</span><span class="p">)</span> </span></span><span class="line"><span class="cl"><span class="n">quotes</span> <span class="o">=</span> <span class="n">page</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;.quote .text::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># 会话请求（保持 Cookie）</span> </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">FetcherSession</span><span class="p">(</span><span class="n">impersonate</span><span class="o">=</span><span class="s1">&#39;chrome&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">session</span><span class="p">:</span> </span></span><span class="line"><span class="cl"> <span class="n">page</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;https://example.com/&#39;</span><span class="p">)</span> </span></span><span class="line"><span class="cl"> <span class="n">data</span> <span class="o">=</span> <span class="n">page</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;.item::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span></span></span></code></pre></div></div> <p><strong>特性：</strong></p>