Scrapling: An Adaptive Web Scraping Framework

What Is Scrapling?

Scrapling is an open-source Python framework for web scraping that combines request fetching, browser automation, and HTML parsing in one library. It ships three fetcher types - a plain HTTP Fetcher, a StealthyFetcher for anti-bot sites, and a DynamicFetcher for full browser automation - plus a Scrapy-style Spider class for concurrent, resumable crawls.

Key Features

• ✓Three fetcher classes for different jobs: Fetcher for plain HTTP with TLS/header impersonation, StealthyFetcher for Cloudflare Turnstile and other anti-bot walls, and DynamicFetcher for full Playwright/Chrome automation.
• ✓Adaptive element tracking - the parser remembers an element's context and relocates it by similarity after a site's HTML changes, instead of just failing the selector.
• ✓A Scrapy-like Spider class with concurrent requests, per-domain throttling, AutoThrottle delay tuning, and checkpoint-based pause/resume via Ctrl+C.
• ✓Built-in proxy rotation, DNS-over-HTTPS to avoid DNS leaks behind a proxy, and the option to connect to a remote browser over CDP instead of launching one locally.
• ✓Background XHR/fetch capture (capture_xhr) that grabs a page's own API responses while it loads, so you don't have to reverse-engineer the network calls yourself.
• ✓An MCP server for AI coding agents (Claude, Cursor) that extracts targeted content before handing it to the model, plus an Agent Skill that teaches the current API.
• ✓CSS, XPath, filter-based, text, and regex selection on the same parser, with a find_similar() method for locating elements like one you already found.
• ✓A CLI (scrapling extract, scrapling shell) for pulling a page to .txt/.md/.html without writing a script, including a curl-to-Scrapling converter in the shell.

What Can You Build With Scrapling?

• •Price and inventory monitors that keep working after a retailer redesigns its product pages, because the adaptive selectors relocate the price and stock fields instead of returning None.
• •Full-site crawls behind Cloudflare Turnstile using StealthyFetcher, with proxy rotation and AutoThrottle so you don't get blocked mid-crawl.
• •A Shopify catalog puller via the ready-made ShopifySpider template - one run pulls every product and variant through the store's own JSON API.
• •Long-running crawls you can pause with Ctrl+C and resume later from a checkpoint directory, useful for jobs that outlive a single work session.
• •An MCP-connected scraping backend for an AI coding agent, so the agent gets pre-extracted page content instead of burning tokens on raw HTML.

Installing Scrapling

Scrapling requires Python 3.10 or higher. Install the base library with pip install scrapling. The README notes that this base install only includes the parser engine and its dependencies, not the fetchers - the exact extras/browser-install step is not clearly documented in what's provided here, so check the docs site linked from the repo before you reach for StealthyFetcher or DynamicFetcher for the first time.

Basic Scrapling Usage

Basic usage is import-and-call. A one-off GET: ```python from scrapling.fetchers import Fetcher page = Fetcher.get('https://quotes.toscrape.com/') quotes = page.css('.quote .text::text').getall() ``` For anti-bot sites, swap the fetcher and add `adaptive=True` so selectors survive later markup changes: ```python from scrapling.fetchers import StealthyFetcher page = StealthyFetcher.fetch('https://example.com', headless=True) products = page.css('.product', adaptive=True) ``` For a real crawl, subclass `Spider`, list `start_urls`, and yield items from an async `parse()`: ```python from scrapling.spiders import Spider, Response class MySpider(Spider): name = "demo" start_urls = ["https://example.com/"] async def parse(self, response: Response): for item in response.css('.product'): yield {"title": item.css('h2::text').get()} MySpider().start() ```

Scrapling Alternatives

Frequently Asked Questions

The problem it solves

Scrapers written against a site's current CSS classes and XPath paths break the moment that site redesigns its markup - dev teams then spend hours re-inspecting the DOM and patching selectors one by one. Scrapling's adaptive parser fingerprints an element's context (tag, attributes, text, position) once, then relocates it after the page changes, so the same selector call keeps returning the same logical field instead of an empty result.

Who should try it — and who should skip

Try Scrapling if you're maintaining scrapers against sites that change their markup often, or you're fighting Cloudflare Turnstile with a bare requests+BeautifulSoup setup and losing. It also makes sense for anyone building an MCP-connected scraping backend for a coding agent - that's a specific niche this library covers that Scrapy doesn't. Skip it if you have a working Scrapy pipeline with custom middlewares already in production; the migration cost likely outweighs what adaptive selectors buy you, and Scrapling's own Spider API borrows from Scrapy rather than replacing the ecosystem around it.

Related repositories