Scrapy: Web Scraping Framework for Python

What Scrapy is for web data extraction

Scrapy is a web crawling and scraping framework for Python, published on PyPI and conda-forge under a BSD-3-Clause license. Per its GitHub description, it's built to pull structured data out of websites instead of leaving you to hand-roll a parser for every site, and it's cross-platform, requiring Python 3.10 or newer. Zyte (formerly Scrapinghub) maintains it along with a wider group of outside contributors.

Core capabilities of Scrapy

• ✓Ships as an installable PyPI package (`pip install scrapy`) and also as a conda-forge package, so it drops into whichever Python environment a team already manages.
• ✓Runs on Python 3.10 or newer, per the version badge in the repository.
• ✓Maintained by Zyte (formerly Scrapinghub) plus a wider base of outside contributors credited on GitHub.
• ✓Carries a public GitHub Actions test suite and Codecov coverage reporting, so changes to the framework get checked before they ship.
• ✓Licensed BSD-3-Clause, a permissive license with no copyleft obligation for projects that depend on it.
• ✓Has a DeepWiki badge in the README linking to deepwiki.com/scrapy/scrapy.
• ✓Tagged on GitHub under crawler, crawling, scraping, web-scraping, and framework topics, and participates in Hacktoberfest.

Getting started with Scrapy

Installation is one command: `pip install scrapy` pulls the package from PyPI, and a conda-forge build is available too, per the badge in the README. Past that single step, the README doesn't walk through project setup or a first run inline; it hands off to the documentation site at docs.scrapy.org for everything else. If you want to contribute code rather than just use the package, the README points to a separate contributing guide instead of describing the process itself.

Basic Scrapy usage

Basic usage isn't documented in the README itself. It defers entirely to the external documentation site rather than showing a first spider, a CLI command, or a minimal working example inline. Anyone evaluating Scrapy from the GitHub page alone has to click through to that documentation before seeing any actual code, since no usage snippet ships in the repository's own README.

Other web scraping options

Common questions about Scrapy

The problem it solves

Scraping more than a handful of pages usually means writing the same boilerplate over and over: send a request, wait, parse the response, retry on failure, save the result somewhere useful. Scrapy exists to take that repeated work off your plate. The project describes itself as a framework for pulling structured data out of websites, not a single script you rewrite for every new source.

Best use cases

• •Building a scraper you expect to maintain for months, not a script you'll throw away after one run, since the project's CI and coverage tooling exist to support ongoing changes.
• •Standardizing how a team writes crawlers across multiple projects instead of every script reinventing request handling and retries from scratch.
• •Working inside a Python codebase already, since Scrapy is a Python-only framework requiring version 3.10 or newer.
• •Pulling structured data at a scale where hand-rolled scripts stop being maintainable, per the project's own framing as an extraction framework rather than a single-page parser.

Who should try it — and who should skip

Try Scrapy if you're a Python developer building a scraper meant to stick around, and you'd rather adopt a maintained framework than assemble retries, parsing, and output handling yourself. Skip it if you need a browser that executes JavaScript to see a page's real content, since the README gives no indication of that capability, or if you just need one page's HTML and don't want to install a framework and read a second documentation site to get there.

Related repositories