Crawlee: web scraping and browser automation
/crawlee
Crawlee is worth setting up once a scraping job outgrows a single script — it wraps Playwright, Puppeteer, and Cheerio behind one interface and handles the request queue, retries, and proxy rotation you'd otherwise write by hand. It's an orchestration layer on those browser tools, not a replacement for them, so you still write the page-specific selectors yourself. One static page doesn't need this much machinery.
What is Crawlee?
Crawlee is an open-source Node.js and TypeScript library for web scraping and browser automation, built by Apify. It gives you one crawler interface across Playwright, Puppeteer, Cheerio, JSDOM, and raw HTTP requests, plus a persistent request queue, integrated proxy rotation, and pluggable storage for the data and files a crawl collects.
Key features for web scraping and automation
- ✓One crawler interface that works across PlaywrightCrawler, PuppeteerCrawler, CheerioCrawler, and JSDOM, so switching between a full browser and plain HTTP doesn't mean rewriting your data-extraction code.
- ✓A persistent request queue that supports both breadth-first and depth-first crawling and survives a restart.
- ✓Proxy rotation and session management built into the crawler config instead of a separate library.
- ✓Automatic concurrency scaling based on available system resources.
- ✓Zero-config generation of browser-like headers and TLS fingerprints, even in plain HTTP mode.
- ✓A CLI (npx crawlee create) that scaffolds a working project with boilerplate code.
- ✓Pluggable storage for both structured data (Dataset) and downloaded files.
- ✓Dockerfiles included in the repo, ready to deploy.
Getting started with Crawlee
Crawlee needs Node.js 16 or higher. The fastest path is the Crawlee CLI: `npx crawlee create my-crawler`, then `cd my-crawler` and `npm start` — it installs dependencies and drops in boilerplate for you. To add Crawlee into an existing project instead, run `npm install crawlee playwright` (Playwright isn't bundled, to keep install size down, so you add it yourself when you need a browser). Pre-release builds are available too: `npm install crawlee@next`.
Writing your first crawler
A minimal crawler imports PlaywrightCrawler and Dataset from `crawlee`, defines a `requestHandler` that runs for each page, and calls `crawler.run()` with a starting URL. Inside the handler you read the page (e.g. `page.title()`), save results with `Dataset.pushData()`, and call `enqueueLinks()` to add the links found on that page to the queue. By default, everything lands in `./storage` in your working directory; the storage location is configurable.
Strengths
- ✓Swapping HTTP scraping for full browser automation is a one-line change (CheerioCrawler to PlaywrightCrawler), since both share the same request-handler shape.
- ✓Proxy rotation, session management, and a persistent queue are built in — you don't assemble them from separate packages.
- ✓The CLI gets a working project running in two commands.
- ✓Written in TypeScript with generics, so request and page types are checked, not just documented.
- ✓Dockerfiles are already in the repo, so deploying a crawler doesn't start from a blank Dockerfile.
When Crawlee may not be the right choice
- △Playwright isn't bundled — you install it separately, and it pulls in real browser binaries, adding real install size to a project that only needed a browser occasionally.
- △Node.js 16+ is a hard requirement, so it's a non-starter on an older runtime without an upgrade first.
- △Default storage is the local filesystem (`./storage`); using it in a serverless or ephemeral environment means configuring an alternative storage backend yourself.
- △Most of the real documentation — guides, configuration details, examples — lives on the separate crawlee.dev site rather than in the repo's README.
Crawlee vs other scraping options
Frequently asked questions
Crawlee supports both Playwright and Puppeteer through PlaywrightCrawler and PuppeteerCrawler, which share the same request-handler interface. That means you can pick whichever browser automation library you already know, or switch between them mid-project, without rewriting how pages get scraped.
Crawlee handles both out of the box — proxy rotation and session management are built into the crawler configuration rather than requiring a separate library, so requests get spread across proxies and sessions automatically as a crawl runs.
Crawlee requires Node.js 16 or higher. That's stated directly in the installation instructions, and it applies whether you set up a project with the Crawlee CLI or add Crawlee manually via npm install.
Crawlee stores scraped data through the Dataset API — calling Dataset.pushData() inside a request handler saves results as JSON. By default that data lands in ./storage/datasets/default in your working directory, though the storage location is configurable.
Crawlee is licensed under Apache-2.0, a permissive open-source license. That covers commercial use, modification, and redistribution without paying anything for the library itself — the same terms apply whether Crawlee runs in a side project or a production scraper you sell access to.
Crawlee ships with Dockerfiles in the repo, so containerizing a crawler doesn't start from scratch. It also runs on the Apify platform, which is built by the same team and offers cloud deployment for Crawlee projects.
The problem it solves
Scraping a real website means juggling several separate concerns at once: a request queue that survives a restart, proxy rotation so requests don't get blocked, browser fingerprints that don't look automated, and a way to switch between plain HTTP and a full headless browser depending on the page. Crawlee bundles those into one library instead of leaving each project to assemble them from separate packages.
Best use cases
- •Scraping JavaScript-rendered pages (product listings, prices, dynamic content) with PlaywrightCrawler or PuppeteerCrawler.
- •Fast bulk crawling of static or JSON-serving pages with CheerioCrawler, skipping the overhead of a real browser.
- •Building a dataset for AI, LLM, or RAG pipelines — the README lists this explicitly as a use case.
- •Downloading files like PDFs, JPGs, and PNGs discovered during a crawl, not just HTML.
- •Crawls that need to survive basic bot detection via rotating proxies and human-like browser fingerprints.
Who should try it — and who should skip
Try Crawlee if you're already writing scraping code in Node.js or TypeScript and keep re-solving the same problems — request queues, retries, proxy rotation, switching between HTTP and a real browser per site. Skip it if your stack is Python (scrapy or Scrapling cover the same ground there) or if the whole job is one script hitting one static page, where pulling in a crawler framework is more setup than the task needs.
Related repositories
Still deciding about crawlee?
One click hands the question to an AI along with this page — see what it says about crawlee.