MediaCrawler: Social Media Data Collection Tool

What is MediaCrawler?

MediaCrawler is an open-source Python tool that automates public-data collection from Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu. It covers keyword search, single-post lookup by ID, nested second-level comments, and creator-homepage crawling for each platform, logging in through a real Chrome session via Playwright's CDP mode so it can read signature parameters without decrypting each site's algorithm.

Key Data Collection Features

• ✓Keyword search, post-ID lookup, nested second-level comments, and creator-homepage crawling — the README's feature table lists all four as supported on all seven platforms.
• ✓Login-state caching and an IP proxy pool, also checked off for every platform in that same table.
• ✓Comment word-cloud generation per platform, a specific output format the README calls out alongside the crawl modes.
• ✓CDP mode by default: MediaCrawler connects to a Chrome browser (version 144 or newer) you already have open and reuses its existing cookies and login session, which the README says lowers platform risk-control detection compared to a fresh automated browser.
• ✓No JS reverse engineering of each platform's signing algorithm — signature parameters are read through JS expressions run inside that already-logged-in browser context instead.
• ✓Six storage options — CSV, JSON, JSONL, Excel, SQLite, or MySQL — documented further in the repo's data_storage_guide.md.
• ✓An optional WebUI (a FastAPI backend plus a Vite frontend) for setting platform, login method, and crawl type, watching logs live, and previewing or exporting data without touching the CLI.

Setting Up MediaCrawler

MediaCrawler's recommended path uses uv: install uv, then Node.js 16.0.0 or newer (the README lists it as a prerequisite), cd into the MediaCrawler directory, and run uv sync to install the Python dependencies. Browser drivers are optional — uv run playwright install is only needed if you plan to run standard Playwright mode instead of the default. For that default CDP mode, install Chrome 144 or newer, open chrome://inspect/#remote-debugging, check "Allow remote debugging for this browser instance," and confirm the page shows "Server running at: 127.0.0.1:9222." The README also documents a plain pip path — python -m venv venv, activate it, pip install -r requirements.txt, then playwright install — but explicitly marks it "not recommended," and notes the bundled requirements.txt was built against Python 3.11.

Running Your First Crawl

Crawls run through main.py with a --platform flag: uv run main.py --platform xhs --lt qrcode --type search reads keywords from config/base_config.py and crawls matching posts and comments, while --type detail pulls a specific list of post IDs instead. Each run needs a QR-code scan in the corresponding app to log in, and the first time you run it under CDP mode, Chrome pops a confirmation dialog you have to accept within 60 seconds. Run uv run main.py --help to see the flags for the other six platforms. Comment crawling is off by default — flip ENABLE_GET_COMMENTS in config/base_config.py to turn it on. The WebUI is an alternative to all of this: start the API server with uv run uvicorn api.main:app --port 8080 --reload, run the frontend separately with npm run dev during development, then configure and watch crawls from http://localhost:5173 instead of editing config files by hand.

Exploring MediaCrawlerPro and Other Tools

Common Questions About MediaCrawler

The problem it solves

Pulling public posts and comments off Chinese social platforms like Xiaohongshu or Douyin usually means reverse-engineering each site's request-signing algorithm by hand, since none of them expose a public API for this data. MediaCrawler's own README frames its whole approach around avoiding that step — it borrows the signature from an already logged-in browser context instead of cracking the encryption.

Best use cases

• •Pulling keyword-search results and their nested comments from one of the seven supported platforms for a research or content-analysis project.
• •Grabbing all posts and comments from a specific creator's homepage on one of the seven platforms, instead of searching by keyword.
• •Archiving a fixed list of post IDs and their comment threads using the --type detail path instead of --type search.
• •Running crawls through the WebUI instead of the CLI, to configure platform, login method, and crawl type visually and watch logs or export data without editing config files.

Who should try it — and who should skip

Reach for MediaCrawler if you specifically need public post and comment data from Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, or Zhihu, you're doing it for learning or research (per the README's own usage restriction), and you're fine keeping a real Chrome window open with remote debugging on while it runs. Skip it if you need scraping for a commercial product — the README explicitly prohibits that — or if you want a fully headless, unattended pipeline with resume support, since those are the features pushed toward the separate paid MediaCrawlerPro.

Related repositories