ScrapeGraphAI: an LLM-based web scraper for Python
/Scrapegraph-ai
ScrapeGraphAI swaps hand-written CSS selectors for an LLM prompt, so a layout change on the target site doesn't necessarily break the pipeline the way it would with Scrapy. The catch: every page costs LLM tokens and returns whatever JSON shape the model feels like producing, better suited to exploratory extraction than a job that has to run identically every night. It earns a spot next to a traditional scraper, not a replacement for one.
What is ScrapeGraphAI?
ScrapeGraphAI is a Python library that pairs an LLM with graph-based pipeline logic to pull structured data out of websites and local files (XML, HTML, JSON, Markdown). Instead of CSS selectors or XPath, you write a plain-language prompt describing what to extract, and a pipeline built as a graph of nodes sends the page content to the LLM and hands back JSON. That graph of steps, not a single scrape call, is where the name comes from.
Core scraping pipelines and what they do
- ✓SmartScraperGraph extracts data from a single page or local file given just a prompt and a source — no selectors to write.
- ✓SearchGraph runs the same extraction across the top results of a search engine query instead of one URL.
- ✓SmartScraperMultiGraph applies one prompt to a list of sources at once, and every pipeline has a matching 'multi' version that runs its LLM calls in parallel.
- ✓SpeechGraph turns a scraped page into an audio file instead of JSON.
- ✓ScriptCreatorGraph and ScriptCreatorMultiGraph generate a standalone Python script that reproduces the scrape, rather than running it live.
- ✓The LLM backend is configurable per pipeline through the graph_config dict — OpenAI, Groq, Azure, Gemini, MiniMax, or a local model via Ollama.
- ✓The same graph pipelines parse local XML, HTML, JSON, and Markdown files, not just live websites.
Installing ScrapeGraphAI
Install from PyPI with `pip install scrapegraphai`. Fetching live websites also needs Playwright's browser binaries, installed separately with `playwright install` — skip that step and you can still parse local files (XML, HTML, JSON, Markdown), just not live URLs. The README recommends a virtual environment to avoid dependency conflicts. You'll also need credentials for whichever LLM the graph_config points at — an API key for OpenAI, Groq, Azure, Gemini, or MiniMax, or a local Ollama install with the model already pulled — none of that ships with the package.
Running your first scrape with SmartScraperGraph
Import SmartScraperGraph from scrapegraphai.graphs, then build a graph_config dict naming the LLM model (`ollama/llama3.2`, or `openai/gpt-4o-mini` with an api_key), plus verbose and headless flags. Instantiate SmartScraperGraph with a prompt describing what to extract, a source URL, and that config, then call `.run()`. The call returns a plain Python dict — the README's own example returns a company description, a list of founders, and social media links pulled from one page — which you can pass straight to `json.dumps()`.
What can you build with it?
- •Pulling structured company data — description, founders, social links — off a single landing page for a lead-gen or research dataset, the exact example in ScrapeGraphAI's own README.
- •Aggregating answers across the top results of a search query with SearchGraph instead of scraping one site at a time.
- •Feeding scraped page content into a RAG pipeline as structured JSON rather than raw HTML.
- •Generating a standalone Python scraping script with ScriptCreatorGraph to hand off scraping logic without a runtime dependency on the library.
- •Converting a scraped article into an audio file via SpeechGraph.
Strengths
- ✓Skips selector-writing entirely — a plain-language prompt reads a page the way a person would, so it survives a layout redesign that would break a CSS-selector scraper.
- ✓Six pipeline types cover more than 'scrape one page': SearchGraph pulls from search results, SmartScraperMultiGraph fans a prompt across a list of URLs, and ScriptCreatorGraph spits out a standalone script instead of running live.
- ✓LLM backend isn't locked in — swap between OpenAI, Groq, Azure, Gemini, MiniMax, or a local Ollama model just by editing the graph_config dict.
- ✓Already documents integrations with LangChain, Llama Index, Crew.ai, Agno, and several no-code platforms (Zapier, n8n, Dify), so it plugs into an existing pipeline instead of demanding a rewrite.
- ✓MIT-licensed, so the open-source library itself is free to self-host and modify.
Limitations of the open-source library
- △Live-site scraping depends on Playwright's browser binaries as a separate install step, not just the pip package.
- △LLM output isn't guaranteed to match a fixed schema every run, since the pipeline asks the model to structure the page rather than parsing it with fixed selectors.
- △Every scrape costs LLM tokens (or local compute for Ollama), on top of whatever the library itself costs to run.
- △Anonymous usage telemetry is on by default. You must explicitly set SCRAPEGRAPHAI_TELEMETRY_ENABLED=false to opt out.
- △The README itself says the library is meant for data exploration and research, not production-scale scraping — for that, the developers point you at their separate paid managed API.
ScrapeGraphAI alternatives
Frequently asked questions
An API key is only needed if you point ScrapeGraphAI at a hosted LLM such as OpenAI, Groq, Azure, Gemini, or MiniMax — you set that provider's key inside the graph_config dict. Running a local model through Ollama instead needs no API key, just Ollama installed with the model already pulled.
The ScrapeGraphAI open-source library is released under the MIT license, so it's free to use, modify, and self-host. You still pay for whichever LLM backend you connect it to — OpenAI or Groq usage, for instance — since the library itself doesn't cover LLM costs. ScrapeGraphAI also sells a separate managed API for teams that don't want to self-host.
ScrapeGraphAI works with OpenAI, Groq, Azure, Gemini, and MiniMax through their APIs, plus local models run through Ollama. You choose the backend by setting the model name inside the graph_config dict passed to each pipeline, with an API key for hosted providers or nothing extra for a local Ollama model.
Traditional scraping with Scrapy or Beautiful Soup is faster and cheaper per page, since it parses HTML directly with no LLM call, but it needs selectors written and maintained per site. ScrapeGraphAI trades that speed for a plain-language prompt, cutting setup time on a site you'll only scrape once or twice.
ScrapeGraphAI documents integrations with both LangChain and Llama Index, alongside Crew.ai, Agno, and CamelAI, so its scraping pipelines can plug into an existing LLM framework instead of running standalone.
Beyond live websites, ScrapeGraphAI can run its extraction pipelines against local XML, HTML, JSON, and Markdown files using the same prompt-based approach it uses for a URL. That makes it usable for structuring existing documents, not just crawling the web.
The problem it solves
Writing and maintaining CSS selectors or XPath for every target site is brittle: a redesign breaks the scraper, and a script tuned to pull 'founder names and social links' off one company page rarely reuses cleanly on the next. ScrapeGraphAI replaces that selector-writing step with a natural-language prompt an LLM reads against the page, so the same prompt can point at a different site's markup without a rewrite.
Who should try it — and who should skip
Developers who already have an LLM subscription and are prototyping an extraction job — pulling structured data off a handful of pages, feeding a RAG pipeline, or generating a one-off dataset — get the most out of ScrapeGraphAI, since writing a prompt is faster than reverse-engineering a page's DOM. Teams running scrapers against thousands of pages on a schedule, where token cost and output-schema drift matter, are better off with a selector-based scraper like Scrapy, or with ScrapeGraphAI's own managed API, which handles proxies and anti-bot measures for a per-credit fee.
Related repositories
Want a second opinion on Scrapegraph-ai?
Ask an AI that can read this page — one click and you get its take on Scrapegraph-ai.