Colly: Go Web Scraper and Crawler Framework
/colly
Colly is worth adopting the moment you're writing a scraper in Go and don't want to hand-roll cookie handling, rate limiting, and robots.txt checks on top of net/http. The callback API (OnHTML, OnRequest) stays out of your way for straightforward HTML scraping, but it has no headless browser, so JavaScript-heavy targets will leave you reaching for a second tool anyway.
What is Colly?
Colly is a scraping and crawling framework for Go that wraps net/http and HTML parsing behind a callback API. You create a Collector, register handlers like OnHTML for elements and OnRequest for outgoing calls, then call Visit on a starting URL and let Colly walk links from there. Cookie handling, session state, and non-Unicode encoding are managed automatically.
Core features of Colly
- ✓Callback-based API: register OnHTML, OnRequest, and similar handlers instead of writing manual parse loops.
- ✓Request delay and per-domain concurrency limits, built in.
- ✓Automatic cookie and session handling across requests.
- ✓Sync, async, and parallel scraping modes on the same Collector.
- ✓Response caching so re-crawls skip pages you already fetched.
- ✓Automatic encoding of non-Unicode responses.
- ✓Robots.txt support.
- ✓Distributed scraping and configuration via environment variables.
What you can build with Colly
- •Structured data extraction for data mining or processing pipelines — the use case the README leads with.
- •Niche search indexing, the way jivesearch/jivesearch built a privacy-focused search engine on Colly (per the README's adopter list).
- •Domain-specific trackers, like gamedb/gamedb's Steam games database and Leagify/colly-draft-prospects' NFL draft prospect scraper.
- •Small CLI scraping tools, such as lawzava/scrape for pulling emails from a target site.
- •Site cloning or archiving, like imthaghost/goclone, which copies a website to your machine.
Installing and using Colly
A minimal Colly program looks like this, straight from the README: ```go import ( "fmt" "github.com/gocolly/colly/v2" ) func main() { c := colly.NewCollector() // Find and visit all links c.OnHTML("a[href]", func(e *colly.HTMLElement) { e.Request.Visit(e.Attr("href")) }) c.OnRequest(func(r *colly.Request) { fmt.Println("Visiting", r.URL) }) c.Visit("http://go-colly.org/") } ``` `NewCollector()` sets up the crawler, `OnHTML` reacts to matched elements (here, every `a[href]`), `OnRequest` fires before each request goes out, and `Visit` kicks off the crawl from a starting URL. The README points to its `_examples` folder on GitHub for more detailed patterns.
Strengths
- ✓One `go get` away — no separate scraping server, browser binary, or language runtime to install.
- ✓Cookies, sessions, and non-Unicode encoding are handled automatically instead of by hand.
- ✓Sync, async, and parallel modes scale a crawl without switching frameworks.
- ✓Robots.txt support and per-domain rate limiting are built in, not bolted on.
- ✓Apache-2.0 licensing keeps it usable in commercial projects without copyleft strings attached.
Limitations and trade-offs
- △No headless browser — JavaScript-rendered pages aren't handled; the feature list covers HTML parsing and caching, not script execution.
- △Proxy rotation isn't a documented Colly feature. The README's only proxy-related content is a sponsor ad for a third-party proxy provider, not a Colly API you call.
- △Distributed scraping is listed as a feature with no configuration detail in the README — you're on your own for how to wire it up.
- △The Bugs section still points people to '#colly on freenode,' an IRC network that's been dead for years, which suggests parts of the README are stale.
Alternatives to Colly
Frequently asked questions
Colly ships with request delays and per-domain concurrency limits built in, which are the two controls large crawls need most. The README cites throughput over 1k requests/sec on a single core, but gives no scale benchmarks beyond that, so test against your actual target before committing to production traffic.
Colly handles rate limiting natively through request delays and maximum concurrency per domain. Proxy rotation isn't part of its documented feature set — the README's only proxy content is a sponsor placement for a third-party proxy provider, not a Colly API, so you'd need to wire proxies in yourself.
Colly cannot render JavaScript on its own. Its feature list covers HTML parsing, cookies, and caching but no headless browser or script execution, so JavaScript-heavy pages need a separate rendering tool paired with it.
Colly is free to use in commercial projects. It's released under the Apache-2.0 license, which allows commercial use without copyleft obligations.
Colly lists robots.txt support as a built-in feature, so a Collector can respect a site's robots.txt rules without extra code. The README doesn't document configuration options beyond naming the feature, so check the source or examples for specifics.
Colly's main difference is scope: it wraps net/http and HTML parsing behind a single callback API (OnHTML, OnRequest) plus built-in cookies, caching, and robots.txt support, so you're not assembling those pieces from separate packages yourself.
The problem it solves
Writing a scraper directly against Go's net/http means handling cookies, per-domain rate limiting, robots.txt parsing, and non-UTF-8 encoding by hand before you get to the actual data extraction. Colly packages that plumbing behind a callback API, so a Go developer's crawler code starts at OnHTML instead of at connection management.
How to install / try
Install Colly with `go get github.com/gocolly/colly/v2`. That's the only install step the README documents — there's no separate binary or service to run.
Who should try it — and who should skip
Go developers who already have a service written in Go and need to add scraping without pulling in Python or a headless browser should try Colly — the callback API mirrors patterns you already know from HTTP middleware. Skip it if your targets are JavaScript-heavy single-page apps: Colly can't render them, so you'd end up pairing it with a browser tool anyway, which erases the simplicity it offers in the first place.
Related repositories
Is colly worth your time?
ChatGPT, Claude and Perplexity can all read this page. Ask one of them what it makes of colly.