Katana: Go Web Crawler for Pipelines
/katana
Katana does one job well: turn a URL or a list of URLs into a clean set of discovered endpoints, fast, without forcing a browser on you unless you ask for one. The standard/headless split is the real value here — plain Go HTTP crawling for most targets, headless Chrome only when JavaScript-rendered content demands it. That flexibility has a cost: headless mode needs a working Chrome install, and captcha solving needs a third-party API key.
What is Katana?
Katana is a command-line crawling and spidering framework from ProjectDiscovery, written in Go, meant to be piped into recon and automation pipelines rather than run as a standalone app. It crawls one or more URLs in either a plain HTTP mode or a headless Chrome mode, follows links, parses JavaScript for endpoints, and writes results to stdout, a file, or JSONL.
Key features of Katana
- ✓Standard mode uses Go's HTTP client for fast, browser-free crawling, while headless mode hooks Chrome directly to render JavaScript-heavy pages.
- ✓JavaScript parsing (-jc) pulls endpoints out of JS files; an optional jsluice-based parser (-jsl) is available but the README flags it as memory-intensive.
- ✓Automatic form filling (-aff) is marked experimental and attempts to complete forms encountered mid-crawl.
- ✓Scope control locks a crawl to a domain, root domain, or FQDN, or accepts custom regex via -crawl-scope and -crawl-out-scope.
- ✓The knowledge base (-kb) downloads an ML model on first use to classify page and form types while crawling.
- ✓Page content similarity filtering runs a second dedup pass after exact-match filtering, choosing between simhash, TF-IDF, or BM25.
- ✓Headless captcha handling hands reCAPTCHA challenges to an external solver like capsolver via -captcha-solver-provider.
- ✓Output goes to stdout, a file, or structured JSONL, with flags to omit raw request/response data or specific fields.
Installing Katana
Katana requires Go 1.26+ to build from source; the README notes that trying the latest available Go version usually fixes install issues, since the minimum version has moved before. The standard path is `CGO_ENABLED=1 go install github.com/projectdiscovery/katana/cmd/katana@latest`. Pre-compiled binaries are also on the GitHub releases page. Docker users run `docker pull projectdiscovery/katana:latest`, then `docker run projectdiscovery/katana:latest -u https://tesla.com` for standard mode, or add `-system-chrome -headless` for headless mode. On Ubuntu, the README's recommended path installs Go via snap and Google Chrome from its apt repo before running go install, since headless mode needs a real Chrome binary.
Basic Katana usage
The base command is `katana -u https://tesla.com` for one target; comma-separate multiple URLs, point `-list` at a file, or pipe URLs in over stdin, e.g. `cat domains | httpx | katana`. `-d` sets crawl depth (default 3), `-c` sets concurrency (default 10), and `-rl` caps requests per second (default 150). Add `-jc` to parse JavaScript files for endpoints, `-headless` to switch to browser-based crawling, and `-j` to write JSONL instead of plain URLs to stdout. `katana -h` prints the full flag list, split into input, configuration, headless, scope, filter, rate-limit, and output sections.
Strengths
- ✓Standard mode has no browser dependency, so it drops into a CI pipeline or a bare Linux box without extra setup.
- ✓Scope control is granular: predefined fields (domain, root domain, FQDN) plus arbitrary regex for both in-scope and out-of-scope filtering.
- ✓Input and output are pipeline-friendly: stdin, file, or URL input; stdout, file, or JSONL output with field-level control.
- ✓Rate limiting has both global and per-host controls (-rl, -hrl, and their per-minute variants), so one crawl doesn't hammer a target.
- ✓MIT license keeps it usable in commercial and internal tooling without licensing friction.
Limitations and known issues
- △Automatic form filling and headless crawling are both marked experimental in the README, not stable, hardened features.
- △jsluice-based JS parsing is explicitly flagged as memory-intensive, which matters when crawling at scale.
- △Headless mode needs a real Chrome install (or katana's own) plus the setup that comes with running a browser in CI or on a server, unlike the dependency-free standard mode.
- △Captcha solving depends on a third-party provider and API key such as capsolver; katana doesn't solve captchas itself.
- △The knowledge base classifier downloads an ML model on first run, an extra dependency and startup step some pipelines won't want.
Katana alternatives
Frequently asked questions
Katana is free and open source, released under the MIT license, so it can be used in commercial and internal tooling without licensing fees.
Katana supports JavaScript rendering through its headless mode, which hooks Chrome directly to render pages and capture browser-rendered endpoints. A separate -js-crawl flag also parses JavaScript files in standard mode without launching a full browser.
Standard mode uses Go's HTTP client to fetch and parse raw responses, which is fast but can't see content added by client-side JavaScript. Headless mode drives a real Chrome browser instead, so it also picks up DOM-rendered and JS-triggered endpoints, at the cost of browser overhead.
Katana can trigger CAPTCHA solving during headless crawls by handing the challenge to an external provider such as capsolver via the -captcha-solver-provider and -captcha-solver-key flags. Katana does not solve CAPTCHAs on its own.
Katana rate-limits crawls with separate global and per-host controls: -rate-limit and -rate-limit-minute cap total requests, -host-rate-limit and -host-rate-limit-minute cap requests to any single host, and -delay adds a fixed pause between requests.
Katana writes results to stdout by default, to a file with -output, or as structured JSONL with -jsonl, and lets you omit raw request/response data or specific fields from the JSONL output to keep files smaller.
The problem it solves
Recon and bug-bounty pipelines usually force a choice: a fast HTTP crawler that misses JavaScript-rendered links and API calls, or a full headless-browser crawler that is slow to run across hundreds of targets. Katana sits between the two, defaulting to a lightweight Go HTTP crawl and only paying the headless-Chrome cost on targets flagged with the -headless flag.
Best use cases
- •Enumerating attack surface for a bug bounty or pentest scope before pointing vulnerability scanners at it.
- •Piping a list of live hosts from a tool like httpx into katana over stdin to crawl each one for endpoints.
- •Crawling JavaScript-heavy single-page apps in headless mode to catch endpoints a plain HTTP crawler would miss.
- •Pulling API endpoints and form fields out of JS bundles for API-surface mapping.
Who should try it — and who should skip
Security teams and bug-bounty hunters running recon pipelines who need fast endpoint discovery across many targets are the clear fit, especially if httpx or a similar tool already feeds target lists in. Developers who just want to scrape a handful of pages for content, without scope-control or headless machinery, will find a plain scraping library simpler. Teams that need captcha-solving or form-filling to be production-stable should wait; the README marks both experimental.
Related repositories
Still deciding about katana?
One click hands the question to an AI along with this page — see what it says about katana.