scrapinghub/frontera is tracked by TopGit as an open-source project, with 1.3k stars on GitHub, written primarily in Python. A scalable frontier for web crawlers
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
Frontera is a web crawling framework consisting of crawl frontier, and distribution/scaling primitives, allowing to build a large scale online web crawler.
Frontera takes care of the logic and policies to follow during the crawl. It stores and prioritises links extracted by
the crawler to decide which pages to visit next, and capable of doing it in distributed manner.
Main features
Online operation: small requests batches, with parsing done right after fetch.
Pluggable backend architecture: low-level backend access logic is separated from crawling strategy.
Two run modes: single process and distributed.
Built-in SqlAlchemy, Redis and HBase backends.
Built-in Apache Kafka and ZeroMQ message buses.
Built-in crawling strategies: breadth-first, depth-first, Discovery (with support of robots.txt and sitemaps).
Battle tested: our biggest deployment is 60 spiders/strategy workers delivering 50-60M of documents daily for 45 days, without downtime,
Transparent data flow, allowing to integrate custom components easily using Kafka.
Message bus abstraction, providing a way to implement your own transport (ZeroMQ and Kafka are available out of the box).
Optional use of Scrapy for fetching and parsing.
3-clause BSD license, allowing to use in any commercial product.
Python 3 support.
Installation
$ pip install frontera
Documentation
Main documentation at RTD
EuroPython 2015 slides
BigDataSpain 2015 slides
Community
Join our Google group at https://groups.google.com/a/scrapinghub.com/forum/#!forum/frontera or check GitHub issues and
pull requests.
TopGit's last sync did not record any GitHub topics for scrapinghub/frontera. GitHub topics appear in the right sidebar of a repository page; that's the authoritative place to check.
How active is development on scrapinghub/frontera?
The most recent commit recorded on scrapinghub/frontera was 1.2 years ago, based on the GitHub push timestamp. The repository has 213 forks — one of the better signals of community interest.
How many stars does scrapinghub/frontera have?
scrapinghub/frontera has 1.3k GitHub stars — refresh the page for the live number, or check github.com/scrapinghub/frontera. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
What is scrapinghub/frontera?
scrapinghub/frontera (scrapinghub/frontera) is a Python project on GitHub. From the project's own README: A scalable frontier for web crawlers
What language is scrapinghub/frontera written in?
scrapinghub/frontera is written primarily in Python. GitHub's language field is based on the largest share of bytes in the default branch.
