Snapshot của rmax/scrapy-redis: 5.6k★ · Python · Data. Redis-based components for Scrapy.
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
You can start multiple spider instances that share a single redis queue.
Best suitable for broad multi-domain crawls.
Distributed post-processing
Scraped items gets pushed into a redis queued meaning that you can start as
many as needed post-processing processes sharing the items queue.
Scrapy plug-and-play components
Scheduler + Duplication Filter, Item Pipeline, Base Spiders.
In this forked version: added json supported data in Redis
data contains url, meta and other optional parameters. meta is a nested json which contains sub-data.
this function extract this data and send another FormRequest with url, meta and addition formdata.
this data can be accessed in scrapy spider through response.
like: request.url, request.meta, request.cookies
.. note:: This features cover the basic case of distributing the workload across multiple workers. If you need more features like URL expiration, advanced URL prioritization, etc., we suggest you to take a look at the Frontera_ project.
Requirements
Python 3.7+
Redis >= 5.0
Scrapy >= 2.0
redis-py >= 4.2
Installation
From pip
.. code-block:: bash
pip install scrapy-redis
From GitHub
.. code-block:: bash
git clone https://github.com/darkrho/scrapy-redis.git
cd scrapy-redis
python setup.py install
.. note:: For using this json supported data feature, please make sure you have not installed the scrapy-redis through pip. If you already did it, you first uninstall that one.
.. code-block:: bash
pip uninstall scrapy-redis
Settings
Idle Redis queue polling can be reduced with opt-in, deadline-gated exponential
backoff:
REDIS_IDLE_BACKOFF_MIN (default 1.0) is the initial delay in seconds.
REDIS_IDLE_BACKOFF_MAX (default 30.0) caps the delay in seconds.
REDIS_IDLE_BACKOFF_FACTOR (default 2.0) multiplies the delay after
each empty poll.
When enabled, the spider never blocks the reactor: while waiting for the next
poll deadline it skips all Redis calls. The close deadline is still checked on
every idle callback. The tradeoff is that new work may take up to the
backoff cap to be picked up, while idle Redis load is reduced when many
workers share a queue.
Connection pooling
Components of one crawler share a single Redis connection pool when using the
default client class and plain connection parameters. Pools are scoped per
``Settings`` object (crawler). Configurations with custom client classes or
client-only parameters (``ssl``, ``unix_socket_path``,
``single_connection_client``) keep the previous per-component behavior.
* ``REDIS_MAX_CONNECTIONS``: Maximum connections in the shared pool (default:
no override is applied — redis-py's own pool default, effectively unlimited
on redis-py 4.x/5.x). It raises ``ConnectionError`` when exhausted; supply a
``BlockingConnectionPool`` via ``REDIS_PARAMS.connection_pool``
(``REDIS_PARAMS["connection_pool"]``) for waiting semantics.
Connection setup, protocols, and timeouts
redis-py 8 changed its default connection protocol to RESP3. Selecting a
protocol changes connection setup only; it does not remove timeout risk from
warm operations on already-established connections.
The REDIS_PROTOCOL setting defaults to None, leaving redis-py's
installed-version default unchanged.
On redis-py >= 5, the recommended, discoverable form for opting into RESP2 is:
.. code-block:: python
REDIS_PROTOCOL = 2
The existing equivalent form remains supported:
.. code-block:: python
REDIS_PARAMS = {"protocol": 2}
Both forms require redis-py >= 5. REDIS_PROTOCOL takes precedence when
both settings specify a protocol. As optional advanced tuning on redis-py >=
5, you can also omit redis-py's connection identification metadata:
.. code-block:: python
REDIS_PARAMS = {"driver_info": None}
This can reduce connection-setup work, but loses useful client-identification
metadata in exchange.
The library's historical retry_on_timeout=True default is deprecated and
has no effect on redis-py >= 6. On those versions, the effective retry policy
comes from redis-py's retry object and its defaults. Measure the resulting
behavior before lowering retries, especially under connection churn or
saturation.
In testing with Redis 8.0.2, CLIENT MAINT_NOTIFICATIONS was rejected during
RESP3 connection setup. This observation is scoped to the tested Redis 8.0.2
version. The rejection is not shown by MONITOR or commandstats; inspect
client logs and Redis's total_error_replies counter for diagnostics.
The connection-related behavior by redis-py version is summarized below:
.. list-table::
:header-rows: 1
redis-py
Connection setup notes
4.2–4.x
No protocol or driver_info knobs.
5.x
Both knobs are available; RESP2 is the default.
6.x–7.x
RESP2 is the default; retry semantics changed in 6.x, and
CLIENT MAINT_NOTIFICATIONS is used on explicit RESP3 from 7.x.
8.x
RESP3 is the default.
Alternative Choice
Frontera_ is a web crawling framework consisting of crawl frontier_, and distribution/scaling primitives, allowing to build a large scale online web crawler.
Trang TopGit này là một snapshot — tab "Readme" hiển thị nguyên văn README của repo (đã bỏ link, giữ ảnh). Repo GitHub ở github.com/rmax/scrapy-redis là nguồn chính thức.
rmax/scrapy-redis có bao nhiêu sao?
rmax/scrapy-redis có 5.6k sao GitHub — tải lại trang để xem số mới nhất, hoặc xem trực tiếp github.com/rmax/scrapy-redis. TopGit phản chiếu số sao của GitHub nhưng không cam kết đến từng phút.
rmax/scrapy-redis có những chủ đề gì?
GitHub topics của rmax/scrapy-redis: "crawler", "distributed", "redis", "scrapy". TopGit xếp repo vào nhóm Data.
rmax/scrapy-redis có trang demo không?
Dự án có trang chủ ở http://scrapy-redis.readthedocs.io. Tab "Readme" ở trang này thường có ảnh chụp và hướng dẫn bắt đầu nhanh.
rmax/scrapy-redis còn đang phát triển không?
Commit gần nhất trên rmax/scrapy-redis là 6 tháng trước (theo timestamp GitHub). Repo có 1.6k fork — một chỉ báo về mức độ quan tâm của cộng đồng.
rmax/scrapy-redis dùng license gì?
rmax/scrapy-redis phát hành theo license MIT. Nên mở file LICENSE trên GitHub để xác nhận — license metadata đôi khi lệch với thực tế dự án.
rmax/scrapy-redis viết bằng ngôn ngữ gì?
rmax/scrapy-redis chủ yếu viết bằng Python. Trường "language" của GitHub dựa trên phần lớn byte ở nhánh mặc định.
Đọc đầy đủ README ở tab phía trên.
scrapy-redis có đáng để bạn bỏ thời gian?
ChatGPT, Claude và Perplexity đều đọc được trang này. Hỏi thử xem họ nghĩ gì về scrapy-redis.