scrapy/scrapely được TopGit xếp vào nhóm dự án mã nguồn mở, với 1.9k sao trên GitHub, viết chủ yếu bằng HTML. A pure-python HTML screen-scraping library
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
Scrapely is a library for extracting structured data from HTML pages. Given
some example web pages and the data to be extracted, scrapely constructs a
parser for all similar pages.
Overview
Scrapinghub wrote a nice blog post_ explaining how scrapely works and how it's used in Portia_.
Scrapely works in Python 2.7 or 3.3+.
It requires numpy and w3lib Python packages.
To install scrapely on any platform use::
pip install scrapely
If you're using Ubuntu (9.10 or above), you can install scrapely from the
Scrapy Ubuntu repos. Just add the Ubuntu repos as described here:
http://doc.scrapy.org/en/latest/topics/ubuntu.html
And then install scrapely with::
aptitude install python-scrapely
Usage (API)
Scrapely has a powerful API, including a template format that can be edited
externally, that you can use to build very capable scrapers.
What follows is a quick example of the simplest possible usage, that you can
run in a Python shell.
Start by importing and instantiating the Scraper class::
>>> from scrapely import Scraper
>>> s = Scraper()
Then, proceed to train the scraper by adding some page and the data you expect
to scrape from there (note that all keys and values in the data you pass must
be strings)::
Finally, tell the scraper to scrape any other similar page and it will return
the results::
>>> url2 = 'http://pypi.python.org/pypi/Django/1.3'
>>> s.scrape(url2)
[{u'author': [u'Django Software Foundation <foundation at djangoproject com>'],
u'description': [u'A high-level Python Web framework that encourages rapid development and clean, pragmatic design.'],
u'name': [u'Django 1.3']}]
That's it! No xpaths, regular expressions, or hacky python code.
Usage (command line tool)
There is also a simple script to create and manage Scrapely scrapers.
It supports a command-line interface, and an interactive prompt. All commands
supported on interactive prompt are also supported in the command-line
interface.
To enter the interactive prompt type the following without arguments::
python -m scrapely.tool myscraper.json
Example::
$ python -m scrapely.tool myscraper.json
scrapely> help
Documented commands (type help <topic>):
========================================
a al s ta td tl
scrapely>
To create a scraper and add a template::
scrapely> ta http://pypi.python.org/pypi/w3lib/1.1
[0] http://pypi.python.org/pypi/w3lib/1.1
This is equivalent as typing the following in one command::
python -m scrapely.tool myscraper.json ta http://pypi.python.org/pypi/w3lib/1.1
Scrapely is created and maintained by the Scrapy group, so you can get help
through the usual support channels described in the Scrapy community_ page.
Architecture
Unlike most scraping libraries, Scrapely doesn't work with DOM trees or xpaths
so it doesn't depend on libraries such as lxml or libxml2. Instead, it uses
an internal pure-python parser, which can accept poorly formed HTML. The HTML is
converted into an array of token ids, which is used for matching the items to
be extracted.
Scrapely extraction is based upon the Instance Based Learning algorithm [1]_
and the matched items are combined into complex objects (it supports nested and
repeated objects), using a tree of parsers, inspired by A Hierarchical
Approach to Wrapper Induction [2]_.
.. [1] Yanhong Zhai , Bing Liu, Extracting Web Data Using Instance-Based Learning, World Wide Web, v.10 n.2, p.113-132, June 2007 <http://portal.acm.org/citation.cfm?id=1265174>_
.. [2] Ion Muslea , Steve Minton , Craig Knoblock, A hierarchical approach to wrapper induction, Proceedings of the third annual conference on Autonomous Agents, p.190-197, April 1999, Seattle, Washington, United States <http://portal.acm.org/citation.cfm?id=301191>_
Known Issues
The training implementation is currently very simple and is only provided for
references purposes, to make it easier to test Scrapely and play with it. On
the other hand, the extraction code is reliable and production-ready. So, if
you want to use Scrapely in production, you should use train() with caution and
make sure it annotates the area of the page you intended.
Alternatively, you can use the Scrapely command line tool to annotate pages,
which provides more manual control for higher accuracy.
How does Scrapely relate to Scrapy_?
Despite the similarity in their names, Scrapely and Scrapy_ are quite
different things. The only similarity they share is that they both depend on
w3lib, and they are both maintained by the same group of developers (which
is why both are hosted on the same Github account).
Scrapy is an application framework for building web crawlers, while Scrapely is
a library for extracting structured data from HTML pages. If anything, Scrapely
is more similar to BeautifulSoup_ or lxml_ than Scrapy.
Scrapely doesn't depend on Scrapy nor the other way around. In fact, it is
quite common to use Scrapy without Scrapely, and viceversa.
If you are looking for a complete crawler-scraper solution, there is (at least)
one project called Slybot_ that integrates both, but you can definitely use
Scrapely on other web crawlers since it's just a library.
Scrapy has a builtin extraction mechanism called selectors_ which (unlike
Scrapely) is based on XPaths.
License
Scrapely library is licensed under the BSD license.
Trang TopGit này là một snapshot — tab "Readme" hiển thị nguyên văn README của repo (đã bỏ link, giữ ảnh). Repo GitHub ở github.com/scrapy/scrapely là nguồn chính thức.
scrapy/scrapely có phải mã nguồn mở không?
TopGit chưa ghi nhận license cho scrapy/scrapely. Phần lớn repo public trên GitHub là mã nguồn mở, nhưng điều khoản khác nhau từng repo — mở file LICENSE để xác nhận.
scrapy/scrapely là gì?
scrapy/scrapely (scrapy/scrapely) là dự án HTML trên GitHub. Theo mô tả gốc: A pure-python HTML screen-scraping library
Đọc đầy đủ README ở tab phía trên.
Vẫn đang phân vân về scrapely?
Một cú bấm sẽ gửi câu hỏi kèm trang này cho AI — xem AI nói gì về scrapely.