Điểm qua scrapinghub/article-extraction-benchmark: 378 sao trên GitHub, viết chủ yếu bằng Python. Article extraction benchmark: dataset and evaluation scripts
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
Article extraction benchmark: open-source libraries and commercial services
We evaluate the quality of article body
extraction for commercial services
Zyte Automatic Extraction (ours) <https://www.zyte.com/data-types/news-scraping-api/>,
Diffbot <https://www.diffbot.com/>
and open-source libraries
newspaper4k <https://github.com/AndyTheFactory/newspaper4k>,
readability-lxml <https://github.com/buriy/python-readability>,
dragnet <https://github.com/dragnet-org/dragnet>,
boilerpipe <https://github.com/misja/python-boilerpipe>,
html-text <https://github.com/TeamHG-Memex/html-text>,
trafilatura <https://github.com/adbar/trafilatura>,
go-trafilatura <https://github.com/markusmobius/go-trafilatura>,
go-readability <https://github.com/go-shiori/go-readability>,
readeck/go-readability <https://codeberg.org/readeck/go-readability>,
Readability.js <https://github.com/mozilla/readability>,
Go-DomDistiller <https://github.com/markusmobius/go-domdistiller>.
news-please <https://github.com/fhamborg/news-please>.
Goose3 <https://github.com/goose3/goose3>,
inscriptis <https://github.com/weblyzard/inscriptis>,
html2text <https://github.com/Alir3z4/html2text>,
jusText <https://github.com/miso-belica/jusText>,
BeautifulSoup <https://www.crummy.com/software/BeautifulSoup/bs4/doc/>.
Rust crates:
august <https://crates.io/crates/august>,
boilerpipe <https://crates.io/crates/boilerpipe>,
dom_smoothie <https://crates.io/crates/dom_smoothie>,
fast_html2md <https://crates.io/crates/fast_html2md>,
htmd <https://crates.io/crates/htmd>,
html2md-rs <https://crates.io/crates/html2md-rs>,
html2text <https://crates.io/crates/html2text>,
llm_readability <https://crates.io/crates/llm_readability>,
mdka <https://crates.io/crates/mdka>,
nanohtml2text <https://crates.io/crates/nanohtml2text>,
readability <https://crates.io/crates/readability>,
readable-readability <https://crates.io/crates/readable-readability>,
readabilityrs <https://crates.io/crates/readabilityrs>,
rs_trafilatura <https://github.com/Murrough-Foley/rs-trafilatura>_.
We release evaluation datasets and scripts,
and provide more details in a whitepaper.
Article extraction is a task of extracting certain fields of an article
(e.g. news or blog post), such as headline, article body, publication date,
authors, etc. Article extraction systems must work on any web-site.
Here we evaluate only the article body field, as this is one of the most important fields
and one of the hardest to get right.
Below you can find more details about the packages and result reproduction.
More details
More details are available:
In the whitepaper at https://www.zyte.com/whitepaper-ebook/in-depth-analysis-and-evaluation-on-the-quality-of-article-body-extraction/
In a technical report attached to the v1.0.0 release at
https://github.com/scrapinghub/article-extraction-benchmark/releases/tag/v1.0.0
Installation
Clone this repo, and use Python 3.6+.
Evaluation does not require any dependencies.
Dependencies listed in requirements.txt are only for re-generating
output files for open-source article extraction libraries.
See below for their installation details.
Data
JSON data format: a dictionary which maps item ids to dictionaries,
with the following fields:
articleBody: text of the article
url: page url (optional)
All files should have the same keys.
Ground truth is in ground-truth.json,
predictions from different systems is in output/*.json files.
Prediction files in output/*.json may optionally be wrapped to include the
system/library version used to generate them::
HTML files are in html folder. They were fetched with Splash headless
browser with JS disabled by default. They are gzip-compressed and utf-8 encoded.
Screenshots of all pages are not in the repo, they are available on github
in the "Releases" section: https://github.com/scrapinghub/article-extraction-benchmark/releases
Open-source libraries
In addition to benchmarking AutoExtract and Diffbot services, we also benchmark several
open-source libraries that work directly on HTML files without a need for rendering
or external resources:
inscriptis: https://github.com/weblyzard/inscriptis -
converts HTML to text with a particular emphasis on nested tables
html2text: https://github.com/Alir3z4/html2text -
converts HTML pages to Markup language
jusText: https://github.com/miso-belica/jusText -
Heuristic based boilerplate removal tool
BeautifulSoup: https://www.crummy.com/software/BeautifulSoup/bs4/doc/ -
Python library for pulling data out of HTML and XML files.
Rust crates
- august (MIT): https://crates.io/crates/august
- boilerpipe (MIT): https://crates.io/crates/boilerpipe
- dom_smoothie (MIT): https://crates.io/crates/dom_smoothie
- fast_html2md (MIT): https://crates.io/crates/fast_html2md
- htmd (Apache-2.0): https://crates.io/crates/htmd
- html2md-rs (MIT): https://crates.io/crates/html2md-rs
- html2text (MIT): https://crates.io/crates/html2text
- llm_readability (MIT): https://crates.io/crates/llm_readability
- mdka (Apache-2.0): https://crates.io/crates/mdka
- nanohtml2text (MIT): https://crates.io/crates/nanohtml2text
- readability (MIT): https://crates.io/crates/readability
- readable-readability (MIT): https://crates.io/crates/readable-readability
- readabilityrs (Apache-2.0): https://crates.io/crates/readabilityrs
- rs_trafilatura (MIT OR Apache-2.0): https://github.com/Murrough-Foley/rs-trafilatura
Output from these libraries is already present in the repo in ``output/*.json`` files.
They were generated with ``extractors/run_*.py`` files.
You can re-generate output JSON files with:
python3 -m venv ./venv
source ./venv/bin/activate
make run-all
This will install Python dependencies from ``requirements.txt`` into a
`virtual environment <https://docs.python.org/3/library/venv.html>`_
Evaluation
----------
For evaluation, run::
python3 evaluate.py
We report precision, recall, F1, accuracy and their standard deviation estimated with bootstrap.
Please refer to the technical report for more details.
License
-------
License is MIT.
Đọc thêm về scrapinghub/article-extraction-benchmark ở đâu?
Trang TopGit này là một snapshot — tab "Readme" hiển thị nguyên văn README của repo (đã bỏ link, giữ ảnh). Repo GitHub ở github.com/scrapinghub/article-extraction-benchmark là nguồn chính thức.
scrapinghub/article-extraction-benchmark có phải mã nguồn mở không?
Có — scrapinghub/article-extraction-benchmark phát hành theo license MIT, nghĩa là mã nguồn mở để đọc, fork và (tùy license) tái sử dụng. Mã: github.com/scrapinghub/article-extraction-benchmark.
scrapinghub/article-extraction-benchmark có tag gì không?
Bản đồng bộ chưa ghi nhận topic GitHub nào cho scrapinghub/article-extraction-benchmark. GitHub topics hiển thị ở thanh bên phải trang repo — đó là nơi đáng kiểm tra nhất.
scrapinghub/article-extraction-benchmark có website riêng không?
TopGit chưa ghi nhận URL trang chủ cho scrapinghub/article-extraction-benchmark. Phần README ở tab phía trên thường có link demo, hoặc xem mô tả GitHub của repo.
scrapinghub/article-extraction-benchmark còn đang phát triển không?
Commit gần nhất trên scrapinghub/article-extraction-benchmark là 5 tháng trước (theo timestamp GitHub). Repo có 37 fork — một chỉ báo về mức độ quan tâm của cộng đồng.
scrapinghub/article-extraction-benchmark dùng license gì?
scrapinghub/article-extraction-benchmark phát hành theo license MIT. Nên mở file LICENSE trên GitHub để xác nhận — license metadata đôi khi lệch với thực tế dự án.
Đọc đầy đủ README ở tab phía trên.
Muốn nghe thêm một ý kiến về article-extraction-benchmark?
Hỏi một AI đọc được trang này — một cú bấm là có ngay nhận định về article-extraction-benchmark.