ageitgey/node-unfluff là dự án mã nguồn mở trên GitHub, viết chủ yếu bằng HTML, với 2.2k sao. Automatically extract body content (and other cool stuff) from an html document
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
An automatic web page content extractor for Node.js!
Automatically grab the main
text out of a webpage like this:
extractor = require('unfluff');
data = extractor(my_html_data);
console.log(data.text);
In other words, it turns pretty webpages into boring plain text/json data:
This might be useful for:
Writing your own Instapaper clone
Easily building ML data sets from web pages
Reading your favorite articles from the console?
Please don't use this for:
Stealing other peoples' web pages
Making crappy spam sites with stolen content from other sites
Being a jerk
Credits / Thanks
This library is largely based on python-goose
by Xavier Grangier which is in turn based on goose
by Gravity Labs. However, it's not an exact
port so it may behave differently on some pages and the feature set is a little
bit different. If you are looking for a python or Scala/Java/JVM solution,
check out those libraries!
Install
To install the command-line unfluff utility:
npm install -g unfluff
To install the unfluff module for use in your Node.js project:
npm install --save unfluff
Usage
You can use unfluff from node or right on the command line!
Extracted data elements
This is what unfluff will try to grab from a web page:
title - The document's title (from the <title> tag)
softTitle - A version of title with less truncation
date - The document's publication date
copyright - The document's copyright line, if present
author - The document's author
publisher - The document's publisher (website name)
text - The main text of the document with all the junk thrown away
image - The main image for the document (what's used by facebook, etc.)
videos - An array of videos that were embedded in the article. Each video has src, width and height.
tags- Any tags or keywords that could be found by checking <rel> tags or by looking at href urls.
canonicalLink - The canonical url of the document, if given.
lang - The language of the document, either detected or supplied by you.
description - The description of the document, from <meta> tags
favicon - The url of the document's favicon.
links - An array of links embedded within the article text. (text and href for each)
This is returned as a simple json object.
Command line interface
You can pass a webpage to unfluff and it will try to parse out the interesting
bits.
You can either pass in a file name:
unfluff my_file.html
Or you can pipe it in:
curl -s "http://somesite.com/page" | unfluff
You can easily chain this together with other unix commands to do cool stuff.
For example, you can download a web page, parse it and then use
jq to print it just the body text.
language (optional): The document's two-letter language code. This will be
auto-detected as best as possible, but there might be cases where you want to
override it.
The extraction algorithm depends heavily on the language, so it probably won't work
if you have the language set incorrectly.
extractor = require('unfluff');
data = extractor(my_html_data);
Or supply the language code yourself:
extractor = require('unfluff');
data = extractor(my_html_data, 'en');
data will then be a json object that looks like this:
{
"title": "Shovel Knight review",
"softTitle": "Shovel Knight review: rewrite history",
"date": "2014-06-26T13:00:03Z",
"copyright": "2016 Vox Media Inc Designed in house",
"author": [
"Griffin McElroy"
],
"publisher": "Polygon",
"text": "Shovel Knight is inspired by the past in all the right ways — but it's far from stuck in it. [.. snip ..]",
"image": "http://cdn2.vox-cdn.com/uploads/chorus_image/image/34834129/jellyfish_hero.0_cinema_1280.0.png",
"tags": [],
"videos": [],
"canonicalLink": "http://www.polygon.com/2014/6/26/5842180/shovel-knight-review-pc-3ds-wii-u",
"lang": "en",
"description": "Shovel Knight is inspired by the past in all the right ways — but it's far from stuck in it.",
"favicon": "http://cdn1.vox-cdn.com/community_logos/42931/favicon.ico",
"links": [
{ "text": "Six Thirty", "href": "http://www.sixthirty.co/" }
]
}
extractor.lazy(html, language)
Lazy version of extractor(html, language).
The text extraction algorithm can be somewhat slow on large documents. If you
only need access to elements like title or image, you can use the
lazy extractor to get them more quickly without running the full processing
pipeline.
This returns an object just like the regular extractor except all fields
are replaced by functions and evaluation is only done when you call those
functions.
extractor = require('unfluff');
data = extractor.lazy(my_html_data, 'en');
// Access whichever data elements you need directly.
console.log(data.title());
console.log(data.softTitle());
console.log(data.date());
console.log(data.copyright());
console.log(data.author());
console.log(data.publisher());
console.log(data.text());
console.log(data.image());
console.log(data.tags());
console.log(data.videos());
console.log(data.canonicalLink());
console.log(data.lang());
console.log(data.description());
console.log(data.favicon());
Some of these data elements require calculating intermediate representations
of the html document. Everything is cached so looking up multiple data elements
and looking them up multiple times should be as fast as possible.
Demo
The easiest way to try out unfluff is to just install it:
But if you can't be bothered, you can check out
fetch text. It's a site by
Andy Jiang that uses unfluff. You send an
email with a url and it emails back with the cleaned content of that url. It
should give you a good idea of how unfluff handles different urls.
What is broken
Parsing web pages in languages other than English is poorly tested and probably
is buggy right now.
This definitely won't work yet for languages like Chinese / Arabic / Korean /
etc that need smarter word tokenization.
This has only been tested on a limited set of web pages. There are probably lots
of lurking bugs with web pages that haven't been tested yet.
ageitgey/node-unfluff có 2.2k sao GitHub — tải lại trang để xem số mới nhất, hoặc xem trực tiếp github.com/ageitgey/node-unfluff. TopGit phản chiếu số sao của GitHub nhưng không cam kết đến từng phút.
ageitgey/node-unfluff có phải mã nguồn mở không?
Có — ageitgey/node-unfluff phát hành theo license Apache-2.0, nghĩa là mã nguồn mở để đọc, fork và (tùy license) tái sử dụng. Mã: github.com/ageitgey/node-unfluff.
ageitgey/node-unfluff còn đang phát triển không?
Commit gần nhất trên ageitgey/node-unfluff là 3.5 năm trước (theo timestamp GitHub). Repo có 210 fork — một chỉ báo về mức độ quan tâm của cộng đồng.
ageitgey/node-unfluff dùng license gì?
ageitgey/node-unfluff phát hành theo license Apache-2.0. Nên mở file LICENSE trên GitHub để xác nhận — license metadata đôi khi lệch với thực tế dự án.
ageitgey/node-unfluff là gì?
ageitgey/node-unfluff (ageitgey/node-unfluff) là dự án HTML trên GitHub. Theo mô tả gốc: Automatically extract body content (and other cool stuff) from an html document
ageitgey/node-unfluff viết bằng ngôn ngữ gì?
ageitgey/node-unfluff chủ yếu viết bằng HTML. Trường "language" của GitHub dựa trên phần lớn byte ở nhánh mặc định.
Đọc thêm về ageitgey/node-unfluff ở đâu?
Trang TopGit này là một snapshot — tab "Readme" hiển thị nguyên văn README của repo (đã bỏ link, giữ ảnh). Repo GitHub ở github.com/ageitgey/node-unfluff là nguồn chính thức.
Đọc đầy đủ README ở tab phía trên.
node-unfluff có đáng để bạn bỏ thời gian?
ChatGPT, Claude và Perplexity đều đọc được trang này. Hỏi thử xem họ nghĩ gì về node-unfluff.