IonicaBizau/scrape-it hiện có 4.1k sao trên GitHub, viết chủ yếu bằng JavaScript. 🔮 A Node.js scraper for humans.
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
Swiftproxy: Reliable rotating and static residential proxies for Node.js web scraping and data collection, with 90M+ IPs across 220+ locations worldwide.
Talordata SERP API: Real-time, accurate search data to power every decision
ProxySeller gives you reliable proxy infrastructure for any workflow: private residential, mobile (4G/5G/LTE), ISP, and datacenter IPv4/IPv6 proxies, all with HTTP(S) and SOCKS5 support. Tap into 47M+ residential IPs across 220+ locations, target exactly the country, city, or ISP you need, and count on 99.7% uptime backed by 24/7 support. Get 15% off with promo code BIZAU.
SerpApi.com
:cloud: Installation
# Using npm
npm install --save scrape-it
# Using yarn
yarn add scrape-it
:bulb: ProTip: You can install the cli version of this module by running npm install --global scrape-it-cli (or yarn global add scrape-it-cli).
FAQ
Here are some frequent questions and their answers.
1. How to parse scrape pages?
scrape-it has only a simple request module for making requests. That means you cannot directly parse ajax pages with it, but in general you will have those scenarios:
The ajax response is in JSON format. In this case, you can make the request directly, without needing a scraping library.
The ajax response gives you HTML back. Instead of calling the main website (e.g. example.com), pass to scrape-it the ajax url (e.g. example.com/api/that-endpoint) and you will you will be able to parse the response
The ajax request is so complicated that you don't want to reverse-engineer it. In this case, use a headless browser (e.g. Google Chrome, Electron, PhantomJS) to load the content and then use the .scrapeHTML method from scrape it once you get the HTML loaded on the page.
2. Crawling
There is no fancy way to crawl pages with scrape-it. For simple scenarios, you can parse the list of urls from the initial page and then, using Promises, parse each page. Also, you can use a different crawler to download the website and then use the .scrapeHTML method to scrape the local files.
3. Local files
Use the .scrapeHTML to parse the HTML read from the local files using fs.readFile.
:clipboard: Example
const scrapeIt = require("scrape-it")
// Promise interface
scrapeIt("https://ionicabizau.net", {
title: ".header h1"
, desc: ".header h2"
, avatar: {
selector: ".header img"
, attr: "src"
}
}).then(({ data, status }) => {
console.log(`Status Code: ${status}`)
console.log(data)
});
// Async-Await
(async () => {
const { data } = await scrapeIt("https://ionicabizau.net", {
// Fetch the articles
articles: {
listItem: ".article"
, data: {
// Get the article date and convert it into a Date object
createdAt: {
selector: ".date"
, convert: x => new Date(x)
}
// Get the title
, title: "a.article-title"
// Nested list
, tags: {
listItem: ".tags > span"
}
// Get the content
, content: {
selector: ".article-content"
, how: "html"
}
// Get attribute value of root listItem by omitting the selector
, classes: {
attr: "class"
}
}
}
// Fetch the blog pages
, pages: {
listItem: "li.page"
, name: "pages"
, data: {
title: "a"
, url: {
selector: "a"
, attr: "href"
}
}
}
// Fetch some other data from the page
, title: ".header h1"
, desc: ".header h2"
, avatar: {
selector: ".header img"
, attr: "src"
}
})
console.log(data)
// { articles:
// [ { createdAt: Mon Mar 14 2016 00:00:00 GMT+0200 (EET),
// title: 'Pi Day, Raspberry Pi and Command Line',
// tags: [Object],
// content: '<p>Everyone knows (or should know)...a" alt=""></p>\n',
// classes: [Object] },
// { createdAt: Thu Feb 18 2016 00:00:00 GMT+0200 (EET),
// title: 'How I ported Memory Blocks to modern web',
// tags: [Object],
// content: '<p>Playing computer games is a lot of fun. ...',
// classes: [Object] },
// { createdAt: Mon Nov 02 2015 00:00:00 GMT+0200 (EET),
// title: 'How to convert JSON to Markdown using json2md',
// tags: [Object],
// content: '<p>I love and ...',
// classes: [Object] } ],
// pages:
// [ { title: 'Blog', url: '/' },
// { title: 'About', url: '/about' },
// { title: 'FAQ', url: '/faq' },
// { title: 'Training', url: '/training' },
// { title: 'Contact', url: '/contact' } ],
// title: 'Ionică Bizău',
// desc: 'Web Developer, Linux geek and Musician',
// avatar: '/images/logo.png' }
})()
:memo: Documentation
scrapeIt(url, opts, cb)
A scraping module for humans.
Params
String|Objecturl: The page url or request options.
Objectopts: The options passed to scrapeHTML method.
Functioncb: The callback function.
Return
Promise A promise object resolving with:
data (Object): The scraped data.
$ (Function): The Cheeerio function. This may be handy to do some other manipulation on the DOM, if needed.
response (Object): The response object.
body (String): The raw body as a string.
scrapeIt.scrapeHTML($, opts)
Scrapes the data in the provided element.
For the format of the selector, please refer to the Selectors section of the Cheerio library
Params
Cheerio$: The input element.
Objectopts: An object containing the scraping information.
If you want to scrape a list, you have to use the listItem selector:
listItem (String): The list item selector.
data (Object): The fields to include in the list objects:
<fieldName> (Object|String): The selector or an object containing:
selector (String): The selector.
convert (Function): An optional function to change the value.
how (Function|String): A function or function name to access the
value.
attr (String): If provided, the value will be taken based on
the attribute name.
trimValue (Boolean): If false, the value will not be trimmed
(default: true).
closest (String): If provided, returns the first ancestor of
the given element.
eq (Number): If provided, it will select the nth element.
texteq (Number): If provided, it will select the nth direct text child.
Deep text child selection is not possible yet.
Overwrites the how key.
listItem (Object): An object, keeping the recursive schema of
the listItem object. This can be used to create nested lists.
Please post questions on Stack Overflow. You can open issues with questions, as long you add a link to your Stack Overflow question.
For bug reports and feature requests, open issues. :bug:
For direct and quick help, you can use Codementor. :rocket:
:yum: How to contribute
Have an idea? Found a bug? See how to contribute.
:sparkling_heart: Support my projects
I open-source almost everything I can, and I try to reply to everyone needing help using these projects. Obviously,
this takes time. You can integrate and use these projects in your applications for free! You can even change the source code and redistribute (even resell it).
However, if you get some profit from this or just want to encourage me to continue creating stuff, there are few ways you can do it:
Starring and sharing the projects you like :rocket:
—I love books! I will remember you after years if you buy me one. :grin: :book:
—You can make one-time donations via PayPal. I'll probably buy a coffee tea. :tea:
—Set up a recurring monthly donation and you will get interesting news about what I'm doing (things that I don't share with everyone).
Bitcoin—You can send me bitcoins at this address (or scanning the code below): 1P9BRsmazNQcuyTxEqveUsnf5CERdq35V6
IonicaBizau/scrape-it thuộc nhóm Automation trên TopGit, cùng 3 topic GitHub. Trang Trending và Topics liệt kê các repo cùng số sao và cùng ngôn ngữ để so sánh.
Đọc thêm về IonicaBizau/scrape-it ở đâu?
Trang TopGit này là một snapshot — tab "Readme" hiển thị nguyên văn README của repo (đã bỏ link, giữ ảnh). Repo GitHub ở github.com/IonicaBizau/scrape-it là nguồn chính thức.
IonicaBizau/scrape-it có bao nhiêu sao?
IonicaBizau/scrape-it có 4.1k sao GitHub — tải lại trang để xem số mới nhất, hoặc xem trực tiếp github.com/IonicaBizau/scrape-it. TopGit phản chiếu số sao của GitHub nhưng không cam kết đến từng phút.
IonicaBizau/scrape-it có những chủ đề gì?
GitHub topics của IonicaBizau/scrape-it: "node-scraper", "scraper", "web-scraping". TopGit xếp repo vào nhóm Automation.
IonicaBizau/scrape-it còn đang phát triển không?
Commit gần nhất trên IonicaBizau/scrape-it là 1 tháng trước (theo timestamp GitHub). Repo có 219 fork — một chỉ báo về mức độ quan tâm của cộng đồng.
IonicaBizau/scrape-it viết bằng ngôn ngữ gì?
IonicaBizau/scrape-it chủ yếu viết bằng JavaScript. Trường "language" của GitHub dựa trên phần lớn byte ở nhánh mặc định.
Vì sao IonicaBizau/scrape-it được xếp vào nhóm Automation?
TopGit xếp IonicaBizau/scrape-it vào nhóm Automation dựa trên GitHub topics và mô tả của repo (gắn thẻ: "node-scraper", "scraper", "web-scraping"). Việc phân loại dựa trên metadata thật của repo, không phải đoán theo cảm tính biên tập.
Đọc đầy đủ README ở tab phía trên.
scrape-it có đáng để bạn bỏ thời gian?
ChatGPT, Claude và Perplexity đều đọc được trang này. Hỏi thử xem họ nghĩ gì về scrape-it.