scrapinghub/extruct
971 GitHub stars and counting — scrapinghub/extruct is a Python project TopGit is tracking across repositories on the platform. Extract embedded metadata from HTML markup
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
Snapshot
Top contributors
Show top contributors
======= extruct
.. image:: https://github.com/scrapinghub/extruct/workflows/build/badge.svg?branch=master :target: https://github.com/scrapinghub/extruct/actions :alt: Build Status
.. image:: https://img.shields.io/codecov/c/github/scrapinghub/extruct/master.svg?maxAge=2592000 :target: https://codecov.io/gh/scrapinghub/extruct :alt: Coverage report
.. image:: https://img.shields.io/pypi/v/extruct.svg :target: https://pypi.python.org/pypi/extruct :alt: PyPI Version
extruct is a library for extracting embedded metadata from HTML markup.
Currently, extruct supports:
W3C's HTML Microdata_embedded JSON-LD_Microformat_ viamf2py_Facebook's Open Graph_- (experimental)
RDFa_ viardflib_ Dublin Core Metadata (DC-HTML-2003)_
.. _W3C's HTML Microdata: http://www.w3.org/TR/microdata/ .. _embedded JSON-LD: http://www.w3.org/TR/json-ld/#embedding-json-ld-in-html-documents .. _RDFa: https://www.w3.org/TR/html-rdfa/ .. _rdflib: https://pypi.python.org/pypi/rdflib/ .. _Microformat: http://microformats.org/wiki/Main_Page .. _mf2py: https://github.com/microformats/mf2py .. _Facebook's Open Graph: http://ogp.me/ .. _Dublin Core Metadata (DC-HTML-2003): https://www.dublincore.org/specifications/dublin-core/dcq-html/2003-11-30/
The microdata algorithm is a revisit of this Scrapinghub blog post_ showing how to use EXSLT extensions.
.. _this Scrapinghub blog post: http://blog.scrapinghub.com/2014/06/18/extracting-schema-org-microdata-using-scrapy-selectors-and-xpath/
Installation
::
pip install extruct
Usage
All-in-one extraction +++++++++++++++++++++
The simplest example how to use extruct is to call
extruct.extract(htmlstring, base_url=base_url)
with some HTML string and an optional base URL.
Let's try this on a webpage that uses all the syntaxes supported (RDFa with ogp_).
First fetch the HTML using python-requests and then feed the response body to extruct::
import extruct import requests import pprint from w3lib.html import get_base_url
pp = pprint.PrettyPrinter(indent=2) r = requests.get('https://www.optimizesmart.com/how-to-use-open-graph-protocol/') base_url = get_base_url(r.text, r.url) data = extruct.extract(r.text, base_url=base_url)
pp.pprint(data) { 'dublincore': [ { 'elements': [ { 'URI': 'http://purl.org/dc/elements/1.1/description', 'content': 'What is Open Graph Protocol ' 'and why you need it? Learn to ' 'implement Open Graph Protocol ' 'for Facebook on your website. ' 'Open Graph Protocol Meta Tags.', 'name': 'description'}], 'namespaces': {}, 'terms': []}],
'json-ld': [ { '@context': 'https://schema.org', '@id': '#organization', '@type': 'Organization', 'logo': 'https://www.optimizesmart.com/wp-content/uploads/2016/03/optimize-smart-Twitter-logo.jpg', 'name': 'Optimize Smart', 'sameAs': [ 'https://www.facebook.com/optimizesmart/', 'https://uk.linkedin.com/in/analyticsnerd', 'https://www.youtube.com/user/optimizesmart', 'https://twitter.com/analyticsnerd'], 'url': 'https://www.optimizesmart.com/'}], 'microdata': [ { 'properties': {'headline': ''}, 'type': 'http://schema.org/WPHeader'}], 'microformat': [ { 'children': [ { 'properties': { 'category': [ 'specialized-tracking'], 'name': [ 'Open Graph ' 'Protocol for ' 'Facebook ' 'explained with ' 'examples\n' '\n' 'Specialized ' 'Tracking\n' '\n' '\n' (...) 'Follow ' '@analyticsnerd\n' '!function(d,s,id){var ' "js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, " "'script', " "'twitter-wjs');"]}, 'type': ['h-entry']}], 'properties': { 'name': [ 'Open Graph Protocol for ' 'Facebook explained with ' 'examples\n' (...) 'Follow @analyticsnerd\n' '!function(d,s,id){var ' "js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, " "'script', 'twitter-wjs');"]}, 'type': ['h-feed']}], 'opengraph': [ { 'namespace': {'og': 'http://ogp.me/ns#'}, 'properties': [ ('og:locale', 'en_US'), ('og:type', 'article'), ( 'og:title', 'Open Graph Protocol for Facebook ' 'explained with examples'), ( 'og:description', 'What is Open Graph Protocol and why you ' 'need it? Learn to implement Open Graph ' 'Protocol for Facebook on your website. ' 'Open Graph Protocol Meta Tags.'), ( 'og:url', 'https://www.optimizesmart.com/how-to-use-open-graph-protocol/'), ('og:site_name', 'Optimize Smart'), ( 'og:updated_time', '2018-03-09T16:26:35+00:00'), ( 'og:image', 'https://www.optimizesmart.com/wp-content/uploads/2010/07/open-graph-protocol.jpg'), ( 'og:image:secure_url', 'https://www.optimizesmart.com/wp-content/uploads/2010/07/open-graph-protocol.jpg')]}], 'rdfa': [ { '@id': 'https://www.optimizesmart.com/how-to-use-open-graph-protocol/#header', 'http://www.w3.org/1999/xhtml/vocab#role': [ { '@id': 'http://www.w3.org/1999/xhtml/vocab#banner'}]}, { '@id': 'https://www.optimizesmart.com/how-to-use-open-graph-protocol/', 'article:modified_time': [ { '@value': '2018-03-09T16:26:35+00:00'}], 'article:published_time': [ { '@value': '2010-07-02T18:57:23+00:00'}], 'article:publisher': [ { '@value': 'https://www.facebook.com/optimizesmart/'}], 'article:section': [{'@value': 'Specialized Tracking'}], 'http://ogp.me/ns#description': [ { '@value': 'What is Open ' 'Graph Protocol ' 'and why you need ' 'it? Learn to ' 'implement Open ' 'Graph Protocol ' 'for Facebook on ' 'your website. ' 'Open Graph ' 'Protocol Meta ' 'Tags.'}], 'http://ogp.me/ns#image': [ { '@value': 'https://www.optimizesmart.com/wp-content/uploads/2010/07/open-graph-protocol.jpg'}], 'http://ogp.me/ns#image:secure_url': [ { '@value': 'https://www.optimizesmart.com/wp-content/uploads/2010/07/open-graph-protocol.jpg'}], 'http://ogp.me/ns#locale': [{'@value': 'en_US'}], 'http://ogp.me/ns#site_name': [{'@value': 'Optimize Smart'}], 'http://ogp.me/ns#title': [ { '@value': 'Open Graph Protocol for ' 'Facebook explained with ' 'examples'}], 'http://ogp.me/ns#type': [{'@value': 'article'}], 'http://ogp.me/ns#updated_time': [ { '@value': '2018-03-09T16:26:35+00:00'}], 'http://ogp.me/ns#url': [ { '@value': 'https://www.optimizesmart.com/how-to-use-open-graph-protocol/'}], 'https://api.w.org/': [ { '@id': 'https://www.optimizesmart.com/wp-json/'}]}]}
Select syntaxes +++++++++++++++ It is possible to select which syntaxes to extract by passing a list with the desired ones to extract. Valid values: 'microdata', 'json-ld', 'opengraph', 'microformat', 'rdfa' and 'dublincore'. If no list is passed all syntaxes will be extracted and returned::
r = requests.get('http://www.songkick.com/artists/236156-elysian-fields') base_url = get_base_url(r.text, r.url) data = extruct.extract(r.text, base_url, syntaxes=['microdata', 'opengraph', 'rdfa'])
pp.pprint(data) { 'microdata': [], 'opengraph': [ { 'namespace': { 'concerts': 'http://ogp.me/ns/fb/songkick-concerts#', 'fb': 'http://www.facebook.com/2008/fbml', 'og': 'http://ogp.me/ns#'}, 'properties': [ ('fb:app_id', '308540029359'), ('og:site_name', 'Songkick'), ('og:type', 'songkick-concerts:artist'), ('og:title', 'Elysian Fields'), ( 'og:description', 'Find out when Elysian Fields is next ' 'playing live near you. List of all ' 'Elysian Fields tour dates and concerts.'), ( 'og:url', 'https://www.songkick.com/artists/236156-elysian-fields'), ( 'og:image', 'http://images.sk-static.com/images/media/img/col4/20100330-103600-169450.jpg')]}], 'rdfa': [ { '@id': 'https://www.songkick.com/artists/236156-elysian-fields', 'al:ios:app_name': [{'@value': 'Songkick Concerts'}], 'al:ios:app_store_id': [{'@value': '438690886'}], 'al:ios:url': [ { '@value': 'songkick://artists/236156-elysian-fields'}], 'http://ogp.me/ns#description': [ { '@value': 'Find out when ' 'Elysian Fields is ' 'next playing live ' 'near you. List of ' 'all Elysian ' 'Fields tour dates ' 'and concerts.'}], 'http://ogp.me/ns#image': [ { '@value': 'http://images.sk-static.com/images/media/img/col4/20100330-103600-169450.jpg'}], 'http://ogp.me/ns#site_name': [{'@value': 'Songkick'}], 'http://ogp.me/ns#title': [{'@value': 'Elysian Fields'}], 'http://ogp.me/ns#type': [{'@value': 'songkick-concerts:artist'}], 'http://ogp.me/ns#url': [ { '@value': 'https://www.songkick.com/artists/236156-elysian-fields'}], 'http://www.facebook.com/2008/fbmlapp_id': [ { '@value': '308540029359'}]}]}
Alternatively, if you already parsed the HTML before calling extruct, you can use the tree instead of the HTML string: ::
using the request from the previous example
base_url = get_base_url(r.text, r.url) from extruct.utils import parse_html tree = parse_html(r.text) data = extruct.extract(tree, base_url, syntaxes=['microdata', 'opengraph', 'rdfa'])
Microformat format doesn't support the HTML tree, so you need to use a HTML string.
Uniform +++++++ Another option is to uniform the output of microformat, opengraph, microdata, dublincore and json-ld syntaxes to the following structure: ::
{'@context': 'http://example.com',
'@type': 'example_type',
/* All other the properties in keys here */
}
To do so set uniform=True when calling extract, it's false by default for backward compatibility. Here the same example as before but with uniform set to True: ::
r = requests.get('http://www.songkick.com/artists/236156-elysian-fields') base_url = get_base_url(r.text, r.url) data = extruct.extract(r.text, base_url, syntaxes=['microdata', 'opengraph', 'rdfa'], uniform=True)
pp.pprint(data) { 'microdata': [], 'opengraph': [ { '@context': { 'concerts': 'http://ogp.me/ns/fb/songkick-concerts#', 'fb': 'http://www.facebook.com/2008/fbml', 'og': 'http://ogp.me/ns#'}, '@type': 'songkick-concerts:artist', 'fb:app_id': '308540029359', 'og:description': 'Find out when Elysian Fields is next ' 'playing live near you. List of all ' 'Elysian Fields tour dates and concerts.', 'og:image': 'http://images.sk-static.com/images/media/img/col4/20100330-103600-169450.jpg', 'og:site_name': 'Songkick', 'og:title': 'Elysian Fields', 'og:url': 'https://www.songkick.com/artists/236156-elysian-fields'}], 'rdfa': [ { '@id': 'https://www.songkick.com/artists/236156-elysian-fields', 'al:ios:app_name': [{'@value': 'Songkick Concerts'}], 'al:ios:app_store_id': [{'@value': '438690886'}], 'al:ios:url': [ { '@value': 'songkick://artists/236156-elysian-fields'}], 'http://ogp.me/ns#description': [ { '@value': 'Find out when ' 'Elysian Fields is ' 'next playing live ' 'near you. List of ' 'all Elysian ' 'Fields tour dates ' 'and concerts.'}], 'http://ogp.me/ns#image': [ { '@value': 'http://images.sk-static.com/images/media/img/col4/20100330-103600-169450.jpg'}], 'http://ogp.me/ns#site_name': [{'@value': 'Songkick'}], 'http://ogp.me/ns#title': [{'@value': 'Elysian Fields'}], 'http://ogp.me/ns#type': [{'@value': 'songkick-concerts:artist'}], 'http://ogp.me/ns#url': [ { '@value': 'https://www.songkick.com/artists/236156-elysian-fields'}], 'http://www.facebook.com/2008/fbmlapp_id': [ { '@value': '308540029359'}]}]}
NB rdfa structure is not uniformed yet.
Returning HTML node +++++++++++++++++++
It is also possible to get references to HTML node for every extracted metadata item. The feature is supported only by microdata syntax.
To use that, just set the return_html_node option of extract method to True.
As the result, an additional key "nodeHtml" will be included in the result for every
item. Each node is of lxml.etree.Element type: ::
r = requests.get('http://www.rugpadcorner.com/shop/no-muv/') base_url = get_base_url(r.text, r.url) data = extruct.extract(r.text, base_url, syntaxes=['microdata'], return_html_node=True)
pp.pprint(data) { 'microdata': [ { 'htmlNode': <Element div at 0x7f10f8e6d3b8>, 'properties': { 'description': 'KEEP RUGS FLAT ON CARPET!\n' 'Not your thin sticky pad, ' 'No-Muv is truly the best!', 'image': ['', ''], 'name': ['No-Muv', 'No-Muv'], 'offers': [ { 'htmlNode': <Element div at 0x7f10f8e6d138>, 'properties': { 'availability': 'http://schema.org/InStock', 'price': 'Price: ' '$45'}, 'type': 'http://schema.org/Offer'}, { 'htmlNode': <Element div at 0x7f10f8e60f48>, 'properties': { 'availability': 'http://schema.org/InStock', 'price': '(Select ' 'Size/Shape ' 'for ' 'Pricing)'}, 'type': 'http://schema.org/Offer'}], 'ratingValue': ['5.00', '5.00']}, 'type': 'http://schema.org/Product'}]}
Single extractors
You can also use each extractor individually. See below.
Microdata extraction ++++++++++++++++++++ ::
import pprint pp = pprint.PrettyPrinter(indent=2)
from extruct.w3cmicrodata import MicrodataExtractor
example from http://www.w3.org/TR/microdata/#associating-names-with-items
html = """ ... ... ... Photo gallery ... ... ...
My photos
... ...... The house I found. ... ... ...
... The mailbox. ... ... ...
All images licensed under the <a itemprop="license" ... href="http://www.opensource.org/licenses/mit-license.php">MIT ... license.
... ... ... """mde = MicrodataExtractor() data = mde.extract(html) pp.pprint(data) [{'properties': {'license': 'http://www.opensource.org/licenses/mit-license.php', 'title': 'The house I found.', 'work': 'http://www.example.com/images/house.jpeg'}, 'type': 'http://n.whatwg.org/work'}, {'properties': {'license': 'http://www.opensource.org/licenses/mit-license.php', 'title': 'The mailbox.', 'work': 'http://www.example.com/images/mailbox.jpeg'}, 'type': 'http://n.whatwg.org/work'}]
JSON-LD extraction ++++++++++++++++++ ::
import pprint pp = pprint.PrettyPrinter(indent=2)
from extruct.jsonld import JsonLdExtractor
html = """ ... ... ... Some Person Page ... ... ...
This guys
... ", ... "@type": "Person", ... "name": "John Doe", ... "jobTitle": "Graduate research assistant", ... "affiliation": "University of Dreams", ... "additionalName": "Johnny", ... "url": "http://www.example.com", ... "address": { ... "@type": "PostalAddress", ... "streetAddress": "1234 Peach Drive", ... "addressLocality": "Wonderland", ... "addressRegion": "Georgia" ... } ... } ... ... ... """jslde = JsonLdExtractor()
data = jslde.extract(html) pp.pprint(data) [{'@context': 'http://schema.org', '@type': 'Person', 'additionalName': 'Johnny', 'address': {'@type': 'PostalAddress', 'addressLocality': 'Wonderland', 'addressRegion': 'Georgia', 'streetAddress': '1234 Peach Drive'}, 'affiliation': 'University of Dreams', 'jobTitle': 'Graduate research assistant', 'name': 'John Doe', 'url': 'http://www.example.com'}]
RDFa extraction (experimental) ++++++++++++++++++++++++++++++
::
import pprint pp = pprint.PrettyPrinter(indent=2) from extruct.rdfa import RDFaExtractor # you can ignore the warning about html5lib not being available INFO:rdflib:RDFLib Version: 4.2.1 /home/paul/.virtualenvs/extruct.wheel.test/lib/python3.5/site-packages/rdflib/plugins/parsers/structureddata.py:30: UserWarning: html5lib not found! RDFa and Microdata parsers will not be available. 'parsers will not be available.')
html = """ ... ... ... ... ... ...
...... ... ... """The trouble with Bob
... ... ...Alice
......... ... ...The trouble with Bob is that he takes much better photos than I do:
...rdfae = RDFaExtractor() pp.pprint(rdfae.extract(html, base_url='http://www.example.com/index.html')) [{'@id': 'http://www.example.com/alice/posts/trouble_with_bob', '@type': ['http://schema.org/BlogPosting'], 'http://purl.org/dc/terms/creator': [{'@id': 'http://www.example.com/index.html#me'}], 'http://purl.org/dc/terms/title': [{'@value': 'The trouble with Bob'}], 'http://schema.org/articleBody': [{'@value': '\n' ' The trouble with Bob ' 'is that he takes much better ' 'photos than I do:\n' ' '}], 'http://schema.org/creator': [{'@id': 'http://www.example.com/index.html#me'}]}]
You'll get a list of expanded JSON-LD nodes.
Open Graph extraction ++++++++++++++++++++++++++++++
::
import pprint pp = pprint.PrettyPrinter(indent=2)
from extruct.opengraph import OpenGraphExtractor
html = """ ... ... ... Himanshu's Open Graph Protocol ... ... ... ... ... ... ... ... ... ... ... ... ... ...
... ... ... """opengraphe = OpenGraphExtractor() pp.pprint(opengraphe.extract(html)) [{"namespace": { "og": "http://ogp.me/ns#" }, "properties": [ [ "og:title", "Himanshu's Open Graph Protocol" ], [ "og:type", "article" ], [ "og:url", "https://www.eventeducation.com/test.php" ], [ "og:image", "https://www.eventeducation.com/images/982336_wedding_dayandouan_th.jpg" ], [ "og:site_name", "Event Education" ], [ "og:description", "Event Education provides free courses on event planning and management to event professionals worldwide." ] ] }]
Microformat extraction ++++++++++++++++++++++++++++++
::
import pprint pp = pprint.PrettyPrinter(indent=2)
from extruct.microformat import MicroformatExtractor
html = """ ... ... ... Himanshu's Open Graph Protocol ... ... ... ... ... ...
...Microformats are amazing
...Published by W. Developer ... on 13th June 2013
...In which I extoll the virtues of using microformats.
......... ... ... ... """Blah blah blah
...microformate = MicroformatExtractor() data = microformate.extract(html) pp.pprint(data) [{"type": [ "h-entry" ], "properties": { "name": [ "Microformats are amazing" ], "author": [ { "type": [ "h-card" ], "properties": { "name": [ "W. Developer" ], "url": [ "http://example.com" ] }, "value": "W. Developer" } ], "published": [ "2013-06-13 12:00:00" ], "summary": [ "In which I extoll the virtues of using microformats." ], "content": [ { "html": "\n
Blah blah blah
\n", "value": "\nBlah blah blah\n" } ] } }]
DublinCore extraction ++++++++++++++++++++++++++++++ ::
>>> import pprint
>>> pp = pprint.PrettyPrinter(indent=2)
>>> from extruct.dublincore import DublinCoreExtractor
>>> html = '''<head profile="http://dublincore.org/documents/dcq-html/">
... <title>Expressing Dublin Core in HTML/XHTML meta and link elements</title>
... <link rel="schema.DC" href="http://purl.org/dc/elements/1.1/" />
... <link rel="schema.DCTERMS" href="http://purl.org/dc/terms/" />
...
...
... <meta name="DC.title" lang="en" content="Expressing Dublin Core
... in HTML/XHTML meta and link elements" />
... <meta name="DC.creator" content="Andy Powell, UKOLN, University of Bath" />
... <meta name="DCTERMS.issued" scheme="DCTERMS.W3CDTF" content="2003-11-01" />
... <meta name="DC.identifier" scheme="DCTERMS.URI"
... content="http://dublincore.org/documents/dcq-html/" />
... <link rel="DCTERMS.replaces" hreflang="en"
... href="http://dublincore.org/documents/2000/08/15/dcq-html/" />
... <meta name="DCTERMS.abstract" content="This document describes how
... qualified Dublin Core metadata can be encoded
... in HTML/XHTML <meta> elements" />
... <meta name="DC.format" scheme="DCTERMS.IMT" content="text/html" />
... <meta name="DC.type" scheme="DCTERMS.DCMIType" content="Text" />
... <meta name="DC.Date.modified" content="2001-07-18" />
... <meta name="DCTERMS.modified" content="2001-07-18" />'''
>>> dublinlde = DublinCoreExtractor()
>>> data = dublinlde.extract(html)
>>> pp.pprint(data)
[ { 'elements': [ { 'URI': 'http://purl.org/dc/elements/1.1/title',
'content': 'Expressing Dublin Core\n'
'in HTML/XHTML meta and link elements',
'lang': 'en',
'name': 'DC.title'},
{ 'URI': 'http://purl.org/dc/elements/1.1/creator',
'content': 'Andy Powell, UKOLN, University of Bath',
'name': 'DC.creator'},
{ 'URI': 'http://purl.org/dc/elements/1.1/identifier',
'content': 'http://dublincore.org/documents/dcq-html/',
'name': 'DC.identifier',
'scheme': 'DCTERMS.URI'},
{ 'URI': 'http://purl.org/dc/elements/1.1/format',
'content': 'text/html',
'name': 'DC.format',
'scheme': 'DCTERMS.IMT'},
{ 'URI': 'http://purl.org/dc/elements/1.1/type',
'content': 'Text',
'name': 'DC.type',
'scheme': 'DCTERMS.DCMIType'}],
'namespaces': { 'DC': 'http://purl.org/dc/elements/1.1/',
'DCTERMS': 'http://purl.org/dc/terms/'},
'terms': [ { 'URI': 'http://purl.org/dc/terms/issued',
'content': '2003-11-01',
'name': 'DCTERMS.issued',
'scheme': 'DCTERMS.W3CDTF'},
{ 'URI': 'http://purl.org/dc/terms/abstract',
'content': 'This document describes how\n'
'qualified Dublin Core metadata can be encoded\n'
'in HTML/XHTML <meta> elements',
'name': 'DCTERMS.abstract'},
{ 'URI': 'http://purl.org/dc/terms/modified',
'content': '2001-07-18',
'name': 'DC.Date.modified'},
{ 'URI': 'http://purl.org/dc/terms/modified',
'content': '2001-07-18',
'name': 'DCTERMS.modified'},
{ 'URI': 'http://purl.org/dc/terms/replaces',
'href': 'http://dublincore.org/documents/2000/08/15/dcq-html/',
'hreflang': 'en',
'rel': 'DCTERMS.replaces'}]}]
Command Line Tool
extruct provides a command line tool that allows you to fetch a page and extract the metadata from it directly from the command line.
Dependencies ++++++++++++
The command line tool depends on requests, which is not installed by default
when you install extruct. In order to use the command line tool, you can
install extruct with the cli extra requirements::
pip install 'extruct[cli]'
Usage +++++
::
extruct "http://example.com"
Downloads "http://example.com" and outputs the Microdata, JSON-LD and RDFa, Open Graph
and Microformat metadata to stdout.
Supported Parameters ++++++++++++++++++++
By default, the command line tool will try to extract all the supported metadata formats from the page (currently Microdata, JSON-LD, RDFa, Open Graph and Microformat). If you want to restrict the output to just one or a subset of those, you can pass their individual names collected in a list through 'syntaxes' argument.
For example, this command extracts only Microdata and JSON-LD metadata from "http://example.com"::
extruct "http://example.com" --syntaxes microdata json-ld
NB syntaxes names passed must correspond to these: microdata, json-ld, rdfa, opengraph, microformat
Development version
::
mkvirtualenv extruct
pip install -r requirements-dev.txt
Tests
Run tests in current environment::
py.test tests
Use tox_ to run tests with different Python versions::
tox
.. _tox: https://testrun.org/tox/latest/ .. _ogp: https://ogp.me/
Related repositories
EbookFoundation/free-programming-books is a community-curated index of free learning materials for developers. What began as a Stack Overflow list has grown into a comprehensive resource, now overseen by the Free Ebook Foundation. It categorizes a wide range of content, including books by language and subject, free online courses, cheat sheets, interactive programming tutorials, problem sets, podcasts, screencasts, and browser-based programming playgrounds. Many of these resources are available in dozens of languages beyond English, and you can navigate the collection through its dedicated search website or by browsing the repository's markdown files.
The Algorithms - Python is an open-source collection of algorithm implementations written in Python, maintained by the TheAlgorithms organization on GitHub. The README frames the whole project as a teaching resource: the code exists to illustrate concepts, can run slower than Python's built-in equivalents, and is indexed in DIRECTORY.md for browsing. It's MIT licensed and available on GitHub.
Awesome Go, maintained at avelino/awesome-go, groups Go frameworks, libraries, and software into topic categories including Actor Model, Artificial Intelligence, Audio and Music, Authentication and Authorization, Blockchain, Bot Building, and Build Automation. Every linked entry gets a short description instead of a bare URL. The repository carries an MIT license and takes new entries through pull requests reviewed against its own contribution guidelines, which makes it a common bookmark for day-to-day Go development.
Master programming by recreating your favorite technologies from scratch.
Quick answers
Does scrapinghub/extruct have a project website?
No homepage URL was recorded for scrapinghub/extruct in TopGit's last sync. The README tab above frequently contains screenshots and demo links, or check the repository description on GitHub.
How many stars does scrapinghub/extruct have?
scrapinghub/extruct has 971 GitHub stars — refresh the page for the live number, or check github.com/scrapinghub/extruct. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
Is scrapinghub/extruct open source?
Yes — scrapinghub/extruct ships under the BSD-3-Clause license, which makes its source code freely readable (and, depending on license terms, forkable and reusable). Source: github.com/scrapinghub/extruct.
What is scrapinghub/extruct?
scrapinghub/extruct (scrapinghub/extruct) is a Python project on GitHub. From the project's own README: Extract embedded metadata from HTML markup
Where do I read more about scrapinghub/extruct?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/scrapinghub/extruct is the definitive source.
Read full README in the tab above.
Curious whether extruct is right for you?
Let ChatGPT, Claude, or Perplexity look into it — click below and see what AI actually says about extruct.
... The house I found.
...
...
...
... The mailbox.
...
...
...