Media Cloud news article metadata extraction
Project description
Media Cloud Metadata Extractor
🚧 under construction 🚧
This is a package to extract a domain, title, publication date, text, and language content from the URL or text of an online news story. The methods for each are extracted from the larger Media Cloud project, but also build on numerous 3rd party libraries. The metadata extracted includes:
- the original URL of publication
- a normalized URL useful for de-duplication
- the canonical domain published on
- the date of publication
- the primary language used in the article text
- the title of the article
- a normalized title useful for de-duplication
- the text content of the news article
- the name of the library used to extract the article content
Installation
pip install mediacloud-metadata
Usage
If you pass in a URL, it will follow redirects and fetch the HTML for you.
from mcmetadata import extract
metadata = extract(url="https://my.awesome.news/story-path")
You can also pass in HTML you already have on hand. Note that in this case it is also useful to pass in the URL because that is used for some for some of the metadata extraction.
from mcmetadata import extract
metadata = extract(url="https://my.awesome.news/story-path",
html_text="<html><head><title>my webpage ... </html>")
Distribution
This is setup to automatically publish new releases to PyPi via GitHub Actions:
- Run
pytestto make sure all the test pass - Update the version number in
mcextractor/__init__.py - Make a brief note in the version history section below about the changes
- Commit the changes
- Tag the commit with a semantic version number - 'v*..'
- Push to repo to GitHub
Manual Release
If you need to push a release manually then you can do this:
- Run
python setup.py sdistto create an installation package - Run
twine upload --repository-url https://test.pypi.org/legacy/ dist/*to upload it to PyPI's test platform - Run
twine upload dist/*to upload it to PyPI
Version History
- v0.7.2: fix extraction argument bug introduced in last release, fix some more test cases
- v0.7.1: fix bug in url normalization, increase robustness in extractor chain
- v0.7.0: fix YouTube url normalization, better Trafilatura defaults, limit to pub dates within 90 days of today, ensure language is 2 letters, content extraction performance improvements, fix some title parsing bugs, add more test cases, add script to compare results to older Media Cloud code (which this stuff is extracted from), resolve language guessing conflicts better, handle text encoding errors
- v0.6.0: prefer language from metadata over guessing, try Trafilatura as first parser, encoding fixes
- v0.5.5: turn off aggressive date finding mode, which was making lots of 1/1 date guesses
- v0.5.4: bug in regex that parses og:title properties into titles
- v0.5.3: bug fixes in title normalization
- v0.5.2: more efficient parsing of dates from HTML, remove failing over-specified canonical domain case
- v0.5.1: fix small bug related to use of BeautifulSoup
- v0.5.0: add normalized URL and normalized title
- v0.4.3: more work on title regex bug
- v0.4.2: work on title regex bug
- v0.4.1: work on deployment
- v0.4.0: performance improvements, dependency updates
- v0.3.1: update dependencies
- v0.3.0: more fault tolerant, faster regex's, track extraction rates, update requirements
- v0.2.0: first packaging release for use in other places
- v0.1.1: first version for testing with collaborators
Contributors
Created as part of the Media Cloud Project:
- Rahul Bhargava (Media Cloud, Northeastern University)
- Vangelis Banos (Internet Archive)
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for mediacloud-metadata-0.7.2.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | e7584a49cd84037e6e5b249ecb3ab785f44826c5f5091fd45bb7c75f3b5a359b |
|
| MD5 | 075745c5f621f1fd7f8170fc1f2d30c2 |
|
| BLAKE2b-256 | a78b654f486ccaba7a59b21d01f84ed4dadc50e208a1e3624b20634e01caaa0e |
