Media Cloud Metadata Extractor

This is a package to extract a domain, title, publication date, text, and language content from the URL or text of an online news story. The methods for each are extracted from the larger Media Cloud project, but also build on numerous 3rd party libraries. The metadata extracted includes:

the original URL of publication
a normalized URL useful for de-duplication
the canonical domain published on
the date of publication
the primary language used in the article text
the title of the article
a normalized title useful for de-duplication
the text content of the news article
the name of the library used to extract the article content

Other often-reused methods and configuration related to the mediacloud service also live in this package.

Installation

pip install mediacloud-metadata

Usage

If you pass in a URL, it will follow redirects and fetch the HTML for you.

from mcmetadata import extract
metadata = extract(url="https://my.awesome.news/story-path")

You can also pass in HTML you already have on hand. Note that in this case it is also useful to pass in the URL because that is used for some for some of the metadata extraction.

from mcmetadata import extract
metadata = extract(url="https://my.awesome.news/story-path",
                   html_text="<html><head><title>my webpage ... </html>")

Development

If you are interested in adding code to this module, first clone the GitHub repository.

Installing

flit install
pre-commit install

Testing

pytest

Distributing a New Version

Run pytest to make sure all the test pass
Update the version number in pyproject.toml
Make a brief note in the CHANGELOG.md about what changes
Commit the changes
Tag the commit with a semantic version number - v*.*.*
Push to repo to GitHub

Test Cache

Test are run against fixtures by default. This can be changed with the use of '--use-cache=False' when running tests. When adding new tests, re-run 'scripts/get-test-web-content.py'

Contributors

Created as part of the Media Cloud Project. Contributes include:

Rahul Bhargava (Media Cloud, Northeastern University)
Paige Gulley (Media Cloud)
Phil Budne (Media Cloud)
Vangelis Banos (Internet Archive)

Metadata

Release files for mediacloud-metadata 2.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mediacloud-metadata 2.0.0
File	Size	Uploaded
mediacloud_metadata-2.0.0.tar.gz	9.5 MB	Oct 6, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for mediacloud-metadata 2.0.0
File	Interpreter	ABI	Platform	Reset
mediacloud_metadata-2.0.0-py3-none-any.whl 9.6 MB Oct 6, 2026	Python 3	none	any	Details

Total release size: 19.1 MB

Release files / mediacloud_metadata-2.0.0.tar.gz

Download URL	mediacloud_metadata-2.0.0.tar.gz
Size	9.5 MB
Tags	Source
SHA-256 checksum How to use checksums	`07de2683cfd3ede8f12dd4663eb82767b00474ffa1270e5bb194f54c8fa3f713`
BLAKE2b-256 checksum How to use checksums	`153fba10bdc75a4003a9b64a8349df35f4cd19026a67beee370e39afbaf94cd3`
Upload date	Oct 6, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/mediacloud/metadata-lib
Publishing commit: github.com/mediacloud/metadata-lib/tree/f02ae1e154ec3ec7578c7431134eaffecf370783

Workflow

Publishing configuration: github.com/mediacloud/metadata-lib/blob/f02ae1e154ec3ec7578c7431134eaffecf370783/.github/workflows/publish-to-pypi.yml
Publishing logs: github.com/mediacloud/metadata-lib/actions/runs/37437383657/attempts/1

Artifact

Subject: mediacloud_metadata-2.0.0.tar.gz
SHA-256 checksum: 07de2683cfd3ede8f12dd4663eb82767b00474ffa1270e5bb194f54c8fa3f713 Verifying attestations

Release files / mediacloud_metadata-2.0.0-py3-none-any.whl

Download URL	mediacloud_metadata-2.0.0-py3-none-any.whl
Size	9.6 MB
Tags	Python 3
SHA-256 checksum How to use checksums	`09354c2f8680e2882948dfc634641ce509a951981386bd660dee278d43b369cc`
BLAKE2b-256 checksum How to use checksums	`a710d0676b17ff63c4086c6dbfd852bb8b6b49fc3a839b2f0164a47f92ec53ef`
Upload date	Oct 6, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/mediacloud/metadata-lib
Publishing commit: github.com/mediacloud/metadata-lib/tree/f02ae1e154ec3ec7578c7431134eaffecf370783

Workflow

Publishing configuration: github.com/mediacloud/metadata-lib/blob/f02ae1e154ec3ec7578c7431134eaffecf370783/.github/workflows/publish-to-pypi.yml
Publishing logs: github.com/mediacloud/metadata-lib/actions/runs/37437383657/attempts/1

Artifact

Subject: mediacloud_metadata-2.0.0-py3-none-any.whl
SHA-256 checksum: 09354c2f8680e2882948dfc634641ce509a951981386bd660dee278d43b369cc Verifying attestations