minet is a webmining command line tool & library for python (>= 3.8) that can be used to collect and extract data from a large variety of web sources such as raw webpages, Facebook, YouTube, Twitter, Media Cloud etc.
It adopts a very simple approach to various webmining problems by letting you perform a wide array of tasks from the comfort of the command line. No database needed: raw CSV files should be sufficient to do most of the work.
In addition, minet also exposes its high-level programmatic interface as a python library so you remain free to use its utilities to suit your use-cases better.
minet is developed by médialab SciencesPo research engineers and is the consolidation of more than a decade of webmining practices targeted at social sciences.
As such, it has been designed to be:
- low-tech, as it requires minimal resources such as memory, CPUs or hard drive space and should be able to work on any low-cost PC.
- fault-tolerant, as it is able to recover when network is bad and retry HTTP calls when suitable. What's more, most of minet commands can be resumed if aborted and are designed to run for a long time (think days or months) without leaking memory.
- unix-compliant, as it can be piped easily and know how to work with the usual streams.
Shortcuts: Command line documentation, Python library documentation.
How to cite?
minet is published on Zenodo as 10.5281/zenodo.4564399.
You can cite it thusly:
Guillaume Plique, Pauline Breteau, Jules Farjas, Héloïse Théro, Jean Descamps, Amélie Pellé, Laura Miguel, César Pichon, Kelly Christensen, Julien Pontoire, Béatrice Mazoyer, & Evan Chevalerias. (2025, Mai 14). Minet, a webmining CLI tool & library for python. (4.1.0). Zenodo. https://doi.org/10.5281/zenodo.15411918
Whirlwind tour
# Downloading large amount of urls as fast as possible
minet fetch url -i urls.csv > report.csv
# Extracting raw text from the downloaded HTML files
minet extract -i report.csv -I downloaded > extracted.csv
# Scraping the urls found in the downloaded HTML files
minet scrape urls -i report.csv -I downloaded > scraped_urls.csv
# Parsing & normalizing the scraped urls
minet url-parse scraped_url -i scraped_urls.csv > parsed_urls.csv
# Scraping data from Twitter
minet twitter scrape tweets "from:medialab_ScPo" > tweets.csv
# Printing a command's help
minet twitter scrape -h
# Searching videos on YouTube
minet youtube search -k "MY-YT-API-KEY" "médialab" > videos.csv
Summary
- What it does
- Documented use cases
- Features (from a technical standpoint)
- Installation
- Upgrading
- Uninstallation
- Documentation
- Contributing
What it does
Minet can single-handedly:
- Extract URLs from a text file (or a table)
- Parse URLs (get useful information, with Facebook- and Youtube-specific stuff)
- Join two CSV files by matching the columns containing URLs
- From a list of URLs, resolve their redirections
- ...and check their HTTP status
- ...and download the HTML
- ...and extract hyperlinks
- ...and extract the text content and other metadata (title...)
- ...and scrape structured data (using a declarative language to define your heuristics)
- Crawl (using a declarative language to define a browsing behavior, and what to harvest)
- Mine or search:
- Scrape (without requiring special access, often just a user account):
- Grab & dump cookies from your browser
- Dump Hyphe data
Documented use cases
- Fetching a large amount of urls
- Joining 2 CSV files by urls
- Using minet from a Jupyter notebook (very useful to experiment with the tool or teach students)
- Downloading images associated with a given hashtag on Twitter
- Scraping DSL Tutorial
Features (from a technical standpoint)
- Multithreaded, memory-efficient fetching from the web.
- Multithreaded, scalable crawling.
- Multiprocessed raw text content extraction from HTML pages.
- Multiprocessed scraping from HTML pages.
- URL-related heuristics utilities such as extraction, normalization and matching.
- Data collection from various APIs such as YouTube.
Installation
minet can be installed as a standalone CLI tool (currently only on mac >= 10.14, ubuntu & similar) by running the following command in your terminal:
curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/install.sh | bash
Don't trust us enough to pipe the result of a HTTP request into bash? We wouldn't either, so feel free to read the installation script here and run it on your end if you prefer.
On ubuntu & similar you might need to install curl and unzip before running the installation script if you don't already have it:
sudo apt-get install curl unzip
Else, minet can be installed directly as a python CLI tool and library using pip:
pip install minet
Finally if you want to install the standalone binaries by yourself (even for windows) you can find them in each release here.
Upgrading
To upgrade the standalone version, simply run the install script once again:
curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/install.sh | bash
To upgrade the python version you can use pip thusly:
pip install -U minet
Uninstallation
To uninstall the standalone version:
curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/uninstall.sh | bash
To uninstall the python version:
pip uninstall minet
Documentation
Contributing
To contribute to minet you can check out this documentation.
1 maintainer
Release files for minet 4.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| minet-4.2.1.tar.gz | 239.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| minet-4.2.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 562.6 kB
Release files / minet-4.2.1.tar.gz
| Download URL | minet-4.2.1.tar.gz |
|---|---|
| Size | 239.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dc49c56a2bfd8bd0657df02c4928b9b4a12470e43fa6971f06bcef51032b9543
|
|
BLAKE2b-256 checksum How to use checksums |
659e4f684a120501ddd908cf6ebd9de32aaa18d6bea1a72f387fd849c2d93c71
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Release files / minet-4.2.1-py3-none-any.whl
| Download URL | minet-4.2.1-py3-none-any.whl |
|---|---|
| Size | 322.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3cd8fea231eba75d6008b2fc70990dfdc6230b105bf72d5b9686f27e3ac4b584
|
|
BLAKE2b-256 checksum How to use checksums |
078e8ab284409452c7d38acfda6c2090a87a250c11869f0fdefd9b705973321f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
