Skip to main content

A webmining CLI tool & library for python.

Project description

Build Status

Minet

minet is a webmining CLI tool & library for python that can be used to collect and extract data from a large variety of web sources such as raw webpages, Facebook, CrowdTangle, YouTube, Twitter, Media Cloud etc.

It adopts a lo-fi approach to various webmining problems by letting you perform a variety of actions from the comfort of your command line. No database needed: raw data files such as CSV should be sufficient to do the work.

In addition, minet also exposes its high-level programmatic interface as a python library so you can tweak its behavior at will.

Shortcuts: Cookbook, CLI Docs, Lib Docs.

Use cases

  • Downloading large amount of urls very fast. (guide)
  • Writing scrapers to extract structured data from HTML pages.
  • Writing crawlers to automatically browse the web.
  • Extract raw text content from HTML pages. (example)
  • Normalize batches of urls contained in a CSV file to perform relevant aggregations (dropping irrelevant query items, extracting domain name etc.) (example)
  • Join two CSV files based on columns containing urls that need to be matched (guide).
  • Collecting data from CrowdTangle API (to collect and search posts mainly from Facebook and Instagram).
  • Collecting data from Facebook (comments, likes etc.)
  • Parsing Facebook urls in a CSV file.
  • Collecting data from Twitter (users, followers, followees etc.)
  • Scraping data (tweets etc.) from Twitter's website public facing search API.
  • Collecting data from YouTube (captions, comments, video metadata etc.)
  • Parsing YouTube urls in a CSV file.
  • Dumping a Hyphe corpus.
  • Collecting data from Media Cloud (search stories, dump topics etc.).

Features (from a technical standpoint)

  • Multithreaded, memory-efficient fetching from the web.
  • Multithreaded, scalable crawling using a comfy DSL.
  • Multiprocessed raw text content extraction from HTML pages.
  • Multiprocessed scraping from HTML pages using a comfy DSL.
  • URL-related heuristics utilities such as extraction, normalization and matching.
  • Data collection from various APIs such as CrowdTangle.

Installation

minet can be installed as a standalone CLI tool (currently only on mac, ubuntu & similar) by running the following command in your terminal:

curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/install.sh | bash

Don't trust us enough to pipe the result of a HTTP request into bash? We wouldn't either, so feel free to read the installation script here and run it on your end if you prefer.

On ubuntu & similar you might need to install curl and unzip before running the installation script if you don't already have it:

sudo apt-get install curl unzip

Else, minet can be installed directly as a python CLI tool and library using pip:

pip install minet

If you need more help to install and use minet from scratch, you can check those installation documents.

Finally if you want to install the standalone binaries by yourself (even for windows) you can find them in each release here.

Upgrading

To upgrade the standalone version, simply run the install script once again:

curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/install.sh | bash

To upgrade the python version you can use pip thusly:

pip install -U minet

Uninstallation

To uninstall the standalone version:

curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/uninstall.sh | bash

To uninstall the python version:

pip uninstall minet

Cookbook

To learn how to use minet and understand how it may fit your use cases, you should definitely check out our Cookbook.

Usage

Contributing

To contribute to minet you can check out this documentation.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

minet-0.44.0.tar.gz (71.9 kB view hashes)

Uploaded Source

Built Distribution

minet-0.44.0-py3-none-any.whl (119.2 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page