pvstream

pvstream is a Rust library with python bindings allowing you to efficiently stream download, parse, and filter pageviews from Wikimedia's hourly dumps.

The library can be used from Rust or python. In both languages you can choose between an iterator of parsed objects, made available on the fly as the file is downloaded, or a complete parquet file of parsed and filtered data.

Installation

Rust

Add pvstream to your Cargo.toml:

[dependencies]
pvstream = "0.1"

Or use cargo-add:

cargo add pvstream

Python

Install from PyPI:

pip install pvstream

Building from Source

To build the Python package for your specific hardware:

pip install maturin
git clone https://github.com/vegardege/pvstream
cd pvstream
maturin develop --release

Or build a wheel:

maturin build --release
pip install target/wheels/pvstream-*.whl

Usage

There are four main entry points for this library:

They all accept similar filters. In python, Regex is a str, Vec is a list, u32 is an int:

Learn more about the format from Wikimedia's documentation.

Example (Rust):

use pvstream::filter::FilterBuilder;
use pvstream::stream_from_file;
use std::path::PathBuf;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Path to your pageviews file
    let path = PathBuf::from("pageviews-20240818-080000.gz");

    // View all English mobile sites containing the word 'Rust'
    let filter = FilterBuilder::new()
        .domain_codes(["en.m"])
        .page_title("Rust")
        .build();

    // Stream rows matching the filter
    let rows = stream_from_file(path, &filter)?;

    // Iterate over results
    for row in rows {
        match row {
            Ok(pageview) => println!("{:?}", pageview),
            Err(e) => eprintln!("Error parsing row: {:?}", e),
        }
    }

    Ok(())
}

Example (python):

import pvstream

rows = pvstream.stream_from_file(
    "pageviews-20240818-080000.gz",
    domain_codes=["en.m"],
    page_title="Rust",
)

for row in rows:
    print(row)