inklog 0.1.0

Analyst-friendly scrapers with TUI and uvx support

Inklog

Inklog is a collection of uvx-friendly scrapers for analysts. It provides a TUI for browsing available scrapers and a CLI for automation. The current focus is downloading files (PDF/EPUB). Webpage-to-markdown scraping is planned but not implemented yet.

All dates in this repository use the ISO8601 format: YYYY-MM-DD.

Quick start

Run the TUI

uvx inklog

List scrapers

uvx inklog list

Show example URLs

uvx inklog info malaysia_parliament

Run a scraper (CLI)

uvx inklog run malaysia_parliament 2025-11-01 2025-11-30

Run a single document type:

uvx inklog run malaysia_parliament:lower_house_hansard 2025-11-01 2025-11-30

TUI

The TUI shows the available scrapers, their descriptions, and lets you run a scraper with date ranges and options. Each Malaysia Parliament document type appears as its own row for quick runs.

Key bindings:

• Enter or Run button: run the selected scraper
• Example URLs can be opened from the details pane
• Q: quit

Malaysia Parliament scraper

This scraper downloads PDFs from the Malaysia Parliament site for the following document types:

• Dewan Rakyat - Jawapan Lisan (Oral Answers)
• Dewan Rakyat - Jawapan Bukan Lisan (Non-Oral Answers)
• Dewan Rakyat - Penyata Rasmi (Official Report)
• Dewan Negara - Jawapan Bertulis (Written Answers)

Filename rules

Some sources change their filename patterns over the years. Inklog models this using date-bound filename rules, so each document type can define a list of templates tied to start/end dates. The Malaysia Parliament scraper ships with the current pattern and a rules table that can be extended when historical patterns are known.

CLI options

inklog run supports overriding boolean options with --set:

uvx inklog run malaysia_parliament 2025-11-01 2025-11-30 \
  --set jawapan_lisan_rakyat=true \
  --set jawapan_bukan_lisan_rakyat=true \
  --set penyata_rasmi_rakyat=true \
  --set jawapan_bertulis_negara=false

Adding a new scraper

1. Copy src/inklog/scrapers/template.py and rename it.
2. Update ScraperMeta and implement run().
3. Ensure the module exports SCRAPER.

Scrapers are auto-discovered from the inklog.scrapers package.

Development

uv sync --extra dev
uv run ruff check --fix .
uv run ruff format .
uv run pytest

Roadmap

• Add markdown scraping mode (TBD).
• Expand filename rules for historical Malaysia Parliament patterns.