rideralerts-scraper

CLI for continuously polling vehicle positions from transit agencies hosted on rideralerts.com InfoPoint (Avail AVL) REST APIs into a SQLite database. Known agencies: Montebello Bus Lines (mbl.rideralerts.com) and Norwalk Transit (nts.rideralerts.com); any other InfoPoint tenant works by subdomain or base URL.

uv run rideralerts-scraper run -a montebello -o vehicle_positions.sqlite   # runs until ^C
uv run rideralerts-scraper run -a mbl -a norwalk \
    -o vehicle_positions.sqlite     # poll both agencies into one db
uv run rideralerts-scraper run -a norwalk -o vehicle_positions.sqlite \
    --within query.geojson          # only store positions inside the boundary
uv run rideralerts-scraper run -a mbl -o vehicle_positions.sqlite \
    --routes 10,40                  # only poll lines 10 + 40
uv run rideralerts-scraper stats -o vehicle_positions.sqlite
uv run rideralerts-scraper vehicles -o vehicle_positions.sqlite \
    -a norwalk                      # only list Norwalk's vehicles

--agency accepts a known name or alias (montebello/mbl, norwalk/nts), a bare rideralerts.com subdomain (sta), a hostname (sta.rideralerts.com), or a full InfoPoint base URL (https://sta.rideralerts.com/InfoPoint). The stored agency slug is the subdomain (first hostname label), so montebello and mbl are the same agency.

Each poll hits GetAllVehiclesForRoutes and returns the full current status of every bus on the selected routes; responses are decomposed and flushed to SQLite once per poll. schema.sql splits the payload into a history table (vehicle_status, deduped on (agency, vehicle_id, observed_at)) with a trigger maintaining the latest-state table vehicle; each agency's route directory from GetVisibleRoutes lands in route at the start of each run. One database can hold any number of agencies — every table is keyed by the agency slug, since vehicle and route ids are only meaningful within one agency. The raw payload JSON is not stored — every field the API sends has its own column, with the .NET-style /Date(<epoch_ms><±hhmm>)/ timestamps normalized to unix seconds. The insert statements live in procedures.sql. The typed wrapper _queries.py is codegen'd from both SQL files — after editing either, regenerate with make (runs solite codegen piped through tools/codegen.py; solite is pinned to 0.0.1a36 because later prereleases stopped embedding the schema in the generated setup).

CLI Reference

run — poll vehicle statuses into SQLite until interrupted

Fetches each agency's route directory once at startup, then polls the vehicles endpoint on an interval and inserts status snapshots. Databases are created in WAL mode. Filters (--routes, --within) compose with AND; filtered statuses are dropped before they reach the database.

Log lines are prefixed with the agency slug and count every drop reason: duplicate (the API re-serves each vehicle's unchanged status every poll; deduped by primary key), outside boundary (--within), invalid (unparseable or failed poll responses).

vehicles — list the latest known position of each vehicle

One line per vehicle from the vehicle table (agency, route, fleet name, lat/lon, speed, schedule deviation, status, age of last observation), ordered by agency then route. Opens the database read-only, so it's safe against a live run.

stats — print row counts and time range

One-shot summary: per-agency counts of statuses, vehicles, and routes, plus the received_at time span of the stored history. Read-only.