mostlyright-data

Browse datasets built by our community or create your own with agents that find the sources and configure the pipeline. Mostly Right treats a successful run as one readable dataset version; publishing that version does not by itself prove that catch-up or scheduled refresh is running.

mostlyright-data is the command-line client. It probes sources, registers one recipe document, starts and follows hosted dataset runs, reads what they delivered, and downloads the datasets after they finish.

Requirements

Use CPython 3.11 or newer.

Install

python -m pip install mostlyright-data

That is the whole install. There is no second profile and no lane to choose: every mr-data command sends its work to the backend, and it works on Linux, macOS, and Windows.

It also carries the mr-data-build agent skill, which Claude Code and Codex read to author a recipe without rediscovering the contract a run at a time. The first mr-data command you run places it in your agent's skill directory; there is nothing to install by hand. A skill you have edited yourself is never replaced, and MOSTLYRIGHT_SKILL_AUTOINSTALL=0 switches the placement off.

Sign in

mr-data login
mr-data whoami

login stores a device key on the current machine. whoami checks that key.

New macOS/Linux logins use ~/.mostlyright/credentials.secure, an unencrypted file restricted to your OS account (0600 inside a 0700 directory). Processes running as you and backups can read it. Older file-based logins migrate to this file without requesting Keychain access. Windows keeps DPAPI encryption as its default. Existing Keychain, DPAPI, or Secret Service logins keep their selected store; an unavailable store never causes an automatic switch to file storage.

If an existing macOS login requests Keychain access, it is requesting the md.mostlyright.cli device credential. To switch that login to file storage, restore access to its Keychain item, run mr-data auth logout successfully, then run mr-data login. The --credential-store secure-file option does not move an existing native-store login.

Check the installation

mr-data --version
mr-data --help

Build a dataset

Create the dataset, choose its primary category from the fixed vocabulary, then register a recipe document with the sources, table schema, transformations and checks. Use the created dataset ID in the recipe:

mr-data dataset create --name "Dataset title" --json
mr-data dataset categories --json
mr-data dataset set DATASET_ID --category climate-environment
mr-data recipe recipe.json --json
mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full
mr-data status RUN_ID
mr-data checks RUN_ID
mr-data download RUN_ID --output ./out

Full builds are progressive by default: one acquisition exposes an inspection checkpoint after about five minutes and continues toward the finished table. Inspect it with mr-data status RUN_ID --inspection --json; the checkpoint does not pause for approval or start another full acquisition. Any required spend confirmation happens before acquisition. When a run has sealed queryable table checkpoints, mr-data status RUN_ID --checkpoints --json lists their exact immutable identifiers and mr-data query RUN_ID "SELECT …" --checkpoint CHECKPOINT_ID reads one without advancing underneath the query. These workspace-private snapshots remain incomplete, keep checks pending, and never become the serving table. --progressive remains a compatibility alias. Use --sample only for an explicit standalone bounded inspection; row ceilings do not impose a five-minute acquisition limit.

Run submission is idempotent: sending identical recipe, mode and bounds returns the same run, including a previous failure. The CLI reports a non-queued result as run_returned and exits with code 2 when that run failed. Read its failure before retrying; mr-data run --retry RUN_ID can reissue supported platform execution failures without changing the recipe.

Use the identifiers returned by registration and run submission. mr-data dataset create creates a dataset page before a recipe is ready. mr-data watch RUN_ID follows a submitted run. A scheduled refresh runs only Studio's persisted bounded source-action plan; it never silently falls back to a mutable whole-source fetch. mr-data table resync TABLE_ID --request-id UUID explicitly requests a full source reread when that is needed. Choose and retain the UUID before submitting: if the response is lost, repeat that exact command with the same UUID rather than starting another reread. A persisted spend hold is still an accepted resync: its receipt names the run, projection and exact mr-data run --confirm-held RUN_ID action. New full resyncs use progressive acquisition too. Existing legacy sample-first pairs can still finish through their original confirmation and approval actions. mr-data recipe readiness reads Studio's paginated, immutable-revision inventory before a production-wide schedule sweep: predecessor, incremental-materialization and differential-proof readiness, together with each source's next action. It reports Studio's persisted facts; it does not authorize a schedule.

To request an offline replay of retained inputs with a registered revision:

mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --mode replay --sources-from RUN_ID

Studio must have replay enabled and the named successful run must belong to the same table with its raw inputs still retained. Replay compares against that run and never becomes the live version. Studio returns a typed refusal when replay is unavailable; the CLI does not fetch sources locally.

The hosted engine executes sample, full and refresh runs. Studio's action classifier can name closed-source reuse, request-window or collection acquisition, and recorded stream input, but normal refresh admits only two families of plan: every source a recorded stream, and at least one direct partition request window with a compatible predecessor, with closed reuse permitted beside it. Once such a plan holds more than one source, every window in it must agree on one merge.materialization, merge.partition.column and merge.partition.key, and that column must be one the table declares. Per-row request collections may join such a plan, each reading a window of the plan and carrying its partition column, with no closed source beside them. Closed-only plans, any plan carrying another kind of collection, a window beside a recorded stream, a mutable snapshot and any other unwindowed source return RESYNC_REQUIRED before acquisition. Nothing is revalidated or re-acquired whole as an implicit fallback. Use mr-data table resync TABLE_ID --request-id UUID when a deliberate full source reread is intended, retaining that UUID through an uncertain response. URL windows may use declared query parameters or path placeholders; a fixed date URL does not advance automatically. Configure a correction lookback where the publisher can revise earlier observations.

A source exposing only its current snapshot cannot supply a historical delta. It is supported only through an explicit table resync; refresh does not turn its whole current result into an incremental update. Recorded streams use their registered capture and continuation semantics.

See Recipe documents for source-window and bootstrap contracts. A table's first succeeded run goes live on its own, whatever mode it was; mr-data promote TABLE_ID records how often it refreshes and why, and puts back a table that was withdrawn. Claim a table is live and current only when Studio returns that evidence.

There is no local execution

There used to be. A second product lane built datasets on your own machine, and mr-data carried about fifty commands for making, running, serving, indexing, checking and signing a build here. That lane is gone: the engine it ran now ships only inside the backend's worker images.

mr-data --help lists every command this product has. The execution engine and its dependency closure live in Mostly Right Studio's apps/worker; this repository ships no engine extras, worker executables or image publisher.

Documentation

• Install and sign in
• Build your first dataset
• CLI reference
• Recipe reference
• Recipe examples
• Join market settlements
• Compare a forecast with observations
• Model a snapshot source — normal refresh is unavailable until its bounded materializer ships; use explicit table resync today
• Union many weather stations
• Aggregate a stream into bars
• Use a public dataset
• Certified document extraction
• Use a table to drive a stream's market roster

Use mr-data --help for the full command list and options. Commands that support --json write one JSON object.

--output may use a relative path.

Behavior and limits

• Weather Reader versions 1 and 2 accept one extracted GRIB2 record and refuse a multi-record GFS or HRRR file. Version 3 reads a bounded collection and selects exactly one sealed record.
• A single-address credential-free public HTTPS source uses the reviewed hosted route. Bounded collections have no route at all; the retired local lane was their only route.
• A recipe supports up to 256 declared sources. Source count is separate from bounded request concurrency and resource limits. Existing source adapters and Readers retain their own format, pagination, byte, row and credential contracts; a URL collection is not an implicit permission to crawl arbitrary links.
• A keyed stream venue is named by a credential reference the backend holds, never by a value. No command flag takes a secret as an argument. Venues reached over an actual WebSocket are covered; a venue whose credential comes from an interactive human login, or that is not WebSocket at all, is not.

Verification

make test

make test runs the thin client suite while editing. make verify is required before review or merge.

Development

uv sync --locked

There is no dev extra to ask for -- uv run --extra dev answers error: Extra `dev` is not defined in the project's `optional-dependencies` table. The test tooling is the default dependency group, which uv sync installs.

make lint is the repository's one lint set (uv run ruff check . plus uv lock --check); it is the first thing make verify runs and the same target the Linux leg of thin-client-smoke.yml runs, so the two cannot disagree about what is linted.

The test suite checks client behavior, contract compatibility and thin package ownership. Backend execution and worker-boundary tests run in Studio. The client supports Linux, macOS and Windows.

To update the public Studio contract pin, run make repin STUDIO_GIT=/path/to/studio STUDIO_REF=<commit>. It reads committed OpenAPI and JSON Schema bytes, records their hashes and source commit in vendor/studio-contracts/client-v4-pin.json, and updates the thin client. Run the cross-repository tests against that Studio revision before release. Worker/client generation and backend job-contract repins now belong to Studio.