Skip to main content

English | 日本語 | 中文 | Español | Français | हिन्दी | Italiano | Português (BR)

rnd-data

CI CC BY 4.0 License Landing Page

rnd-data

The raw data behind the studio's research: every run's receipts and outputs, with a manifest that pins each file by SHA-256. It's public, because the studio's research is open by default. The few files with a named reason to be private are held in the private companion, rnd-data-private, and no private file's hash is published here.

The repository holds data, plus its MIT read-only package under src/ (rnd-data: ls, show, where, schema). rnd writes and checks it (rnd raw ingest, rnd raw verify, rnd where). rnd's index reads its manifests when this checkout sits beside the rnd library, or is named by $RND_DATA. The repository's own scripts are verify.sh and its two workflows; the .sh and .py files under runs/ are receipts of runs, kept as data.

The handbook has the manifest field by field, the private-file rules and the checks.

What's here

Path What
runs/<id>/manifest.json One run's files (schema rnd-data-run/v1): each file's path, SHA-256, size, tier, licence and visibility, and every copy
runs/<id>/files/… That run's small public files (256 KB or less), in git
store.json The stores that hold bytes outside this repository, and how a machine finds each one
src/rnd_data/ The MIT read-only Python package (rnd-data: ls, show, where, schema) and the manifest JSON Schemas
  • Files over 256 KB are stored once, by content, at blobs/<sha256[:2]>/<sha256>. Every run that has one names it by hash, so a build several windows used is stored once.
  • They go public as a Hugging Face dataset at the next phase. Until then they're on the studio's backup drive.

A run id

A run id is its source folder, with / written as --. For example, receipts--colab--window14b-…-20261019 is the Publisher's receipts/colab/window14b-…-20261019.

  • Its files. A run's files are those under the nearest folder holding an outcome.json, so each retry (try1/) is its own run.
  • Files outside a run folder are grouped under their window folder (for Colab), or under their top folder.

An experiment's runs[].raw_manifest_id in rnd names one of these ids.

Private files

A file is private only for a named reason: sealed or held-out keys, blind files, item text, unapproved plans, private projects, or content the studio's identity scan flags. Its copy is in rnd-data-private, and its run's manifest says only this much about it:

  • Private for its content: it's left out, and counted by reason in the manifest's private_files.
  • Private only for identity: it's listed by path and size, with "visibility": "private", "private_why": "identity" and "sha256": null.

No private file's hash is published, since a short label file's hash could be matched by guessing. Everything else is public.

Copies and verifying

A file's copies are listed in its manifest's copies. Each store's root is found per machine, through store.local.json (gitignored) or $RND_DATA_STORE_<NAME>, so no machine path is committed.

Every public file's SHA-256 is in its run's manifest, so a downloaded copy can be checked without rnd, or with rnd raw verify. By hand, from the repository root (any platform with sha256sum; jq for the second):

sha256sum runs/<id>/files/<path>      # compare with that file's "sha256" in runs/<id>/manifest.json
jq -r '.copies[] | select(.store == "rnd-data") | "\(.sha256)  \(.location)"' runs/*/manifest.json \
  | sha256sum -c --quiet              # every git copy at once; silent when all match

With rnd installed, bash verify.sh runs rnd raw verify --public-only, as CI does on every pull request: it re-hashes every git copy, skips the stores a public clone doesn't have, and fails on a tracked file under runs/ that no manifest names, a private entry with a hash, the private tag, a private-looking file name, or a machine path in a manifest. On the rig, where the blob store is configured, bash verify.sh --full runs the full rnd raw verify.

Nothing is deleted from the studio rig until two verified copies exist, a restore test has passed, and the Director has given the go.

Owners

  • R&D owns the layout and the tooling.
  • The Publisher owns its receipts, and reviews every ingest of them.

Security and trust

  • What it holds: run manifests, the runs' small public files, store.json, the MIT read-only package under src/, tests/, verify.sh, two workflows and the site.
  • What it never holds: credentials; a private file's bytes or its hash; machine paths.
  • What touches it: rnd's ingest (rnd raw ingest) writes every manifest and every file under runs/, with the Publisher's review of every ingest of its receipts. CI checks every pull request with read-only permissions.
  • No telemetry is collected or sent; the site is static.

To report a private file that reached this repository, a published hash of a private file, or identity in a public file, see SECURITY.md.

Licence

rnd-data is licensed like rnd:

  • The data is CC BY 4.0.
  • The package and scripts (src/, tests/, *.sh, *.py) are MIT.
  • A third-party file keeps the licence its manifest names. The llama.cpp build is MIT (commit 620d2225).

A file's manifest licence field is the final word for that file. Every public file's manifest names its licence; private entries carry "licence": null, because they aren't published.

Status

Maintained, and growing with each ingest. There are no releases: each run is pinned by its manifest, and the repository by its commit. CHANGELOG.md lists the changes by date.

Metadata

Release files for rnd-data 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for rnd-data 0.1.0
File Size Uploaded Provenance Actions
rnd_data-0.1.0.tar.gz 27.3 kB With provenance Details for rnd_data-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for rnd-data 0.1.0
File Interpreter ABI Platform Actions
rnd_data-0.1.0-py3-none-any.whl Python 3 none any Details for rnd_data-0.1.0-py3-none-any.whl

Total release size: 45.9 kB

Release files / rnd_data-0.1.0.tar.gz

Download URL rnd_data-0.1.0.tar.gz
Size 27.3 kB
Tags Source
SHA-256 checksum
How to use checksums
b2639974d0e088e911e737ada6a3a5689c57a3d84ef807455d00dfa871923880
BLAKE2b-256 checksum
How to use checksums
5acb4f7487b0945bbc21fda737457a5b2a8d324e890a744b4dc8897988843eaf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

Release files / rnd_data-0.1.0-py3-none-any.whl

Download URL rnd_data-0.1.0-py3-none-any.whl
Size 18.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
dd1326261d39c1ad6472bda5061972e0b25a029e08cecc071c9994c91be1fd11
BLAKE2b-256 checksum
How to use checksums
965d6e5ed2687da799e0b5307a001142287d59ab6cbb51dcb2271d09afc15eb7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

Security for rnd-data 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for rnd_data-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page