lightrider-huggingface

Hugging Face datasets with provable, reproducible origin. Each dataset is built from a 32-byte seed drawn from a named quantum entropy source through Light Rider. The draw's EMS-signed receipt commits to those exact bytes. Anyone can check where the dataset came from and regenerate it exactly, without a Light Rider account.

This is an alternative entropy source with verifiable provenance. It is not a claim that quantum randomness makes a dataset better than one built with a good classical generator.

Quickstart

pip install lightrider-huggingface
export LR_API_KEY=lr_live_...   # Light Rider API key (Light Rider Cloud > Settings > API Keys)
export HF_TOKEN=hf_...          # Hugging Face token with write access

import lightrider_huggingface as lrhf

# 500 random circuits for benchmarking: 5 qubits, depth 10 (one draw, 1 credit)
ds = lrhf.circuits(500, num_qubits=5, depth=10, gate_set=["h", "sx", "rz", "cx"], source="anu")

# 200 random rows of an existing dataset's test split, pinned to today's commit
sub = lrhf.eval_subset("openai/gsm8k", "test", n=200, config="main", source="anu")   # or fraction=0.1

lrhf.publish(ds, "my-org/random-circuits-5q")      # private by default; parquet + card + receipt
print(lrhf.verify("my-org/random-circuits-5q"))     # anyone can run this; no Light Rider key needed

ds.table is a pyarrow.Table (ds.to_pandas() for pandas), and ds.provenance is the full record. ds.save("dir") writes the same files locally, and lrhf.verify_dir("dir") checks them.

What gets published

Random circuits (random_circuits/1): one row per circuit with circuit_id, num_qubits, depth, num_gates, num_two_qubit_gates, instructions (JSON, the lightrider.Circuit format) and qasm (OpenQASM 2.0). Each of the depth layers acts on every qubit. Two-qubit gates pair neighbours in a fresh random permutation, and angles are uniform in [0, 2π). The full rule is in random_circuits.py.

Evaluation subsets (eval_subset/1): n distinct rows of one split, unchanged (images and audio stay encoded), plus _lr_source_index. The source is pinned to an exact commit sha, so the same rows can always be fetched again.

How the proof works

1. The seed is one draw's output. EMS signs output_commitment = sha256(salt || bytes) into the receipt and returns the salt with the bytes. The salt keeps short draws from being brute-forced out of a public receipt. The dataset publishes seed, salt and receipt, so sha256(salt || seed) == receipt.output_commitment ties the seed to that signed receipt. A genuine receipt from some other draw doesn't match.
2. The receipt is genuine. Its Ed25519 signature verifies against the key published at https://entropy.lightriderinc.com/api/v1/public-key. The signed message is the receipt without signature, keys sorted, compact JSON.
3. The rows follow from the seed. Generation uses LR-ChaCha20 v1 (prng.py), a short written spec on the RFC 8439 ChaCha20 stream. NumPy is deliberately not used: its Generator output may change between releases. Regeneration is compared row for row through a canonical content hash. When the verifier runs the same pyarrow version, the parquet file is also compared byte for byte.

What it can't show: that the publisher drew only once and didn't pick a favourite draw. The receipt's timestamp shows when this draw happened.

verify(repo_id, indices_only=True) checks a subset's row indices without re-downloading a large source split.

Tests

pip install -e ../lightrider -e ".[dev]"
pytest                    # mocked entropy site and mocked Hub
LRHF_LIVE=1 LR_API_KEY=lr_live_... HF_TOKEN=hf_... pytest tests/test_live.py   # production + Hub, 1 credit

The live test publishes to the private repo LightRiderInc/lr-hf-livetest. It overwrites the repo on every run and never deletes it.

See examples/provenance_datasets.py for a runnable example.