RIDA Python library — 0.3.1

RIDA means Range Indexed Data Access. It is an installable Python library and terminal tool that builds one .rida file from a collection of JSON records. The file contains:

• a fixed 128-byte RIDA header;
• small, readable UTF-8 metadata at the start of the file;
• a compressed dense or sparse address catalog; and
• field values stored in a data region.

The reader loads the catalog once and then reads only the requested field bytes. It works locally and over an HTTP server that supports byte-range requests.

RIDA 0.3.1 writes the RIDA 2 layout by default: every stored field value is its own gzip member. A reader still fetches exactly one value range, then inflates that one member. On RIDA's reference dataset (27,647 records, 34 fields) the data region drops from 191,378,148 to 110,924,973 bytes, and the average stored value drops from 244 to 143 bytes. Older RIDA 1 files stay readable, and --value-codec plain still writes the original RIDA 1 layout byte for byte.

The figures in this document that say "the reference dataset" all come from that one measured file. It is the dataset RIDA was built against; it is not a special case, and any JSON collection of the same shape behaves the same way.

Install locally

From this project folder:

python3 -m pip install .

After that, the rida command is available in the same Python environment. The library uses only Python's standard library at runtime: no database, cloud account, or server process is required.

Install the published package with:

python3 -m pip install rida-format

The PyPI package name is rida-format. After installation, the terminal command and Python import name are both rida. Check the installed version with:

rida --version

Library project files

Keep the source project in one folder, for example rida/:

rida/
├── pyproject.toml           package name and install command
├── MANIFEST.in              files included in the public source archive
├── RIDA-README.md           usage instructions
├── RIDA-FORMAT.md           byte-level file format specification
├── CHANGELOG.md             release history
├── LICENSE                  RIDA license
├── docs/
│   └── rida-metadata.html   optional local metadata helper
├── tests/
│   ├── test_rida.py         test suite
│   ├── make_fixtures.py     regenerates the RIDA 1 golden fixtures
│   └── fixtures/            RIDA 1 files written by the 0.1.1 library
└── src/
    └── rida/
        ├── __init__.py      RIDA builder and reader library
        └── __main__.py      enables: python -m rida

src/rida/ is internal library code. After installation, Python copies it into its own site-packages/rida/ location. A developer never needs to run an individual .py file: they use import rida, rida ..., or python -m rida.

Requirements and downloads

• Python 3.10 or newer and pip are required.
• RIDA has zero third-party runtime dependencies. It uses only Python's built-in json, gzip, lzma, hashlib, urllib, and related standard modules.
• The rida command has its own progress display for loading, scanning, value writing, catalog compression, and final file writing. It does not require tqdm.
• It does not install or require AWS SDK, Cloudflare SDK, SQLite, Node.js, Rust, a database server, or a Worker.
• When installing this source project, pip may temporarily download setuptools (version 77.0.3 or newer) to build the package. It is an install tool, not a RIDA runtime dependency.
• Official Python installers for macOS and Windows include the required lzma support. On an unusually minimal Linux Python build, install the operating system's python3-lzma package before using RIDA.

Build

Scan first. It reports the field density and recommends dense or sparse mode:

rida scan users.json

Build a RIDA file. The command asks for confirmation unless --yes is supplied:

rida build users.json users.rida \
  --title "Users dataset" \
  --description "Public sample data" \
  --yes

During a build, the terminal shows RIDA's own progress, for example:

Reading JSON bytes: 217,221,288/217,221,288 (100%)
Parsing JSON: 1/1 (100%)
Scanning field names: 39,098/39,098 (100%)
Measuring values: 39,098/39,098 (100%)
Writing value data: 39,098/39,098 (100%)
Compressing catalog: 1/1 (100%)
Writing RIDA file: 217,221,288/217,221,288 (100%)

For large inputs, low-memory progress reports the record count as it reads. When the total is not known in advance, updates appear every 1,000 records.

Input can be:

• an object that maps IDs to JSON record objects;
• an array of JSON record objects, with --id-key; or
• a folder containing individual JSON record files. Filenames become IDs.

rida build records/ records.rida --yes
rida build users-array.json users.rida --id-key id --yes

If the input folder has JSON files in more than one place, RIDA lists the groups and asks which folder contains the records. For example, if a folder has config.json at its top level and thousands of record files inside items/, choose the items/ group. For a command that runs without a prompt, select it explicitly:

rida scan "./my-data" --records-dir items
rida build "./my-data" my-data.rida --records-dir items --yes

--records-dir is a folder inside the input folder. RIDA searches that folder and its subfolders for .json files; it does not mix separate groups unless they are all inside the selected folder. The scan prints the number of records before a build starts.

Use --mode dense or --mode sparse to override the recommendation. The builder does not overwrite an existing output file.

Choosing a value codec

--value-codec selects the stored value layout:

rida build users.json users.rida --value-codec plain --yes

Each gzip member carries a CRC-32 and a length, so a damaged RIDA 2 value fails to inflate instead of returning wrong data. Values that are only a few bytes long can be larger when gzipped, because a gzip member has about 20 bytes of overhead; scan measures the real stored size for the codec you select, so its estimates and recommendation match the file you are about to write.

Memory modes for large inputs

The default --memory-mode auto estimates the input's working-memory needs and checks currently available RAM. Choose a mode explicitly when needed:

rida build records.json records.rida --id-key id --memory-mode low-memory --yes
rida build records/ records.rida --memory-mode fast --yes

low-memory reads one JSON record at a time instead of keeping the whole JSON input in RAM. For one JSON file, supported layouts are an object mapping IDs to record objects, or an array of record objects with --id-key. For a folder, RIDA reads one record file at a time. The current record and the address catalog still use RAM, so this is not a zero-memory mode; an individual record must fit in memory.

When you pass an explicit --mode dense or --mode sparse, the low-memory builder stages only that one index layout instead of both, which lowers its peak catalog memory and CPU.

If the input folder contains separate JSON groups, RIDA asks which group to use. For non-interactive commands, pass --records-dir, for example:

rida build "./my-data" my-data.rida \
  --records-dir items --memory-mode low-memory --yes

Read locally

rida inspect users.rida
rida fields users.rida
rida get users.rida ali city

inspect prints the header and the embedded metadata as JSON. get prints one value and exits with status 1 and a one-line message if the record ID or field name does not exist.

fields answers "which key do I ask for?" — the field names live inside the file, so nothing has to be guessed or documented separately:

$ rida fields users.rida
Records: 27,647
Fields: 4
   1  id
   2  name
   3  city
   4  joined

It also accepts an http(s) URL, and --json prints just the array.

Check a file for damage

Reading one field never hashes the whole data region, so a normal lookup trusts the transport. verify is the explicit, slower check: it streams every data byte and compares it with the SHA-256 digest recorded in the header.

rida verify users.rida

{
  "file": "users.rida",
  "format_version": 2,
  "value_codec": "gzip-per-field",
  "record_count": 27647,
  "data_bytes": 110924973,
  "size_ok": true,
  "metadata_ok": true,
  "catalog_ok": true,
  "data_ok": true,
  "ok": true
}

verify exits with status 1 when ok is false. From Python, use rida.verify("users.rida"), or RidaFile.verify_data() on an already open file.

Where RIDA can read data

RIDA 0.3.1 can read a .rida file from the local computer or from a direct HTTP/HTTPS file URL. For remote reads, RIDA sends ordinary HTTP byte-range requests; it does not use a cloud-provider SDK. This makes it compatible with object storage that returns the requested bytes as HTTP 206 Partial Content and includes the matching Content-Range header.

Examples expected to work because their object APIs support byte-range reads include Cloudflare R2, Amazon S3, Google Cloud Storage, Azure Blob Storage, Backblaze B2, and Bunny.net through a Pull Zone. See their docs: R2, S3, Google Cloud Storage, Azure Blob Storage, Backblaze B2, and Bunny.net. This is protocol-based expected compatibility, not a separate integration with each provider. The exact URL, permissions, and any CDN in front of a bucket still need to allow byte-range responses; the providers have not all been individually tested with this RIDA release.

For Bunny Storage, connect the Storage Zone to a Pull Zone and use the Pull Zone URL (or its custom hostname) for the file. Bunny documents range support for cached content by default; enable Cache Slicing in the Pull Zone for uncached content.

The current reader sends a Range header and a rida/<version> User-Agent, but does not add provider-specific authentication headers. Use a public object URL or a signed URL that permits the range request. RIDA does not upload files to cloud storage: upload the .rida file using that storage provider, then give RIDA the file URL.

Read from R2 or another HTTP URL

After uploading the .rida file to an HTTP byte-range server:

rida remote-get https://data.example.com/users.rida ali city

The remote reader fetches the 128-byte header, then the metadata and catalog, then only the requested field range. It checks that the server returns HTTP 206 and the correct Content-Range; if the server sends the whole file instead, RIDA stops with an error rather than silently downloading it all.

Catalog cache

The metadata and catalog are cached on disk, keyed by URL. Every open re-fetches the 128-byte header and uses it to validate the cache, because the header already contains the metadata length, the catalog length and the catalog digest. The result is that a warm lookup needs two requests instead of three and no longer downloads the catalog at all. For the reference dataset, whose compressed catalog is 536,472 bytes, that removes roughly 537 KB from every lookup.

rida remote-get https://data.example.com/users.rida ali city
rida remote-get https://data.example.com/users.rida ali city --no-cache
rida remote-get https://data.example.com/users.rida ali city --timeout 10
rida cache            # show the cache folder and size
rida cache --clear    # delete cached catalogs

The cache lives in the user cache folder (~/.cache/rida on Linux and macOS). Set RIDA_CACHE_DIR, or pass --cache-dir, or pass cache_dir= to rida.RemoteRidaFile, to move it. A remote object that changes invalidates its own cache entry automatically, because the header no longer matches.

Network reads fail after --timeout seconds (default 30) instead of hanging.

Connection reuse

A range request sent over a fresh TLS connection pays a full handshake, which on a CDN costs several times more than the lookup itself. The remote reader therefore keeps one HTTP(S) connection open per reading thread and reuses it, so a warm lookup spends its time on the value, not on the handshake.

Measured against Bunny CDN on the reference dataset:

Use --no-keepalive, or keepalive=False in Python, to get the old behaviour.

Asking for several fields at once

A request header costs more than a small payload. Measured against Bunny, every response carries about 660 bytes of headers, while a stored field value in the reference dataset averages a few hundred bytes. Many small single-field lookups therefore spend most of their bytes on headers.

Because a record's values are stored next to each other, get_many and get_record fetch several fields in one range request:

Two fields are merged into one request when the bytes between them are cheaper than the request header they save (MERGE_GAP_LIMIT, 660 bytes), and stay separate otherwise — a field asked on its own is still exactly one request.

Use from Python

import rida

# Scan first if you want to show the publisher the recommendation.
report = rida.scan("users.json")
print(report.recommended_mode)  # "dense" or "sparse"

# Build once. The source JSON stays unchanged.
rida.build("users.json", "users.rida", title="Users dataset")

# Read only one stored value from a local RIDA file.
with rida.RidaFile("users.rida") as data:
    print(data.get("ali", "city"))

# Read several fields, or the whole record, in one request.
with rida.RidaFile("users.rida") as data:
    print(data.get_many("ali", ["name", "city"]))
    print(data.get_record("ali"))

# Check the whole file against its stored digests.
print(rida.verify("users.rida")["ok"])

# The same lookups work against an R2/public URL with Range support.
remote = rida.RemoteRidaFile("https://data.example.com/users.rida")
print(remote.get("fatima", "languages"))
print(remote.get_many("fatima", ["languages", "city"]))
print(remote.cache_hit)  # True when the cached catalog was reused

get_many raises KeyError for a field name the file does not have, so a typo never passes silently. A field the file has but this record lacks is simply left out of the returned dictionary.

A desktop app or another Python program can show the same stages in its own interface by passing a callback:

def progress(stage, completed, total):
    print(f"{stage}: {completed}/{total}")

rida.build("users.json", "users.rida", progress=progress)

Useful names:

Run the tests

python3 -m unittest discover -s tests -v

The suite needs no third-party packages and does not use the network. It starts its own local range server for the remote tests. tests/fixtures contains RIDA 1 files written by the published 0.1.1 library; the suite checks that this version still reads them, and still reproduces their catalog and data bytes with --value-codec plain (metadata now differs by the one removed key). Regenerate those fixtures only with tests/make_fixtures.py pointed at an older library.

Current scope

RIDA 0.3.1 implements primary lookup — record ID + field name → value — plus multi-field reads (get_many, get_record) and a whole-file integrity check. Optional exact-match, numeric-range, and Unicode full-text search indexes are not part of this release, and neither is a JavaScript reader or a persistent catalog cache for the browser.

License

RIDA is distributed under the RIDA Proprietary License in the LICENSE file. You may download and modify RIDA, and use it in free or paid applications. You may not sell or redistribute RIDA as a standalone package or library. Applications that include RIDA must show the attribution notice specified in the license.