RIDA Python library — 0.1

RIDA means Range Indexed Data Access. It is an installable Python library and terminal tool that builds one .rida file from a collection of JSON records. The file contains:

• a fixed RIDA header;
• small, readable UTF-8 metadata at the start of the file;
• a compressed dense or sparse address catalog; and
• UTF-8 JSON values stored in a data region.

The reader loads the catalog once and then reads only the requested field bytes. It works locally and over an HTTP server that supports byte-range requests.

Install locally

From this project folder:

python3 -m pip install .

After that, the rida command is available in the same Python environment. The library uses only Python's standard library at runtime: no database, cloud account, or server process is required.

After the package is published, another person can install it with:

python3 -m pip install rida-format

The public package name is provisionally rida; publishing to PyPI will happen only after the RIDA file format is frozen. The command and Python import name are both simply rida.

Library project files

Keep the source project in one folder, for example rida/:

rida/
├── pyproject.toml          package name and install command
├── MANIFEST.in             files included in the public source archive
├── RIDA-README.md           usage instructions
├── RIDA-DESIGN-0.1.md       RIDA format design
├── rida-metadata.html       optional browser metadata opener
└── src/
    └── rida/
        ├── __init__.py     RIDA builder and reader library
        └── __main__.py     enables: python -m rida

src/rida/ is internal library code. After installation, Python copies it into its own site-packages/rida/ location. A developer never needs to run an individual .py file: they use import rida, rida ..., or python -m rida.

Requirements and downloads

• Python 3.10 or newer and pip are required.
• RIDA has zero third-party runtime dependencies. It uses only Python's built-in json, lzma, hashlib, urllib, and related standard modules.
• The rida command has its own progress display for loading, scanning, value writing, catalog compression, and final file writing. It does not require tqdm.
• It does not install or require AWS SDK, Cloudflare SDK, SQLite, Node.js, Rust, a database server, or a Worker.
• When installing this source project, pip may temporarily download setuptools (version 68 or newer) to build the package. It is an install tool, not a RIDA runtime dependency.
• Official Python installers for macOS and Windows include the required lzma support. On an unusually minimal Linux Python build, install the operating system's python3-lzma package before using RIDA.

Build

Scan first. It reports the field density and recommends dense or sparse mode:

rida scan users.json

Build a RIDA file. The command asks for confirmation unless --yes is supplied:

rida build users.json users.rida \
  --title "Users dataset" \
  --description "Public sample data" \
  --reader-url "https://example.com/rida-explorer" \
  --yes

During a build, the terminal shows RIDA's own progress, for example:

Reading JSON bytes: 217,221,288/217,221,288 (100%)
Parsing JSON: 1/1 (100%)
Scanning field names: 39,098/39,098 (100%)
Measuring values: 39,098/39,098 (100%)
Writing value data: 39,098/39,098 (100%)
Compressing catalog: 1/1 (100%)
Writing RIDA file: 217,221,288/217,221,288 (100%)

For a folder of JSON record files, loading progress counts record files. For one large JSON file, it counts source bytes while reading, then shows a separate parsing stage. RIDA 0.1 still keeps a single JSON source in memory while parsing; the planned streaming importer is for very large single files.

Input can be:

• an object that maps IDs to JSON record objects;
• an array of JSON record objects, with --id-key; or
• a folder containing individual JSON record files. Filenames become IDs.

rida build records/ records.rida --yes
rida build users-array.json users.rida --id-key id --yes

Use --mode dense or --mode sparse to override the recommendation. The builder does not overwrite an existing output file.

Read locally

rida inspect users.rida
rida get users.rida ali city

Read from R2 or another public HTTP URL

After uploading the .rida file to an HTTP byte-range server:

rida remote-get https://data.example.com/users.rida ali city

The remote reader first fetches the 128-byte header and catalog, then fetches only the requested field range. For browser readers, configure CORS to allow GET, HEAD, and the Range request header, and expose Content-Range, Content-Length, and ETag.

One RemoteRidaFile object keeps that catalog in RAM, so later .get() calls on the same object fetch only the requested value range. The rida remote-get terminal command creates a new object each time, so persistent on-disk catalog cache is a later RIDA Reader feature.

Metadata opener page

Open rida-metadata.html in a browser and choose any .rida file. It reads only the embedded header and metadata on the local computer, then displays the title, record count, index mode, and RIDA Explorer download URL. It never uploads the file.

Use from Python

import rida

# Scan first if you want to show the publisher the recommendation.
report = rida.scan("users.json")
print(report.recommended_mode)  # "dense" or "sparse"

# Build once. The source JSON stays unchanged.
rida.build(
    "users.json",
    "users.rida",
    title="Users dataset",
    reader_url="https://example.com/rida",
)

# Read only one stored value from a local RIDA file.
with rida.RidaFile("users.rida") as data:
    print(data.get("ali", "city"))

# The same primary lookup works against an R2/public URL with Range support.
remote = rida.RemoteRidaFile("https://data.example.com/users.rida")
print(remote.get("fatima", "languages"))

A desktop app or another Python program can show the same stages in its own interface by passing a callback:

def progress(stage, completed, total):
    print(f"{stage}: {completed}/{total}")

rida.build("users.json", "users.rida", progress=progress)

Current scope

RIDA 0.1 implements primary lookup: record ID + field name → value.

The approved design for optional exact-match, numeric-range, and Unicode full-text indexes is in RIDA-DESIGN-0.1.md. Those search-index blocks are the next format layer; they are not claimed as implemented by this library yet.

License

RIDA is distributed under the RIDA Proprietary License in the LICENSE file. You may download and modify RIDA, and use it in free or paid applications. You may not sell or redistribute RIDA as a standalone package or library. Applications that include RIDA must show the attribution notice specified in the license.