RIDA Python library — 0.1.1

RIDA means Range Indexed Data Access. It is an installable Python library and terminal tool that builds one .rida file from a collection of JSON records. The file contains:

• a fixed RIDA header;
• small, readable UTF-8 metadata at the start of the file;
• a compressed dense or sparse address catalog; and
• UTF-8 JSON values stored in a data region.

The reader loads the catalog once and then reads only the requested field bytes. It works locally and over an HTTP server that supports byte-range requests.

Install locally

From this project folder:

python3 -m pip install .

After that, the rida command is available in the same Python environment. The library uses only Python's standard library at runtime: no database, cloud account, or server process is required.

Install the published package with:

python3 -m pip install rida-format

The PyPI package name is rida-format. After installation, the terminal command and Python import name are both rida.

Library project files

Keep the source project in one folder, for example rida/:

rida/
├── pyproject.toml          package name and install command
├── MANIFEST.in             files included in the public source archive
├── RIDA-README.md           usage instructions
├── RIDA-DESIGN-0.1.md       RIDA format design
├── LICENSE                  RIDA license
├── docs/
│   └── rida-metadata.html   optional local metadata helper
└── src/
    └── rida/
        ├── __init__.py     RIDA builder and reader library
        └── __main__.py     enables: python -m rida

src/rida/ is internal library code. After installation, Python copies it into its own site-packages/rida/ location. A developer never needs to run an individual .py file: they use import rida, rida ..., or python -m rida.

Requirements and downloads

• Python 3.10 or newer and pip are required.
• RIDA has zero third-party runtime dependencies. It uses only Python's built-in json, lzma, hashlib, urllib, and related standard modules.
• The rida command has its own progress display for loading, scanning, value writing, catalog compression, and final file writing. It does not require tqdm.
• It does not install or require AWS SDK, Cloudflare SDK, SQLite, Node.js, Rust, a database server, or a Worker.
• When installing this source project, pip may temporarily download setuptools (version 77.0.3 or newer) to build the package. It is an install tool, not a RIDA runtime dependency.
• Official Python installers for macOS and Windows include the required lzma support. On an unusually minimal Linux Python build, install the operating system's python3-lzma package before using RIDA.

Build

Scan first. It reports the field density and recommends dense or sparse mode:

rida scan users.json

Build a RIDA file. The command asks for confirmation unless --yes is supplied:

rida build users.json users.rida \
  --title "Users dataset" \
  --description "Public sample data" \
  --reader-url "https://example.com/rida-explorer" \
  --yes

During a build, the terminal shows RIDA's own progress, for example:

Reading JSON bytes: 217,221,288/217,221,288 (100%)
Parsing JSON: 1/1 (100%)
Scanning field names: 39,098/39,098 (100%)
Measuring values: 39,098/39,098 (100%)
Writing value data: 39,098/39,098 (100%)
Compressing catalog: 1/1 (100%)
Writing RIDA file: 217,221,288/217,221,288 (100%)

For large inputs, low-memory progress reports the record count as it reads. When the total is not known in advance, updates appear every 1,000 records.

Input can be:

• an object that maps IDs to JSON record objects;
• an array of JSON record objects, with --id-key; or
• a folder containing individual JSON record files. Filenames become IDs.

rida build records/ records.rida --yes
rida build users-array.json users.rida --id-key id --yes

If the input folder has JSON files in more than one place, RIDA lists the groups and asks which folder contains the records. For example, if a folder has book_details.json at its top level and thousands of record files inside hadiths/, choose the hadiths/ group. For a command that runs without a prompt, select it explicitly:

rida scan "/path/to/musnad-ahmed" --records-dir hadiths
rida build "/path/to/musnad-ahmed" musnad-ahmed.rida \
  --records-dir hadiths --yes

--records-dir is a folder inside the input folder. RIDA searches that folder and its subfolders for .json files; it does not mix separate groups unless they are all inside the selected folder. The scan prints the number of records before a build starts.

Use --mode dense or --mode sparse to override the recommendation. The builder does not overwrite an existing output file.

Memory modes for large inputs

The default --memory-mode auto estimates the input's working-memory needs and checks currently available RAM. Choose a mode explicitly when needed:

rida build records.json records.rida --id-key id --memory-mode low-memory --yes
rida build records/ records.rida --memory-mode fast --yes

low-memory reads one JSON record at a time instead of keeping the whole JSON input in RAM. For one JSON file, supported layouts are an object mapping IDs to record objects, or an array of record objects with --id-key. For a folder, RIDA reads one record file at a time. The current record and the address catalog still use RAM, so this is not a zero-memory mode; an individual record must fit in memory.

If the input folder contains separate JSON groups, RIDA asks which group to use. For non-interactive commands, pass --records-dir, for example:

rida build "/path/to/musnad-ahmed" musnad-ahmed.rida \
  --records-dir hadiths --memory-mode low-memory --yes

Read locally

rida inspect users.rida
rida get users.rida ali city

Where RIDA can read data

RIDA 0.1.1 can read a .rida file from the local computer or from a direct HTTP/HTTPS file URL. For remote reads, RIDA sends ordinary HTTP byte-range requests; it does not use a cloud-provider SDK. This makes it compatible with object storage that returns the requested bytes as HTTP 206 Partial Content and includes the matching Content-Range header.

Examples expected to work because their object APIs support byte-range reads include Cloudflare R2, Amazon S3, Google Cloud Storage, Azure Blob Storage, Backblaze B2, and Bunny.net through a Pull Zone. See their docs: R2, S3, Google Cloud Storage, Azure Blob Storage, Backblaze B2, and Bunny.net. This is protocol-based expected compatibility, not a separate integration with each provider. The exact URL, permissions, and any CDN in front of a bucket still need to allow byte-range responses; the providers have not all been individually tested with this RIDA release.

For Bunny Storage, connect the Storage Zone to a Pull Zone and use the Pull Zone URL (or its custom hostname) for the file. Bunny documents range support for cached content by default; enable Cache Slicing in the Pull Zone for uncached content.

The current reader sends a Range header but does not add provider-specific authentication headers. Use a public object URL or a signed URL that permits the range request. RIDA does not upload files to cloud storage: upload the .rida file using that storage provider, then give RIDA the file URL.

Read from R2 or another HTTP URL

After uploading the .rida file to an HTTP byte-range server:

rida remote-get https://data.example.com/users.rida ali city

The remote reader first fetches the 128-byte header and catalog, then fetches only the requested field range. It checks that the server returns HTTP 206 and the correct Content-Range; if the server sends the whole file instead, RIDA stops with an error rather than silently downloading it all.

One RemoteRidaFile object keeps that catalog in RAM, so later .get() calls on the same object fetch only the requested value range. The rida remote-get terminal command creates a new object each time, so persistent on-disk catalog cache is a later RIDA Reader feature.

Use from Python

import rida

# Scan first if you want to show the publisher the recommendation.
report = rida.scan("users.json")
print(report.recommended_mode)  # "dense" or "sparse"

# Build once. The source JSON stays unchanged.
rida.build(
    "users.json",
    "users.rida",
    title="Users dataset",
    reader_url="https://example.com/rida",
)

# Read only one stored value from a local RIDA file.
with rida.RidaFile("users.rida") as data:
    print(data.get("ali", "city"))

# The same primary lookup works against an R2/public URL with Range support.
remote = rida.RemoteRidaFile("https://data.example.com/users.rida")
print(remote.get("fatima", "languages"))

A desktop app or another Python program can show the same stages in its own interface by passing a callback:

def progress(stage, completed, total):
    print(f"{stage}: {completed}/{total}")

rida.build("users.json", "users.rida", progress=progress)

Current scope

RIDA 0.1.1 implements primary lookup: record ID + field name → value. Optional exact-match, numeric-range, and Unicode full-text search indexes are not part of this release.

License

RIDA is distributed under the RIDA Proprietary License in the LICENSE file. You may download and modify RIDA, and use it in free or paid applications. You may not sell or redistribute RIDA as a standalone package or library. Applications that include RIDA must show the attribution notice specified in the license.