ReliaDL: Adaptive Fault-Tolerant Chunked Transfer with Homomorphic Verification and Stochastic Scheduling

Research Paper Reference

Paper Title: ReliaDL: Adaptive Fault-Tolerant Chunked Transfer with Homomorphic Verification and Stochastic Scheduling
Authors: Purvansh Joshi, Archit Mittal (PxA Labs), Aviral Mittal Category: Networked Systems, Transport Protocols, Distributed Systems Reliability
Primary Specification: docs/NOVEL_ALGORITHMS.md
Empirical Benchmarks & Methodology: docs/PERFORMANCE.md

@article{reliadl2026,
  title={{ReliaDL: Adaptive Fault-Tolerant Chunked Transfer with Homomorphic Verification and Stochastic Scheduling}},
  author={Joshi, Purvansh and Mittal, Archit},
  journal={arXiv preprint arXiv:2608.xxxxx},
  year={2026},
  publisher={PxA Labs},
  url={https://github.com/PxA-Labs/ReliaDL}
}

Overview

ReliaDL is a research-oriented, fault-tolerant file transfer framework that formulates the reliable download problem as a constrained stochastic optimization over non-stationary channels. Unlike conventional download managers that employ static chunking and reactive retransmission (ARQ), ReliaDL introduces five novel algorithmic contributions spanning adaptive coding theory, algebraic verification, and optimal resource allocation.

The central research question addressed by this work is:

How can we minimize the expected data retransmission cost while guaranteeing byte-level integrity under non-stationary, bursty channel conditions, subject to throughput and resource constraints?

ReliaDL provides both a theoretical framework with provable performance bounds and a practical Python-based reference implementation suitable for empirical evaluation.

Novel Algorithmic Contributions

1. AdaChunk -- Lyapunov-Based Adaptive Chunk Sizing. Formulates chunk size selection as an online stochastic optimization problem. Using the Lyapunov drift-plus-penalty framework (Neely, 2010), AdaChunk dynamically adjusts chunk boundaries based on real-time network state observations, achieving an O(1/V) optimality gap with O(V) queue backlog tradeoff.

2. LtHash -- Homomorphic Hash Aggregation. Employs lattice-based homomorphic hashing to enable O(1) whole-file integrity verification post-assembly without re-reading the file from disk. The homomorphic property H(x || y) = H(x) + H(y) mod p allows algebraic accumulation of per-chunk hashes during download.

3. Sub-Chunk Merkle Localization. Embeds hierarchical Merkle trees within each chunk at 4 KB segment granularity. Upon detecting a chunk-level hash mismatch, the system localizes corruption to specific 4 KB segments in O(log(B/s)) comparisons, reducing retransmission by up to 99.95%.

4. Predictive Parity Injection. Implements proactive XOR-based forward error correction (FEC) with injection rate governed by a Gilbert-Elliott two-state Markov channel model estimator. Enables zero-RTT chunk recovery without network round trips under moderate loss conditions.

5. Whittle Index Worker Scheduling. Models multi-source worker allocation as a restless multi-armed bandit (RMAB) problem. Employs Whittle index policies (Whittle, 1988) for asymptotically optimal stochastic scheduling of download workers across heterogeneous sources.

Key Mathematical Formulations

AdaChunk Optimization Objective

min  lim_{T->inf} (1/T) * sum_{t=0}^{T-1} E[C(B_t, s_t)]
s.t. lim_{T->inf} (1/T) * sum_{t=0}^{T-1} E[G(B_t, s_t)] >= G_min

Where:
  B_t           = chunk size at time t (decision variable)
  s_t           = (RTT_t, sigma_RTT, p_t, G_t, BDP_t)  (network state)
  C(B, s)       = B * (1 - (1-p)^{B/MSS})              (retransmission cost)
  G(B, s)       = B*(1-p_retry) / (T_dl + T_overhead)   (goodput)
  Q_{t+1}       = max(Q_t - G(B_t, s_t) + G_min, 0)    (virtual queue)
  B_t*          = argmin V*C(B,s_t) - Q_t*G(B,s_t)      (per-slot decision)

LtHash Homomorphic Aggregation

H_file = sum_{i=0}^{k-1} H_LtHash(chunk_i) mod p

Where H(x) = A * x mod p, A in Z_p^{n x m}, satisfying:
  H(x || y) = H(x) + H(y) mod p  (homomorphic property)
  Collision resistance reduces to SIS hardness

Whittle Index Policy

W_i(s) = inf{ w : V_active(s, w) = V_passive(s, w) }

At each slot, activate K arms with highest W_i(s_i(t))
Achieves asymptotic optimality as N -> infinity (Weber & Weiss, 1990)

System Architecture & Workflow Diagrams

Layered System Architecture

graph TB
    subgraph ClientLayer["1. Client & Configuration Layer"]
        CLI["CLI Interface (main.py)"]
        CONF["Configuration Engine (config.py)"]
        SM["State Manager (state_manager.py - ACID Persistence)"]
    end

    subgraph ControlLayer["2. Stochastic Control & Optimization Layer"]
        AC["AdaChunk Optimizer<br/>(Lyapunov Drift-Plus-Penalty)"]
        WS["Whittle Scheduler<br/>(Restless Bandit Allocation)"]
        PE["Predictive Parity Encoder<br/>(Gilbert-Elliott Markov FEC)"]
    end

    subgraph TransportLayer["3. Concurrent Transport Engine"]
        DE["Download Engine (download_engine.py)"]
        W1["Worker 1 (HTTP/2 Range GET)"]
        W2["Worker 2 (HTTP/2 Range GET)"]
        WN["Worker K (HTTP/2 Range GET)"]
    end

    subgraph VerificationLayer["4. Verification & Localization Layer"]
        DH["Dual Hasher<br/>(Streaming SHA-256 + LtHash)"]
        ML["Sub-Chunk Merkle Localizer<br/>(4 KB Segment Binary Search)"]
        HH["Homomorphic Aggregator<br/>(O(1) Whole-File Integrity Gate)"]
    end

    subgraph StorageLayer["5. Direct Zero-Copy Storage Engine"]
        FA["Positional Disk Writer (os.pwrite)"]
        OUT["Target Payload Artifact"]
    end

    CLI --> CONF
    CLI --> DE
    DE <--> SM
    DE --> AC
    DE --> WS
    AC -. "Optimal Chunk Size B_t*" .-> DE
    WS -. "Top-K Arm Assignments" .-> DE
    DE --> PE
    PE -. "Parity Allocation" .-> DE
    DE --> W1 & W2 & WN
    W1 & W2 & WN --> DH
    DH --> ML
    ML --> HH
    DH --> FA
    FA --> OUT
    HH -. "O(1) Verification Pass" .-> SM

End-to-End Download & Recovery Sequence

sequenceDiagram
    autonumber
    participant U as User / CLI
    participant DE as Download Engine
    participant AC as AdaChunk Optimizer
    participant WS as Whittle Scheduler
    participant W as Worker Pool
    participant PE as Parity Engine
    participant ML as Merkle Localizer
    participant HH as LtHash Aggregator

    U->>DE: Initiate Download(URL, Target)
    DE->>AC: Sample Channel State (RTT, Loss, BDP)
    AC-->>DE: Return Optimal Chunk Size B_t*
    DE->>WS: Query Active Arm Rankings
    WS-->>DE: Allocate Top-K Sources
    DE->>PE: Compute Parity Redundancy r*(t)
    PE-->>DE: Attach Proactive FEC Units
    DE->>W: Dispatch Range GET [Start, End]
    W-->>DE: Ingest Chunks + Stream Hashes
    DE->>HH: Accumulate Homomorphic Hash H(chunk)
    alt Chunk Hash Valid
        DE->>DE: Commit to Disk via os.pwrite()
    else Chunk Hash Corrupt
        DE->>ML: Traverse Hierarchical Merkle Tree
        ML-->>DE: Pinpoint Corrupted 4 KB Sub-Blocks
        alt Parity Recoverable
            DE->>PE: Execute Zero-RTT XOR Reconstruction
            PE-->>DE: Reconstructed Valid Sub-Blocks
        else Parity Exhausted
            DE->>W: Issue Targeted 4 KB Range GET
            W-->>DE: Recovered Byte Segment
        end
    end
    DE->>HH: Evaluate Sum(H_i) == ExpectedRoot mod p
    HH-->>U: Instantaneous O(1) Verification Complete

Chunk Lifecycle State Machine

stateDiagram-v2
    [*] --> PENDING: Range Partitioned
    PENDING --> SCHEDULED: Whittle Index Priority
    SCHEDULED --> DOWNLOADING: Worker Assigned
    DOWNLOADING --> STREAM_VERIFYING: Buffer Ingestion Complete
    
    STREAM_VERIFYING --> HOMOMORPHIC_ACCUMULATED: SHA-256 & LtHash Pass
    HOMOMORPHIC_ACCUMULATED --> COMMITTED: Written via os.pwrite()
    COMMITTED --> [*]
    
    STREAM_VERIFYING --> MERKLE_LOCALIZING: Checksum Mismatch Detected
    MERKLE_LOCALIZING --> PARITY_RECOVERY: 4 KB Corrupted Segments Identified
    
    PARITY_RECOVERY --> HOMOMORPHIC_ACCUMULATED: Zero-RTT XOR Recovery Pass
    PARITY_RECOVERY --> TARGETED_RETRY: FEC Overhead Insufficient
    TARGETED_RETRY --> DOWNLOADING: Re-request 4 KB Sub-Range

Repository Structure

ReliaDL/
├── README.md                          # Project documentation entry point
├── LICENSE                            # Apache 2.0 License
│
├── docs/
│   ├── NOVEL_ALGORITHMS.md            # Formal mathematical formulations & proofs
│   ├── PROJECT_OVERVIEW.md            # High-level system overview
│   ├── ARCHITECTURE.md                # Component design & architectural decisions
│   ├── TECHNICAL_SPECIFICATION.md     # Detailed protocols, data schemas, & algorithms
│   ├── API_REFERENCE.md               # Complete Python & CLI API specification
│   ├── DATA_FLOW.md                   # State machine diagrams & data path sequences
│   ├── ERROR_HANDLING.md              # Error taxonomy & recovery strategies
│   ├── SECURITY.md                    # Security analysis & threat model
│   ├── MANIFEST_SPECIFICATION.md      # Chunk manifest (.cgmanifest) & Merkle tree spec
│   ├── CLOUD_ADAPTERS.md              # AWS S3, GCS, Azure Blob, & proxy protocol adapters
│   ├── OBSERVABILITY.md               # Prometheus metrics, OpenTelemetry, & logging
│   ├── DEPLOYMENT_GUIDE.md            # Installation, configuration, & operations
│   ├── USER_GUIDE.md                  # Comprehensive end-user guide
│   ├── TESTING_STRATEGY.md            # Test plans, benchmarks, & coverage targets
│   ├── PERFORMANCE.md                 # Benchmarks, memory profile, & tuning
│   ├── CONTRIBUTING.md                # Developer contribution standards
│   ├── CHANGELOG.md                   # Version history
│   ├── GLOSSARY.md                    # Terminology index
│   ├── FAQ.md                         # Frequently asked questions
│   └── agents.md                      # AI agents & Mem0 memory configuration
│
├── src/                               # System implementation
│   ├── chunk_manager.py               # Chunk partitioning & boundary logic
│   ├── download_engine.py             # Asynchronous download orchestrator
│   ├── hash_verifier.py               # Streaming SHA-256 verification
│   ├── state_manager.py               # Atomic state file manager
│   ├── retry_handler.py               # Exponential backoff & retry policies
│   ├── file_assembler.py              # Chunk reassembly & final integrity check
│   ├── config.py                      # System configuration parser
│   ├── models.py                      # Data models & schemas
│   ├── exceptions.py                  # Custom exception definitions
│   ├── logger.py                      # Structured JSON logging engine
│   └── main.py                        # Command Line Interface (CLI) entry point
│
├── tests/                             # Test suite
│   ├── unit/                          # Unit test modules
│   ├── integration/                   # Integration test modules
│   └── fixtures/                      # Mock data & server fixtures
│
└── config/
    └── default_config.yaml            # Default system configuration template

Quick Start

Installation

git clone https://github.com/PxA-Labs/ReliaDL.git
cd ReliaDL
pip install -r requirements.txt

Usage

# Execute a download with adaptive chunking and homomorphic verification
python -m src.main download \
  --url "https://example.com/dataset.tar.gz" \
  --output "./downloads/dataset.tar.gz" \
  --adachunk \
  --whittle

# Resume an interrupted transfer
python -m src.main resume \
  --state-file "./downloads/.reliadl/dataset.tar.gz.state"

# Verify file integrity
python -m src.main verify \
  --file "./downloads/dataset.tar.gz" \
  --expected-hash "sha256:abcdef1234567890..."

Documentation Index

Security & OpenSSF Compliance

ReliaDL adheres to the Open Source Security Foundation (OpenSSF) Best Practices and Scorecard standards:

• Cryptographic Verification: Dual-tier SHA-256 and binary Merkle Tree validation against tampered payloads.
• Supply Chain Security: Pinned GitHub Actions dependencies, strict branch protection rules, and signed manifest catalogs.
• Vulnerability Disclosure: Coordinated security response process documented in SECURITY.md.
• Automated CI Gates: Automated novelty scanning, type checking, and unit test enforcement on all pull requests.

License

This project is licensed under the Apache License 2.0. See the LICENSE file for the complete terms.