JSON Tools RS

A high-performance Rust library for advanced JSON manipulation with SIMD-accelerated parsing, providing unified flattening and unflattening operations through a clean builder pattern API. Ships with Rust and Python bindings.

Why JSON Tools RS?

JSON Tools RS is designed for developers who need to:

Transform nested JSON into flat structures for databases, CSV exports, or analytics
Clean and normalize JSON data from external APIs or user input
Process large batches of JSON documents efficiently
Maintain type safety with perfect roundtrip support (flatten → unflatten → original)
Work with both Rust and Python using the same consistent API

Unlike simple JSON parsers, JSON Tools RS provides a complete toolkit for JSON transformation with production-ready performance and error handling.

Features

🚀 Unified API: Single JSONTools entry point for flattening, unflattening, or pass-through transforms (.normal())
🔧 Builder Pattern: Fluent, chainable API for easy configuration and method chaining
⚡ High Performance: SIMD-accelerated JSON parsing with FxHashMap, SmallVec stack allocation, and tiered caching
🚄 Parallel Processing: Built-in Rayon-based parallelism (persistent work-stealing pool) for faster batch operations and large nested structures
🎯 Complete Roundtrip: Flatten JSON and unflatten back to original structure with perfect fidelity
🧹 Comprehensive Filtering: Remove empty strings, nulls, empty objects, and empty arrays (works for both flatten and unflatten)
🔄 Advanced Replacements: Key/value replacements, literal (exact substring match) by default, or regex by wrapping the pattern in r'...'
🚫 Key/Value Exclusion: Drop entire keys (and their subtree) or key-value pairs by pattern match with .exclude_key()/.exclude_value()
🛡️ Collision Handling: Intelligent .handle_key_collision(true) to collect colliding values into arrays
📅 Date Normalization: Automatic detection and normalization of ISO-8601 dates to UTC
🔀 Automatic Type Conversion: Convert strings to numbers, booleans, and nulls with .auto_convert_types(true)
📦 Batch Processing: Process single JSON or batches; Python also supports dicts and lists of dicts
🐍 Python Bindings: Full Python support with perfect type preservation (input type = output type)
📊 DataFrame/Series Support: Native support for Pandas, Polars, PyArrow, and PySpark DataFrames and Series in Python

Why JSON Tools RS?
Features
Quick Start
Quick Reference
Installation
Architecture
Performance
Contributing
License
Changelog

Quick Start

Rust - Unified JSONTools API

The JSONTools struct provides a unified builder pattern API for all JSON manipulation operations. Simply call .flatten() or .unflatten() to set the operation mode, then chain configuration methods and call .execute().

Basic Flattening

use json_tools_rs::{JSONTools, JsonOutput};

let json = r#"{"user": {"name": "John", "profile": {"age": 30, "city": "NYC"}}}"#;
let result = JSONTools::new()
    .flatten()
    .execute(json)?;

if let JsonOutput::Single(flattened) = result {
    println!("{}", flattened);
}
// Output: {"user.name": "John", "user.profile.age": 30, "user.profile.city": "NYC"}

Advanced Flattening with Filtering

use json_tools_rs::{JSONTools, JsonOutput};

let json = r#"{"user": {"name": "John", "details": {"age": null, "city": ""}}}"#;
let result = JSONTools::new()
    .flatten()
    .separator("::")
    .lowercase_keys(true)
    .key_replacement("r'(User|Admin)_'", "")
    .value_replacement("@example.com", "@company.org")
    .remove_empty_strings(true)
    .remove_nulls(true)
    .remove_empty_objects(true)
    .remove_empty_arrays(true)
    .execute(json)?;

if let JsonOutput::Single(flattened) = result {
    println!("{}", flattened);
}
// Output: {"user::name": "John"}

Automatic Type Conversion

Convert string values to numbers, booleans, dates, and null automatically for data cleaning and normalization.

use json_tools_rs::{JSONTools, JsonOutput};

let json = r#"{
    "id": "123",
    "price": "$1,234.56",
    "discount": "15%",
    "active": "yes",
    "verified": "1",
    "created": "2024-01-15T10:30:00+05:00",
    "status": "N/A"
}"#;

let result = JSONTools::new()
    .flatten()
    .auto_convert_types(true)
    .execute(json)?;

if let JsonOutput::Single(flattened) = result {
    println!("{}", flattened);
}
// Output: {
//   "id": 123,
//   "price": 1234.56,
//   "discount": 15.0,
//   "active": true,
//   "verified": 1,
//   "created": "2024-01-15T05:30:00Z", // Normalized to UTC
//   "status": null
// }

Python - Unified JSONTools API

The Python bindings provide the same unified JSONTools API with perfect type matching: input type equals output type.

Basic Usage

import json_tools_rs as jt

# Basic flattening - dict input → dict output
result = jt.JSONTools().flatten().execute({"user": {"name": "John", "age": 30}})
print(result)  # {'user.name': 'John', 'user.age': 30}

# Basic unflattening - dict input → dict output
result = jt.JSONTools().unflatten().execute({"user.name": "John", "user.age": 30})
print(result)  # {'user': {'name': 'John', 'age': 30}}

Advanced Configuration & Parallelism

import json_tools_rs as jt

# Configure tools with parallel processing settings
tools = (jt.JSONTools()
    .flatten()
    .separator("::")
    .lowercase_keys(True)
    .remove_empty_strings(True)
    .parallel_threshold(50)       # Parallelize batches >= 50 items
    .num_threads(4)               # Use 4 threads
    .nested_parallel_threshold(200) # Parallelize large objects
)

# Process a batch of data
batch = [{"data": i} for i in range(100)]
results = tools.execute(batch)

DataFrame & Series Support

import json_tools_rs as jt
import pandas as pd

# Pandas DataFrame input → Pandas DataFrame output
df = pd.DataFrame([
    {"user": {"name": "Alice", "age": 30}},
    {"user": {"name": "Bob", "age": 25}},
])
result = jt.JSONTools().flatten().execute(df)
print(type(result))  # <class 'pandas.core.frame.DataFrame'>

# Also works with Polars, PyArrow Tables, and PySpark DataFrames
# Series input → Series output (Pandas, Polars, PyArrow)

# Or skip having a DataFrame at all -- normalise=True always returns a wide
# DataFrame regardless of input shape (dict, str, list), with target= picking
# the library (pandas/polars/pyarrow/pyspark) or auto-resolving if omitted.
df = jt.JSONTools().flatten().execute(
    {"user": {"name": "Alice", "age": 30}}, normalise=True
)

Runnable Examples

Every builder feature has a standalone, runnable example in both languages, plus curated multi-feature pipelines (not an exhaustive combinatorial sweep -- the builder has ~10 independent toggles -- but realistic groupings commonly used together, and one "kitchen sink" pipeline exercising nearly everything at once). Both language versions use matching inputs and produce matching output.

	Individual features	Curated combinations
Rust	`examples/feature_by_feature.rs`	`examples/feature_combinations.rs`
Python	`python/examples/feature_by_feature.py`	`python/examples/feature_combinations.py`

# Rust
cargo run --example feature_by_feature
cargo run --example feature_combinations

# Python
python3 python/examples/feature_by_feature.py
python3 python/examples/feature_combinations.py

There are also narrative walkthroughs for a quicker first read: examples/basic_usage.rs / examples/advance_usage.rs (Rust) and python/examples/examples.py (Python).

Quick Reference

Method Cheat Sheet

Method	Description	Example
`.flatten()`	Set operation mode to flatten	`JSONTools::new().flatten()`
`.unflatten()`	Set operation mode to unflatten	`JSONTools::new().unflatten()`
`.normal()`	Set mode to pass-through (transform only)	`JSONTools::new().normal()`
`.separator(sep)`	Set key separator (default: `"."`)	`.separator("::")`
`.lowercase_keys(bool)`	Convert keys to lowercase	`.lowercase_keys(true)`
`.remove_empty_strings(bool)`	Remove empty string values	`.remove_empty_strings(true)`
`.remove_nulls(bool)`	Remove null values	`.remove_nulls(true)`
`.remove_empty_objects(bool)`	Remove empty objects `{}`	`.remove_empty_objects(true)`
`.remove_empty_arrays(bool)`	Remove empty arrays `[]`	`.remove_empty_arrays(true)`
`.key_replacement(find, repl)`	Replace key patterns (literal, or regex via `r'...'`)	`.key_replacement("r'user_'", "")`
`.value_replacement(find, repl)`	Replace value patterns (literal, or regex via `r'...'`)	`.value_replacement("@old.com", "@new.com")`
`.exclude_key(pattern)`	Drop a key (and its entire subtree) matching a pattern	`.exclude_key("crypto")`
`.exclude_value(pattern)`	Drop a key-value pair whose value matches a pattern	`.exclude_value("banned")`
`.handle_key_collision(bool)`	Collect colliding keys into arrays	`.handle_key_collision(true)`
`.always_array_keys([...])`	Always render these flattened keys as arrays, even with one value -- consistent shape across documents	`.always_array_keys(["name"])`
`.auto_convert_types(bool)`	Convert types (nums, bools, dates, nulls) -- all 4 categories, default behavior	`.auto_convert_types(true)`
`.convert_dates/nulls/booleans/numbers(bool)`	Convert types independently per category, with optional `_config(...)` customization	`.convert_numbers(true)`
`.parallel_threshold(n)`	Min batch size for parallelism	`.parallel_threshold(500)`
`.num_threads(n)`	Number of threads (default: CPU count)	`.num_threads(Some(4))`
`.nested_parallel_threshold(n)`	Nested object parallelism size	`.nested_parallel_threshold(50)`
`.max_array_index(n)`	Max array index for unflatten (DoS protection)	`.max_array_index(100_000)`

Automatic Type Conversion

When .auto_convert_types(true) is enabled, the library performs smart parsing on string values. For independent control over each category below (e.g. only converting numbers, or customizing date/null/boolean matching), use .convert_dates()/.convert_nulls()/.convert_booleans()/.convert_numbers() instead -- see Automatic Type Conversion for the full per-category reference across all three language bindings.

Date & Time (ISO-8601):

Detects date strings to avoid converting them to numbers (e.g., "2024-01-01").
Normalizes datetimes to UTC.
Supports offsets (+05:00), Z suffix, and naive datetimes.

Numbers:

Basic: "123" → 123, "45.67" → 45.67
Separators: "1,234.56" (US), "1.234,56" (EU), "1 234.56" (Space)
Currency: "$123", "€99", "£50", "¥1000", "R$50"
Scientific: "1e5" → 100000
Percentages: "50%" → 50.0, "12.5%" → 12.5
Basis Points: "50bps" → 0.005, "100 bp" → 0.01
Suffixes: "1K", "2.5M", "5B" (Thousand, Million, Billion)

Booleans:

"true", "false", "yes", "no", "on", "off", "y", "n" (case-insensitive).
Note: "1" and "0" are treated as numbers, not booleans.

Nulls:

"null", "nil", "none", "N/A" (case-insensitive) → null.

Installation

Rust

cargo add json-tools-rs

Python

pip install json-tools-rs

Architecture

The codebase is organized into focused, single-responsibility modules:

src/
├── lib.rs            Facade: mod declarations + pub use re-exports
├── json_parser.rs    Conditional SIMD parser (sonic-rs on 64-bit, simd-json on 32-bit)
├── types.rs          Core types: JsonInput, JsonOutput
├── error.rs          Error types with codes E001-E008
├── config.rs         Configuration structs and operation modes
├── cache.rs          Tiered regex pattern caching (compile-time table, thread-local, global)
├── convert.rs        Type conversion: numbers, dates, booleans, nulls (SIMD-optimized)
├── transform.rs      Filtering, key/value replacements, collision handling
├── flatten.rs        Flattening algorithm with Rayon parallelism
├── unflatten.rs      Unflattening with SIMD separator detection
├── builder.rs        Public JSONTools builder API and execute() entry point
├── python.rs         Python bindings via PyO3
├── tests.rs          Unit tests
└── main.rs           CLI examples

The processing pipeline:

Parse -- SIMD-accelerated JSON parsing (json_parser)
Flatten/Unflatten -- Recursive traversal with CompactString/arena-backed key storage (flatten/unflatten)
Transform -- Lowercase, replacements (cached regex), collision handling (transform)
Filter -- Remove empty strings, nulls, empty objects/arrays (transform)
Convert -- Type conversion with first-byte discriminators (convert)
Serialize -- Output to JSON string or native Python types

Performance

Benchmark Results

Benchmark	Time	Description
Deep nesting (100 levels)	~2.17 µs	Deeply nested JSON objects
Wide objects (1,000 keys)	~24.8 µs	Flat objects with many keys
Large arrays (5,000 items)	~406 µs	Arrays with many elements
Parallel batch (10,000 items)	~635 µs	Batch processing with Rayon (`nested_parallel_threshold`)

Measured on Apple Silicon (M4) via cargo bench --bench stress_benchmarks, v0.9.5. Results may vary by platform and data shape.

Optimization Techniques

JSON Tools RS uses several techniques to achieve high performance:

SIMD-JSON: Hardware-accelerated parsing via sonic-rs (64-bit) / simd-json (32-bit).
SIMD Byte Search: memchr/memmem for SIMD-accelerated string operations and pattern matching.
FxHashMap: Faster hashing for string keys via a hand-rolled FxHash-style hasher (src/fxhash.rs; no external hashing crate dependency).
Tiered Caching: Three-level regex cache (compile-time pattern table → thread-local FxHashMap → global RwLock<FxHashMap>).
SmallVec & Cow: Stack allocation for depth stacks and number buffers; zero-copy string handling.
CompactString & Arena Keys: Object keys are inlined via CompactString (no heap allocation up to 24 bytes); flatten's slow path additionally uses a bumpalo arena for deep-nested keys, to minimize allocations in wide/deep JSON.
First-Byte Discriminators: Rapid rejection of non-convertible strings during type conversion.
Parallelism: Rayon's persistent work-stealing thread pool for batch processing and large nested structures (avoids per-call OS thread spawn cost).

CLI Demo

The crate includes an educational demo binary that showcases library features:

cargo run

This prints progressive examples covering basic flattening, unflattening, custom separators, filtering, replacements, collision handling, type conversion, and batch processing.

Contributing

See CONTRIBUTING.md for development setup, testing, benchmarking, and PR guidelines.

License

Dual-licensed under either MIT or Apache-2.0, at your option.