transmog 2.0.4

A data transformation library for flattening complex nested structures into tabular formats while preserving hierarchical relationships

Transmog - Flatten Nested JSON to Tabular Formats

---

A configurable data flattening tool that transforms nested JSON data into flat, tabular formats while preserving parent-child relationships.

---

Installation

# Full install (CSV, Parquet, ORC, Avro output)
pip install transmog

# CSV only (no pyarrow, fastavro, or cramjam)
pip install transmog[minimal]

Quick Start

import transmog as tm

data = {"user": "Alice", "orders": [{"id": 101}, {"id": 102}]}
result = tm.flatten(data, name="users")

result.main                    # Main table
result.tables["users_orders"]  # Child tables
result.save("output.csv")      # Save to file

In-Memory vs Streaming

1. flatten(data, name, config) — Flatten data in memory

   result = tm.flatten("data.json", name="products")
   result = tm.flatten([{"id": 1}, {"id": 2}])
   result.save("output.parquet")
   

2. flatten_stream(data, output_path, name, output_format) — Stream directly to disk

   tm.flatten_stream("large.jsonl", "output/", name="events", output_format="parquet")
   

Configuration

config = tm.TransmogConfig(
    # Array handling
    array_mode=tm.ArrayMode.SMART,   # SMART (default), SEPARATE, INLINE, SKIP

    # ID generation and metadata fields
    id_generation="random",          # random (default), natural, hash, or ["field1", "field2"]
    id_field="_id",                  # Field name for record IDs
    parent_field="_parent_id",       # Field name for parent references
    time_field="_timestamp",         # Field name for timestamps (None to disable)

    # Data transformation
    include_nulls=False,             # Include null/empty values in output
    stringify_values=False,          # Convert all leaf values to strings

    # Processing controls
    max_depth=100,                   # Maximum recursion depth
    batch_size=1000                  # Records per batch for streaming
)

result = tm.flatten(data, config=config)

Array Modes

Mode	Behavior
SMART	Preserve simple arrays, extract complex arrays to child tables
SEPARATE	Extract all arrays to child tables
INLINE	Serialize arrays as JSON strings
SKIP	Omit arrays from output

ID Generation

Strategy	Description
random	Generate random UUID (default)
natural	Use existing ID field from data
hash	Deterministic hash of entire record
["field1", ...]	Deterministic hash of specified fields

Documentation

Full documentation: scottdraper8.github.io/transmog

• Getting Started
• Configuration
• Array Handling
• ID Management
• Error Handling
• Output Formats
• Streaming
• API Reference
• Contributing

License

MIT License - see LICENSE file for details.