iceberg-meta

CLI and TUI for exploring Apache Iceberg table metadata. Inspect snapshots, schemas, manifests, data files, partition health, and column-level statistics -- without Spark or notebooks.

Why use this?

While the official pyiceberg CLI is great for basic operations, iceberg-meta is built specifically for data engineers who need to debug, optimize, and understand their tables.

• ⚡ Instant Feedback: No Spark cluster startup time.
• 🔍 Deep Visibility: Inspect manifest lists, manifest files, and individual data files.
• 🏥 Health Checks: Detect small files, partition skew, and compaction needs in seconds.
• 📉 Diff Everything: Compare two snapshots to see exactly what changed (files added/deleted, rows changed).
• 🖥️ Interactive TUI: Browse your catalog with a powerful terminal UI.
• 🤖 CI/CD Ready: Output JSON/CSV for automated checks in your pipelines.

Performance & Benchmarks

We take performance seriously. Data engineers shouldn't wait minutes for simple metadata checks.

Benchmark Scenario:

• Table: 5,000 data files, 50,000 rows (simulating a "small file problem" scenario).
• Environment: Local development machine (SQLite catalog).
• Task: Full metadata scan, partition skew calculation, and file listing.

Note: Spark times include JVM startup overhead, which is the main friction point iceberg-meta eliminates for quick checks.

Install

From PyPI

pip install iceberg-meta

From Homebrew (macOS/Linux)

You can install directly from the repository:

brew tap MandlaSibanda196/iceberg-meta https://github.com/MandlaSibanda196/iceberg-meta
brew install iceberg-meta

Try it instantly

No config, no Docker, no credentials:

iceberg-meta demo

Creates a temporary catalog with sample tables, launches the TUI, and cleans up on exit.

Quick Start

Connect to your data:

iceberg-meta init              # interactive config setup
iceberg-meta doctor            # verify config + connectivity
iceberg-meta list-tables       # explore
iceberg-meta tui               # interactive browser

Or spin up a Docker playground:

iceberg-meta quickstart        # starts MinIO, seeds data, configures everything
iceberg-meta quickstart --down # tear down when done

Commands

Every data command supports --output json and --output csv.

Use Cases

Pre-merge validation in CI/CD

Confirm a pipeline write actually landed before merging:

iceberg-meta summary staging.orders          # row count, file count, latest snapshot
iceberg-meta diff staging.orders $OLD $NEW   # what changed between two snapshots
iceberg-meta files staging.orders -o csv     # pipe file-level stats into a check script

Debugging failing writes

Spark job "succeeded" but downstream dashboards are empty:

iceberg-meta snapshots staging.orders        # did the snapshot actually land?
iceberg-meta schema staging.orders --history # did a schema evolution break compatibility?
iceberg-meta files staging.orders            # are the new data files present and non-empty?

Monitoring table health

Spot small-file problems, partition skew, and compaction needs before they impact query performance:

iceberg-meta health warehouse.events

The health report covers file sizes (min/avg/median/max with small-file warnings), delete file accumulation, partition skew detection, column null rates, column storage distribution, and column value bounds.

Live monitoring

Watch for new snapshots as a pipeline runs:

iceberg-meta snapshots warehouse.events --watch 5

Onboarding and knowledge transfer

New team member needs to understand the data platform:

iceberg-meta tui

Incident response

Production data looks wrong — find when the issue was introduced:

iceberg-meta snapshots prod.customers        # find the suspicious snapshot IDs
iceberg-meta diff prod.customers 111 222     # compare record counts and file changes
iceberg-meta tree prod.customers             # drill into manifests and data files

Scripting and automation

Pipe machine-readable output into alerts or dashboards:

FILE_COUNT=$(iceberg-meta -o json summary db.events | jq '.file_count')
[ "$FILE_COUNT" -gt 1000 ] && echo "Small file problem detected"

iceberg-meta -o csv snapshots db.events > snapshots.csv
iceberg-meta -o json health db.events | jq '.[] | select(.Section == "Column Nulls")'

See it in action

$ iceberg-meta summary sales.orders

┌─────────────────────────────────────────────────────────────────┐
│                     sales.orders  Summary                       │
├──────────────────────┬──────────────────────────────────────────┤
│ Format version       │ 2                                        │
│ Total snapshots      │ 4                                        │
│ Total data files     │ 1                                        │
│ Total records        │ 15                                       │
│ Total size           │ 5.2 KB                                   │
│ Partition spec       │ region (identity)                        │
├──────────────────────┴──────────────────────────────────────────┤
│ Recent Operations                                               │
│  overwrite   2025-02-28 19:04    +15 rows   -60 rows            │
│  append      2025-02-28 19:04    +20 rows   -0 rows             │
│  append      2025-02-28 19:04    +15 rows   -0 rows             │
└─────────────────────────────────────────────────────────────────┘

$ iceberg-meta schema sales.customers --history

Schema 0  (initial)
├── customer_id: long
├── name: string
└── email: string

Schema 1  (+2 fields)
├── phone: string          ← added
└── signup_date: date      ← added

Schema 2  (1 rename)
└── email_address: string  ← renamed from email

TUI Keybindings

Configuration

Run iceberg-meta init for interactive setup, or create ~/.iceberg-meta.yaml manually. Credentials use ${VAR} placeholders resolved from the environment -- secrets never touch disk.

See docs/configuration.md for full details, environment variable overrides, and .env file support.

License

MIT