tablesleuth 0.4.2.post1

TableSleuth - a Textual TUI for Open Table Format forensics (OTF) with data profiling.

TableSleuth

A powerful terminal-based tool for deep inspection of Parquet files and Apache Iceberg tables. Analyze file structure, metadata, row groups, column statistics, and table evolution with an intuitive TUI interface.

Key Features

Parquet Analysis

• Deep File Inspection - Comprehensive metadata extraction using PyArrow
• Row Group Analysis - Examine distribution, compression, and statistics
• Column Profiling - Profile data using GizmoSQL (DuckDB over Arrow Flight SQL)
• Data Sampling - Preview and filter data with column selection
• Directory Scanning - Recursively discover and inspect Parquet files

Iceberg Table Analysis

• Snapshot Navigation - Browse table history and metadata evolution
• Performance Testing - Compare query performance across snapshots
• Delete File Inspection - Analyze MOR (Merge-on-Read) delete files
• Schema Evolution - Track schema changes over time
• Catalog Support - Local SQLite, AWS Glue, and AWS S3 Tables

Interface

• Interactive TUI - Keyboard-driven navigation with rich visualizations
• Multi-Source Support - Local files, S3, and Iceberg catalogs
• Performance Optimized - Async operations, caching, and lazy loading

Screenshots

Parquet File Inspection

File Structure & Schema	Row Group Analysis
Data Sample View	Column Profiling

Iceberg Table Analysis

Snapshot Overview	Performance Testing
Delete Files (MOR)	Snapshot Comparison

Quick Start

# Install with uv (recommended)
uv sync

# Inspect a Parquet file
tablesleuth inspect data/file.parquet

# Inspect a directory (recursive)
tablesleuth inspect data/warehouse/

# Inspect an Iceberg table
tablesleuth inspect db.table --catalog local

# Inspect AWS S3 Tables (using ARN)
tablesleuth inspect "arn:aws:s3tables:us-east-2:123456789012:bucket/my-bucket/table/db.table"

📚 Documentation:

• Quick Start Guide - Get started with examples
• Setup Guide - Complete installation and configuration
• User Guide - Comprehensive usage documentation

Installation

Requirements: Python 3.13+ and uv

# Install from PyPI
pip install tablesleuth

# Or install from source
git clone https://github.com/jamesbconner/TableSleuth
cd TableSleuth
uv sync

# Verify installation
tablesleuth --version

# Initialize configuration files
tablesleuth init

See TABLESLEUTH_SETUP.md for detailed setup including AWS, GizmoSQL, and catalog configuration.

Quick Start

# 1. Initialize configuration (first time only)
tablesleuth init

# 2. Edit configuration files
#    - tablesleuth.toml (main config)
#    - .pyiceberg.yaml (catalog config)

# 3. Verify configuration
tablesleuth config-check

# 4. Start inspecting files
tablesleuth inspect data/file.parquet

Configuration

Quick Setup

# Initialize configuration files with interactive prompts
tablesleuth init

# Check configuration and test connections
tablesleuth config-check
tablesleuth config-check -v  # Verbose output

Configuration Files

tablesleuth.toml - Main configuration:

[catalog]
default = "local"  # Default Iceberg catalog

[gizmosql]
uri = "grpc+tls://localhost:31337"
username = "gizmosql_username"
password = "gizmosql_password"
tls_skip_verify = true

Configuration Priority:

1. Environment variables (TABLESLEUTH_*)
2. Local config files (./tablesleuth.toml, ./.pyiceberg.yaml)
3. Home config files (~/tablesleuth.toml, ~/.pyiceberg.yaml)
4. Built-in defaults

Iceberg Catalogs

Configure PyIceberg in .pyiceberg.yaml:

catalog:
  local:
    type: sql
    uri: sqlite:////path/to/catalog.db
    warehouse: file:///path/to/warehouse

For detailed configuration:

• Setup Guide - All catalog types and AWS configuration
• GizmoSQL Deployment - Profiling backend setup

Usage

CLI Commands

# Configuration management
tablesleuth init                    # Initialize config files
tablesleuth config-check            # Validate configuration
tablesleuth config-check -v         # Detailed validation

# Inspect Parquet files
tablesleuth inspect file.parquet
tablesleuth inspect directory/
tablesleuth inspect s3://bucket/path/file.parquet

# Inspect Iceberg tables
tablesleuth inspect db.table --catalog local
tablesleuth inspect "arn:aws:s3tables:region:account:bucket/name/table/db.table"

# Launch Iceberg viewer
tablesleuth iceberg --catalog local --table db.table

TUI Navigation

Key	Action
q	Quit
r	Refresh
f	Filter columns
Tab	Switch tabs
↑/↓	Navigate
Enter	Select

See User Guide for complete keyboard shortcuts and features.

Optional: GizmoSQL Profiling

Enable column profiling and performance testing with GizmoSQL (DuckDB over Arrow Flight SQL).

Quick Setup:

# Install GizmoSQL (macOS ARM64 example)
curl -L https://github.com/gizmodata/gizmosql/releases/download/v1.12.10/gizmosql_cli_macos_arm64.zip \
  | sudo unzip -o -d /usr/local/bin -

# Start server
gizmosql_server -P password -Q -T ~/.certs/cert0.pem ~/.certs/cert0.key

See GizmoSQL Deployment Guide for complete setup and EC2 deployment.

Architecture

TableSleuth uses a layered architecture:

• TUI Layer - Textual-based terminal interface with rich visualizations
• Service Layer - Business logic for file inspection, profiling, and discovery
• Integration Layer - PyArrow for Parquet, PyIceberg for tables, GizmoSQL for profiling

See Architecture Guide for detailed technical documentation.

Development

# Install with dev dependencies
uv sync --all-extras

# Run tests
pytest

# Run quality checks
uv run pre-commit run --all-files

# Type checking
mypy src/

See Development Setup for complete development environment setup.

Documentation

Getting Started

• Quick Start - Examples and common workflows
• Setup Guide - Installation and configuration
• User Guide - Complete feature documentation

Advanced Topics

• Performance Profiling - Query performance analysis
• GizmoSQL Deployment - Profiling backend setup
• EC2 Deployment - Automated AWS deployment

Development

• Development Setup - Dev environment and workflows
• Architecture - System design and technical details
• Developer Guide - API reference and contributing

What's New

v0.4.0 (Current)

• 🎉 Now available on PyPI! Install with pip install tablesleuth
• 🔄 Package renamed to tablesleuth for consistency
• 🤖 Automated CI/CD with GitHub Actions
• 📦 Enhanced PyPI metadata and publishing workflow
• ✅ All existing features from v0.3.0

v0.3.0

• ✅ Parquet file inspection (local and S3)
• ✅ Iceberg snapshot navigation and analysis
• ✅ Delete file inspection and MOR forensics
• ✅ Snapshot comparison and performance testing
• ✅ Column profiling with GizmoSQL
• ✅ AWS Glue and S3 Tables catalog support
• ✅ Interactive TUI with rich visualizations

Roadmap

• Delta Lake and Hudi support
• Schema evolution visualization
• Export capabilities (JSON, CSV reports)
• REST catalog support
• Automated optimization recommendations

Contributing

Contributions welcome! See Developer Guide and Development Setup.

License

MIT License - See LICENSE for details.

Support

• Issues & Features: GitHub Issues
• Documentation: See docs/ directory
• Changelog: CHANGELOG.md