cloudcat 0.6.2

Preview and analyze data files in Google Cloud Storage, AWS S3, and Azure Data Lake Storage Gen2 from your terminal

CloudCat

The Swiss Army knife for viewing cloud storage data from your terminal

Documentation • Installation • Quick Start • Features • Examples

---

CloudCat is a powerful command-line tool that lets you instantly preview and analyze data files stored in Google Cloud Storage (GCS), Amazon S3, and Azure Blob Storage — without downloading entire files. Think of it as cat, head, and less combined, but for cloud storage with built-in support for CSV, JSON, Parquet, Avro, ORC, and plain text formats.

Why CloudCat?

• No Downloads Required — Stream and preview data directly from cloud storage
• Format-Aware — Intelligently handles CSV, JSON, Parquet, Avro, ORC, and plain text files
• Directory Smart — Automatically discovers data files in Spark/Hive/Kafka output directories
• Beautiful Output — Colorized tables, pretty-printed JSON, and schema visualization
• Developer Friendly — Simple CLI with sensible defaults and powerful options
• Compression Support — Automatic decompression of gzip, zstd, lz4, snappy, and bz2 files
• SQL-like Filtering — Filter rows with WHERE clauses (e.g., --where "status=active")

Installation

Homebrew (macOS Apple Silicon)

The easiest way to install on Apple Silicon Macs (M1/M2/M3/M4) — no Python required:

brew tap jonathansudhakar1/cloudcat https://github.com/jonathansudhakar1/cloudcat.git && brew install cloudcat

This installs a self-contained binary that includes Python and all dependencies.

Intel Mac users: Homebrew bottles are not available for Intel. Please use pip install 'cloudcat[all]' instead.

To upgrade:

brew update && brew upgrade cloudcat

Note: On first run, macOS may block the app. Go to System Settings > Privacy & Security and click "Allow", or run:

xattr -d com.apple.quarantine $(which cloudcat)

pip (Python)

# Full installation with all formats and compression
pip install 'cloudcat[all]'

# Standard installation (includes GCS, S3, and Azure support)
pip install cloudcat

# With Parquet file support
pip install 'cloudcat[parquet]'

# With Avro file support
pip install 'cloudcat[avro]'

# With ORC file support (uses pyarrow)
pip install 'cloudcat[orc]'

# With compression support (zstd, lz4, snappy)
pip install 'cloudcat[compression]'

# With lakehouse table support (Delta Lake / Apache Iceberg)
pip install 'cloudcat[tables]'

Note: If using zsh (default on macOS), quotes around extras are required to prevent shell interpretation of brackets.

To upgrade:

pip install --upgrade 'cloudcat[all]'

Requirements

• Homebrew: macOS (Apple Silicon only). Intel Mac users should use pip.
• pip: Python 3.9+ (all platforms)
• Cloud provider credentials configured (see Authentication)

Quick Start

# Preview a CSV file from GCS (PATH is positional; -p/--path also works)
cloudcat gcs://my-bucket/data.csv

# Preview a local file — no cloud credentials needed
cloudcat ./data.parquet

# Preview a Parquet file from S3
cloudcat -p s3://my-bucket/analytics/events.parquet

# Preview JSON data from Azure with pretty formatting
cloudcat -p abfss://[e-mail]/logs.json -o jsonp

# Read Avro files from Kafka
cloudcat -p s3://my-bucket/kafka-export.avro

# Read ORC files from Hive
cloudcat -p gcs://my-bucket/hive-table.orc

# Read log files as plain text
cloudcat -p abfss://[e-mail]/app.log -i text

# Read from a Spark output directory
cloudcat -p s3://my-bucket/spark-output/ -i parquet

# Read a Delta Lake or Iceberg table (auto-detected, current snapshot only)
cloudcat s3://lake/orders_delta/ -w "status=refunded" -n 10

# Read compressed files (auto-detected)
cloudcat -p gcs://my-bucket/data.csv.gz

# Filter rows with WHERE (streams; stops at -n matches; AND/OR supported)
cloudcat s3://bucket/users.parquet --where "status=active AND age>30"

# Skip first 100 rows (pagination)
cloudcat -p gcs://bucket/data.csv --offset 100 -n 10

Features

Cloud Storage Support

Provider	URL Scheme	Status
Google Cloud Storage	gcs:// or gs://	✅ Supported
Amazon S3	s3://	✅ Supported
Azure Data Lake Gen2	abfss://	✅ Supported
Local files	file:// or a plain path	✅ Supported (no credentials needed)

File Format Support

Format	Auto-Detect	Use Case
CSV	✅	General data files
JSON	✅	API responses, configs
JSON Lines	✅	Log files, streaming data
Parquet	✅	Spark/analytics data
Avro	✅	Kafka, data pipelines
ORC	✅	Hive, Hadoop ecosystem
Text	✅	Log files, plain text
TSV	Via --delimiter	Tab-separated data
Delta Lake	✅ table dirs	Lakehouse tables (cloudcat[delta])
Apache Iceberg	✅ table dirs	Lakehouse tables (cloudcat[iceberg])

Streaming Efficiency

CloudCat uses intelligent streaming to minimize data transfer and egress costs:

Format	Compression	Streams	Column Projection	Early Row Stop
Parquet	None/Internal	✅	✅ Range requests	✅
Parquet	External (.gz)	❌	❌	❌
ORC	None/Internal	✅ Stripe-by-stripe	✅	✅
ORC	External (.gz)	❌	✅	✅
CSV	None	✅	❌	✅
CSV	gzip/zstd/lz4/bz2	✅	❌	✅
CSV	snappy	❌	❌	❌
JSON Lines	None/streamable	✅	❌	✅
JSON Array	Any	❌	❌	❌
Avro	Any	✅	✅ Record-level	✅
Text	Any streamable	✅	N/A	✅

• Streams: Only reads data as needed, stops early when row limit is reached
• Column Projection: For Parquet, only fetches required column chunks via HTTP range requests
• Early Row Stop: Stops reading when --num-rows limit is reached

Compression Support

Format	Extension	Built-in	Use Case
Gzip	.gz, .gzip	✅	Most common, universal
Bzip2	.bz2	✅	High compression ratio
Zstandard	.zst, .zstd	Optional	Fast, modern compression
LZ4	.lz4	Optional	Very fast decompression
Snappy	.snappy	Optional	Hadoop ecosystem

CloudCat automatically detects and decompresses files based on extension (e.g., data.csv.gz, logs.json.zst).

Output Formats

Format	Flag	Description
Table	-o table	Rounded table with bold headers and type-aware coloring (default)
JSON	-o json	Standard JSON Lines output
Pretty JSON	-o jsonp	Syntax-highlighted, indented JSON
CSV	-o csv	Comma-separated values

Key Capabilities

• Schema Inspection — View column names and data types
• Column Selection — Display only the columns you need
• Row Limiting — Control how many rows to preview
• Row Offset — Skip first N rows for pagination/sampling
• WHERE Filtering — Filter rows with SQL-like conditions
• Record Counting — Get total record counts (with Parquet metadata optimization)
• Multi-File Reading — Combine data from multiple files in a directory
• Custom Delimiters — Support for tab, pipe, semicolon, and other delimiters
• Auto Decompression — Transparent handling of compressed files

Examples

Basic Usage

# Preview first 10 rows (default)
cloudcat -p gcs://bucket/data.csv

# Preview 50 rows
cloudcat -p s3://bucket/data.parquet -n 50

# Show only specific columns
cloudcat -p gcs://bucket/users.json -c id,name,email

# View schema only (no data)
cloudcat -p s3://bucket/events.parquet -s schema_only

Working with Different Formats

# CSV with custom delimiter (tab-separated)
cloudcat -p gcs://bucket/data.tsv -d "\t"

# Pipe-delimited file
cloudcat -p s3://bucket/export.txt -d "|"

# Semicolon-delimited (common in European data)
cloudcat -p gcs://bucket/report.csv -d ";"

# JSON array file
cloudcat -p s3://bucket/config.json

# JSON Lines file (auto-detected)
cloudcat -p gcs://bucket/events.jsonl

Filtering and Pagination

# Filter rows with WHERE (streams; stops at -n matches; AND/OR supported)
cloudcat s3://bucket/users.parquet --where "status=active AND age>30"
cloudcat -p gcs://bucket/events.json --where "age>30"
cloudcat -p s3://bucket/logs.csv --where "level=ERROR"

# String matching filters
cloudcat -p gcs://bucket/data.csv --where "name contains john"
cloudcat -p s3://bucket/emails.json --where "email endswith @gmail.com"
cloudcat -p abfss://[e-mail]/app.log --where "message startswith ERROR"

# Skip first N rows (pagination)
cloudcat -p gcs://bucket/data.csv --offset 100 -n 10

# Combine offset with filters
cloudcat -p s3://bucket/users.parquet --where "active=true" --offset 50 -n 20

Compressed Files

# Gzip compressed (built-in)
cloudcat -p gcs://bucket/data.csv.gz
cloudcat -p s3://bucket/logs.json.gz

# Zstandard compressed (requires: pip install cloudcat[zstd])
cloudcat -p gcs://bucket/events.parquet.zst

# LZ4 compressed (requires: pip install cloudcat[lz4])
cloudcat -p s3://bucket/data.csv.lz4

# Bzip2 compressed (built-in)
cloudcat -p abfss://[e-mail]/archive.json.bz2

Directory Operations

CloudCat intelligently handles directories containing multiple data files (common with Spark, Hive, and distributed processing outputs):

# Auto-detect and read first data file in directory
cloudcat -p gcs://bucket/spark-output/

# Read and combine multiple files (up to 25MB by default)
cloudcat -p s3://bucket/daily-logs/ -m all

# Read up to 100MB of data from multiple files
cloudcat -p gcs://bucket/events/ -m all --max-size-mb 100

# Force reading only the first file
cloudcat -p s3://bucket/output/ -m first

CloudCat automatically:

• Skips empty files
• Ignores metadata files (_SUCCESS, _metadata, .crc, etc.)
• Prioritizes files matching the specified format
• Reports which files were selected

Output Format Examples

# Default table output (great for terminals)
cloudcat -p gcs://bucket/data.csv
# ┌────┬────────────┬─────────┐
# │ id │ name       │ value   │
# ├────┼────────────┼─────────┤
# │ 1  │ Alice      │ 100     │
# │ 2  │ Bob        │ 200     │
# └────┴────────────┴─────────┘

# Pretty JSON (great for nested data)
cloudcat -p s3://bucket/events.json -o jsonp
# {
#   "id": 1,
#   "name": "Alice",
#   "metadata": {
#     "created": "2024-01-15"
#   }
# }

# JSON Lines (great for piping to jq)
cloudcat -p gcs://bucket/data.parquet -o json | jq '.name'

# CSV (great for further processing)
cloudcat -p s3://bucket/data.json -o csv > output.csv

Data Pipeline Examples

# Convert Parquet to CSV
cloudcat -p gcs://bucket/data.parquet -o csv -n 0 > data.csv

# Preview and filter with jq
cloudcat -p s3://bucket/events.json -o json | jq 'select(.status == "error")'

# Quick data validation
cloudcat -p gcs://bucket/import.csv -s schema_only

# Sample data from large dataset
cloudcat -p s3://bucket/big-table.parquet -n 100 -c user_id,event_type

# Export specific columns to CSV
cloudcat -p gcs://bucket/users.parquet -c email,created_at -o csv -n 0 > emails.csv

Real-World Use Cases

Debugging Spark Jobs

# Check output of a Spark job
cloudcat -p gcs://data-lake/jobs/daily-etl/output/ -i parquet -n 20

# Verify schema matches expectations
cloudcat -p s3://analytics/spark-output/ -s schema_only

Log Analysis

# Preview recent logs
cloudcat -p gcs://logs/app/2024-01-15/ -m all -n 50

# Check error logs (combine with grep)
cloudcat -p s3://logs/errors/ -o json | grep "ERROR"

Data Validation

# Quick sanity check on data export
cloudcat -p gcs://exports/daily/users.csv -s show

# Verify record count
cloudcat -p s3://warehouse/transactions.parquet --count

Format Conversion

# Convert tab-separated to comma-separated
cloudcat -p gcs://imports/data.tsv -d "\t" -o csv > converted.csv

# Convert JSON to CSV for spreadsheet import
cloudcat -p s3://api-dumps/response.json -o csv > data.csv

Command Reference

Usage: cloudcat [OPTIONS]

Options:
  PATH                         Positional path: gs://, gcs://, s3://, abfss://,
                               file:// URL, or a plain local path
                               (-p/--path remains as a compatible alias)
                               Format: gcs://bucket/path, s3://bucket/path,
                               or abfss://[e-mail]/path

  -o, --output-format TEXT     Output format: table, json, jsonp, csv
                               [default: table]

  -O, --output-file PATH       Write rendered data to a file instead of stdout

  -i, --input-format TEXT      Input format: csv, json, parquet, avro, orc,
                               text, delta, iceberg
                               [default: auto-detect from extension]

  -c, --columns TEXT           Comma-separated list of columns to display
                               [default: all columns]

  -n, --num-rows INTEGER       Number of rows to display (0 for all)
                               [default: 10]

  --offset INTEGER             Skip first N rows
                               [default: 0]

  -w, --where TEXT             Filter rows with SQL-like conditions; combine
                               with AND/OR. Streams the file and stops at
                               --num-rows matches; Parquet skips row groups
                               via column statistics.
                               Examples: "status=active", "age>30 AND age<65",
                               "level=ERROR or level=FATAL"

  -s, --schema TEXT            Schema display: show, dont_show, schema_only
                               [default: show]

  --count                      Show total record count (scans entire file)

  -m, --multi-file-mode TEXT   Directory handling: auto, first, all
                               [default: auto]

  --max-size-mb INTEGER        Max data size for multi-file mode in MB
                               [default: 25]

  -d, --delimiter TEXT         CSV delimiter (use \t for tab)
                               [default: comma]

  --no-color                   Disable colored output (also honors NO_COLOR).
                               Color is auto-disabled when output is piped.

  --profile TEXT               AWS profile name (for S3 access)

  --project TEXT               GCP project ID (for GCS access)

  --credentials TEXT           Path to GCP service account JSON file

  --stats                      Show per-column statistics (nulls, distinct,
                               min/max) over the retrieved rows

  --config-profile TEXT        Named profile from ~/.config/cloudcat/config.toml

  --install-completion [bash|zsh|fish]
                               Enable completion with one command (adds one
                               line to your shell rc file; idempotent)

  --completion [bash|zsh|fish] Print the shell completion script and exit.
                               Once enabled, TAB completes PATH too: local
                               files natively, s3://<TAB> lists buckets,
                               s3://bucket/dir/<TAB> lists children (uses
                               --profile/--project if already typed).
                               bash note: escape the URL colon issue with
                               COMP_WORDBREAKS=${COMP_WORDBREAKS//:/}

  --az-access-key TEXT         Azure storage account access key

  -y, --yes                    Skip confirmation prompts (for scripting)

  --help                       Show this message and exit

WHERE Clause Operators

Operator	Example	Description
=	status=active	Exact match
!=	type!=deleted	Not equal
>	age>30	Greater than
<	price<100	Less than
>=	count>=10	Greater than or equal
<=	score<=50	Less than or equal
contains	name contains john	Case-insensitive substring match
startswith	email startswith admin	String prefix match
endswith	file endswith .csv	String suffix match

Authentication

Google Cloud Storage

CloudCat uses Application Default Credentials (ADC). Set up authentication using one of these methods:

# Option 1: User credentials (for development)
gcloud auth application-default login

# Option 2: Service account via environment variable
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"

# Option 3: Service account via CLI option
cloudcat -p gcs://bucket/data.csv --credentials /path/to/service-account.json

# Option 4: Specify GCP project
cloudcat -p gcs://bucket/data.csv --project my-gcp-project

Amazon S3

CloudCat uses the standard AWS credential chain:

# Option 1: Environment variables
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_DEFAULT_REGION="us-east-1"

# Option 2: AWS credentials file (~/.aws/credentials)
aws configure

# Option 3: AWS named profile
cloudcat -p s3://bucket/data.csv --profile production

# Option 4: IAM role (for EC2/ECS/Lambda)
# Automatically detected

Azure Data Lake Storage Gen2

CloudCat supports multiple authentication methods for Azure ADLS Gen2:

# Option 1: Access key (simplest)
cloudcat -p abfss://[e-mail]/data.csv --az-access-key "YOUR_KEY"

# Option 2: Access key via environment variable
export AZURE_STORAGE_ACCESS_KEY="YOUR_KEY"
cloudcat -p abfss://[e-mail]/data.csv

# Option 3: Azure CLI with DefaultAzureCredential
az login
cloudcat -p abfss://[e-mail]/data.csv

Path format: abfss://[e-mail]/path/to/file

Performance Tips

1. Counting is off by default — use --count only when you need the total record count
2. Prefer Parquet format when possible — record counts are instant from metadata
3. Use --num-rows to limit data transfer for large files
4. Use --columns to select only needed columns (especially effective with Parquet)
5. Use -m first when you only need a sample from directories with many files

Troubleshooting

Common Issues

"google-cloud-storage package is required"

pip install cloudcat[gcs]

"boto3 package is required"

pip install cloudcat[s3]

"pyarrow package is required"

pip install cloudcat[parquet]

"azure-storage-blob package is required"

pip install cloudcat[azure]

"fastavro package is required"

pip install cloudcat[avro]

"pyarrow with ORC support is required"

pip install cloudcat[orc]

"zstandard package is required for .zst files"

pip install cloudcat[zstd]
# or for all compression formats:
pip install cloudcat[compression]

"lz4 package is required for .lz4 files"

pip install cloudcat[lz4]

"python-snappy package is required for .snappy files"

pip install cloudcat[snappy]

Authentication errors

• GCS: Run gcloud auth application-default login
• S3: Run aws configure or check your credentials
• Azure: Set AZURE_STORAGE_CONNECTION_STRING or AZURE_STORAGE_ACCOUNT_URL and run az login

"Could not infer format from path"

# Specify the format explicitly
cloudcat -p gcs://bucket/data -i parquet

Contributing

Contributions are welcome! Here's how you can help:

1. Report bugs — Open an issue with reproduction steps
2. Suggest features — Open an issue describing the use case
3. Submit PRs — Fork, create a branch, and submit a pull request

Development Setup

# Clone the repository
git clone https://github.com/jonathansudhakar1/cloudcat.git
cd cloudcat

# Create virtual environment
python -m venv venv
source venv/bin/activate  # or `venv\Scripts\activate` on Windows

# Install in development mode with all dependencies
pip install -e ".[all]"

# Run tests
pytest

Use with AI Agents

CloudCat is optimized for AI coding agents: stdout carries only data (diagnostics go to stderr, color auto-disables when piped), -o json emits parseable NDJSON, -y makes every command non-interactive, filtering streams with Parquet row-group pushdown, and local files need no credentials.

The repo ships an Agent Skill at skills/cloudcat/SKILL.md — a compact reference that teaches agents the optimal scanning recipes (schema discovery, filtered sampling, metadata-fast counts, column profiling) with zero trial-and-error.

Install the skill (pick one):

# 1. Built-in installer — the skill ships inside the package (works offline)
cloudcat --install-skill claude          # Claude Code, all projects (~/.claude/skills)
cloudcat --install-skill claude-project  # Claude Code, this project (./.claude/skills)
cloudcat --install-skill codex           # OpenAI Codex CLI (~/.codex/skills)
cloudcat --install-skill print           # stdout — pipe into any other agent's instructions

# 2. Claude Code plugin marketplace
/plugin marketplace add jonathansudhakar1/cloudcat
/plugin install cloudcat@cloudcat

# 3. Or fetch it directly (no cloudcat install needed)
curl -fsSL --create-dirs -o ~/.claude/skills/cloudcat/SKILL.md \
  https://raw.githubusercontent.com/jonathansudhakar1/cloudcat/main/skills/cloudcat/SKILL.md

The recipes agents learn:

cloudcat s3://bucket/events/ -s schema_only -y                 # structure, no data read
cloudcat s3://bucket/events/ --count -s schema_only -y         # exact rows (Parquet/ORC: metadata-only)
cloudcat s3://bucket/events/ -w "type=purchase AND amount>250" \
         -n 5 -o json -s dont_show -y                          # filtered NDJSON sample
cloudcat s3://bucket/events/ --stats -n 0 -s dont_show -y      # nulls/distinct/min/max per column

Works with any agent framework that can run shell commands and load the SKILL.md (or plain markdown) as instructions. See the AI Agents docs for details.

Roadmap

• Azure Blob Storage support
• Avro format support
• ORC format support
• Plain text format support
• SQL-like filtering (--where clause)
• Compression support (gzip, zstd, lz4, snappy, bz2)
• Row offset/pagination (--offset)
• Output to file with --output-file
• Compound WHERE conditions (AND/OR)
• Streaming WHERE with Parquet row-group pushdown
• Local file support (file:// or plain paths)
• Configuration file with named profiles
• Column statistics (--stats)
• Shell completion (bash/zsh/fish)
• Delta Lake support (current snapshot; cloudcat[delta])
• Apache Iceberg support (current snapshot, catalog-less; cloudcat[iceberg])
• Interactive mode with pagination
• Table time travel (--at-version / --at-snapshot)

Related Projects

• s3cmd — S3 command-line tool
• gsutil — Google Cloud Storage CLI
• aws-cli — AWS command-line interface
• azcopy — Azure Storage data transfer tool
• duckdb — In-process SQL OLAP database

License

MIT License — see LICENSE for details.

Report Bug • Request Feature