parquet-lf 1.0.0

A lingua franca utility for converting between data formats (JSON, CSV) and Parquet.

parquet-lf

A lingua franca utility for converting between data formats (NDJSON, CSV) and Parquet.

Installation

uv tool install parquet-lf

Usage

Convert to Parquet

# Convert CSV to Parquet
parquet-lf to-parquet csv input.csv -o output.parquet

# Convert NDJSON to Parquet
parquet-lf to-parquet ndjson input.ndjson -o output.parquet

# jsonl is an alias for ndjson
parquet-lf to-parquet jsonl input.jsonl -o output.parquet

Convert from Parquet

# Convert Parquet to CSV
parquet-lf from-parquet csv input.parquet -o output.csv

# Convert Parquet to NDJSON
parquet-lf from-parquet ndjson input.parquet -o output.ndjson

# jsonl is an alias for ndjson
parquet-lf from-parquet jsonl input.parquet -o output.jsonl

Output to stdout

When the -o/--output flag is omitted, output is written to stdout:

# Output CSV to stdout
parquet-lf from-parquet csv input.parquet

# Pipe to another command
parquet-lf from-parquet csv input.parquet | head -10

# Output Parquet to stdout (binary) and redirect to file
parquet-lf to-parquet csv input.csv > output.parquet

Note: Logs are written to stderr, so they won't interfere with piped data.

Inspect Files

Use the info command to view file metadata and schema without loading the entire dataset:

# Show file info (schema, row count, size)
parquet-lf info examples/sample.parquet

# Show file info with preview of first N rows
parquet-lf info --head 5 examples/sample.parquet
parquet-lf info -n 5 examples/sample.csv

The info command supports all formats (Parquet, CSV, NDJSON) and auto-detects the format from the file extension.

Help

parquet-lf --help
parquet-lf to-parquet --help
parquet-lf from-parquet --help
parquet-lf info --help

Supported Formats

NDJSON (Newline Delimited JSON)

NDJSON is a format where each line is a valid JSON object. It's a true tabular peer to CSV, making it ideal for data interchange.

Example NDJSON file:

{"name": "alice", "value": 10}
{"name": "bob", "value": 20}
{"name": "charlie", "value": 30}

Both ndjson and jsonl commands are supported as synonyms.

CSV

Standard comma-separated values format with a header row.

Example CSV file:

name,value
alice,10
bob,20
charlie,30

Example Files

The examples/ directory contains sample data files for experimenting with the CLI:

• examples/sample.parquet - Parquet format
• examples/sample.csv - CSV format
• examples/sample.ndjson - NDJSON format

These files contain the same 5-row dataset with columns: id, name, age, city, score.

Development

See CONTRIBUTING.md for development setup and guidelines.

License

MIT License - see LICENSE for details.