sqlstream 0.1.0

A lightweight SQL query engine for data exploration with lazy evaluation and intelligent optimizations

SQLStream

A lightweight, pure-Python SQL query engine for CSV and Parquet files with lazy evaluation and intelligent optimizations.

📖 Full Documentation | 🚀 Quick Start | 💬 Discussions

---

Quick Example

# Query a CSV file
$ sqlstream query "SELECT * FROM 'data.csv' WHERE age > 25"

# Join multiple files
$ sqlstream query "SELECT c.name, o.total FROM 'customers.csv' c JOIN 'orders.csv' o ON c.id = o.customer_id"

# Interactive mode for wide tables
$ sqlstream query data.csv "SELECT * FROM data" --interactive

Features

• 🚀 Pure Python - No database installation required
• 📊 Multiple Formats - CSV, Parquet files, HTTP URLs
• ⚡ 10-100x Faster - Optional pandas backend for performance
• 🔗 JOIN Support - INNER, LEFT, RIGHT joins
• 📈 Aggregations - GROUP BY with COUNT, SUM, AVG, MIN, MAX
• 🔢 Type System - Automatic schema inference with type checking
• 🎨 Beautiful Output - Rich tables, JSON, CSV formatting
• 🖥️ Interactive Mode - Scrollable table viewer with Textual
• 🔍 Smart Optimizations - Column pruning, predicate pushdown, lazy evaluation
• 📦 Lightweight - Minimal dependencies, works everywhere

Installation

Basic (CSV only):

pip install sqlstream

All features (recommended):

pip install "sqlstream[all]"

See Installation Guide for more options.

Quick Start

CLI Usage

# Simple query
$ sqlstream query data.csv "SELECT name, salary FROM data WHERE salary > 80000"

# With pandas backend for performance
$ sqlstream query data.csv "SELECT * FROM data" --backend pandas

# JSON output
$ sqlstream query data.csv "SELECT * FROM data" --format json

# Interactive mode
$ sqlstream query data.csv "SELECT * FROM data" --interactive

Python API

from sqlstream import query

# Execute query (lazy evaluation)
results = query("data.csv").sql("SELECT * FROM data WHERE age > 25")

# Iterate over results
for row in results:
    print(row)

# Or convert to list
results_list = query("data.csv").sql("SELECT * FROM data").to_list()

Documentation

Full documentation: https://subhayu99.github.io/sqlstream

Key sections:

• Quick Start Guide - Get started in 5 minutes
• SQL Reference - Supported SQL syntax
• CLI Reference - Command-line interface
• Python API - Programmatic usage
• Examples - Real-world examples
• Architecture - How it works

Development Status

Current Phase: 8 (Type System & Schema Inference)

• ✅ Phase 0-2: Core query engine with Volcano model
• ✅ Phase 3: Parquet support
• ✅ Phase 4: Aggregations & GROUP BY
• ✅ Phase 5: JOIN operations (INNER, LEFT, RIGHT)
• ✅ Phase 5.5: Pandas backend (10-100x speedup)
• ✅ Phase 6: HTTP data sources
• ✅ Phase 7: CLI with beautiful output
• ✅ Phase 7.5: Interactive mode with Textual
• ✅ Phase 7.6: Inline file path support
• ✅ Phase 8: Type system & schema inference
• 🚧 Phase 9: Error handling & user feedback
• 🚧 Phase 10: Testing & documentation

Test Coverage: 358 tests, 53% coverage

Performance

SQLStream offers two execution backends:

Backend	Speed	Use Case
Python	Baseline	Learning, small files (<100K rows)
Pandas	10-100x faster	Production, large files (>100K rows)

Benchmark (1M rows):

• Python backend: 52s
• Pandas backend: 0.8s ⚡ 65x faster

Architecture

SQLStream uses the Volcano iterator model for query execution:

SQL Query → Parser → AST → Planner → Optimizer → Executor → Results
                                          ↓
                            (Column Pruning, Predicate Pushdown,
                             Lazy Evaluation)

Key concepts:

• Lazy Evaluation: Rows are processed on-demand
• Column Pruning: Only read columns that are used
• Predicate Pushdown: Apply filters early to reduce data scanned
• Two Backends: Pure Python (learning) and Pandas (performance)

See Architecture Guide for details.

Contributing

Contributions are welcome! See Contributing Guide for details.

Development setup:

# Clone repository
git clone https://github.com/subhayu99/sqlstream.git
cd sqlstream

# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Format code
ruff format .
ruff check .

License

MIT License - see LICENSE for details.

---

Built with ❤️ by the SQLStream Team

📖 Documentation • 🐛 Issues • 💬 Discussions