dataspoc-lens 0.1.0

Virtual warehouse — SQL over cloud Parquet via DuckDB

DataSpoc Lens

SQL over cloud Parquet. Query your data lake from the terminal.

Docs  |  Tutorial  |  Discord

Why Lens?

Data teams store Parquet in S3, GCS, or Azure but still spin up heavy warehouses just to run SQL. DataSpoc Lens mounts cloud buckets as DuckDB views and gives you an interactive shell, notebooks, AI-powered queries, and local caching -- all from a single CLI. No servers, no infrastructure, no data copying.

Highlights

• Zero infrastructure -- DuckDB runs in-process, no server or daemon
• Multi-cloud -- S3, GCS, and Azure Blob Storage via fsspec + httpfs
• Interactive shell -- SQL REPL with syntax highlighting and autocomplete
• AI Ask -- natural language to SQL using Ollama, Anthropic, or OpenAI
• Local cache -- download once, query offline
• Transforms -- numbered .sql files for repeatable data pipelines
• Notebook ready -- launch JupyterLab or Marimo with tables pre-mounted
• Export anywhere -- CSV, JSON, or Parquet with a single flag

Installation

pip install dataspoc-lens[s3]

Other install options

# Google Cloud Storage
pip install dataspoc-lens[gcs]

# Azure Blob Storage
pip install dataspoc-lens[azure]

# JupyterLab integration
pip install dataspoc-lens[jupyter]

# AI natural language queries
pip install dataspoc-lens[ai]

# Everything
pip install dataspoc-lens[all]

Quick Start

# Initialize configuration
dataspoc-lens init

# Register a cloud bucket
dataspoc-lens add-bucket s3://my-data-lake

# Run a SQL query
dataspoc-lens query "SELECT * FROM orders LIMIT 10"

# Launch the interactive shell
dataspoc-lens shell

# Ask a question in plain English
dataspoc-lens ask "how many orders were placed yesterday?"

Features Overview

Shell

An interactive SQL REPL powered by prompt_toolkit with syntax highlighting, autocomplete for table and column names, and dot commands (.tables, .schema, .export, .help).

$ dataspoc-lens shell
lens> SELECT customer_id, count(*) FROM orders GROUP BY 1 ORDER BY 2 DESC LIMIT 5;

Notebook

Launch JupyterLab or Marimo with all discovered tables pre-mounted as DuckDB views -- ready to query from the first cell.

dataspoc-lens notebook

AI Ask

Turn plain English into SQL. Lens sends your schema and a data sample to the LLM, gets back SQL, and executes it.

# Default: Ollama (local, no API key needed)
dataspoc-lens ask "top 10 customers by revenue this month"

# Or use a cloud provider
export DATASPOC_LLM_PROVIDER=anthropic
export DATASPOC_LLM_API_KEY=sk-...
dataspoc-lens ask "which products have declining sales?"

Cache

Download tables to ~/.dataspoc-lens/cache/ for offline work. Cache freshness is tracked automatically and views switch between remote and local transparently.

# Inside the shell
lens> .cache orders

Transforms

Numbered .sql files in ~/.dataspoc-lens/transforms/ that run in order, writing results to the /curated/ prefix.

transforms/
  001_clean_users.sql
  002_aggregate_orders.sql
  003_build_summary.sql

dataspoc-lens transform run

Export

Add --export to any query or ask command to save results as CSV, JSON, or Parquet.

dataspoc-lens query "SELECT * FROM orders" --export csv -o orders.csv
dataspoc-lens ask "monthly revenue" --export parquet -o revenue.parquet

Access Control

DataSpoc delegates all access control to your cloud provider's IAM. The recommended pattern is one bucket per permission level:

Bucket	Audience
s3://company-public	All employees
s3://company-finance	Finance team
s3://company-executive	C-level only

Each user's cloud credentials determine which buckets they can see. If a user lacks IAM permission, add-bucket fails with "Access Denied" and no data is exposed. Lens needs only read access.

Part of the DataSpoc Platform

Layer	Role
DataSpoc Pipe	Ingest data from APIs into cloud storage as Parquet
DataSpoc Lens	Query cloud Parquet with SQL, shell, notebooks, and AI
DataSpoc ML	Machine learning on your data lake (commercial)

Pipe writes. Lens reads. ML learns.

Community

• Discord -- Join the conversation for questions and discussion
• GitHub Issues -- Report bugs or request features
• Contributing -- PRs welcome. Run pytest tests/ -v before submitting.

License

Apache-2.0