DataSpark — AI-Powered Data Science Co-Pilot

A Python library that brings Claude's data science expertise into your local workflow — Jupyter notebooks, scripts, terminal, anywhere Python runs.

No browser needed. No logging in. Just import and go.

---

Quick Start

1. Install

pip install dataspark-ai              # from PyPI
# or
pip install dataspark-ai[full]        # includes sklearn, matplotlib, seaborn, plotly, scipy
# or from source
git clone https://github.com/KTG0409/dataspark.git
cd dataspark && pip install -e ".[dev]"

2. Set your API key

export ANTHROPIC_API_KEY="sk-ant-api03-..."

Get a key at console.anthropic.com/settings/keys

3. Use It

from dataspark import Spark

spark = Spark()

# Explore a dataset — get instant analysis, quality checks, and recommendations
spark.explore("sales_data.csv")

# Ask any data science question
spark.ask("Should I use one-hot encoding or target encoding for high-cardinality categoricals?")

# Design a complete project
spark.project("Build a customer churn prediction model for our SaaS platform")

# Brainstorm creative analysis ideas
spark.brainstorm("I have 3 years of e-commerce transaction data with 2M rows")

# Generate production code
spark.code("Build a feature engineering pipeline for time series with lag features and rolling stats")

# Get best practices guidance
spark.best_practices("Cross-validation strategies for time series data")

# Interactive session (like chatting with Claude)
spark.chat()

---

Core Features

spark.explore(source) — Dataset Analysis

Pass a CSV, Excel file, DataFrame, or URL. DataSpark will:

• Profile every column (types, distributions, outliers, correlations)
• Flag data quality issues
• Recommend specific analyses based on what it sees
• Ask you clarifying questions about your goals
• Provide ready-to-run code snippets

spark.explore("customers.csv")
spark.explore("https://data.example.com/dataset.csv")
spark.explore(my_dataframe, name="revenue")

# Focus on something specific
spark.explore("data.csv", focus="I need to predict the 'churned' column")

spark.project(description) — Project Design

Describe what you want to build. DataSpark designs the full pipeline:

spark.project("Forecast demand for 500 SKUs across 12 warehouses, daily granularity")
spark.project("Build a recommendation engine for our content platform")
spark.project("Anomaly detection for network traffic logs, ~10M events/day")

spark.brainstorm(context) — Idea Generation

Get creative, ranked ideas from quick wins to big bets:

spark.brainstorm("We have clickstream data, purchase history, and customer support tickets")
spark.brainstorm("Our marketing team wants to understand campaign attribution")

spark.code(request) — Code Generation

Get complete, production-quality Python code:

spark.code("XGBoost pipeline with Optuna hyperparameter tuning")
spark.code("Automated EDA function that generates a PDF report")
spark.code("FastAPI endpoint that serves predictions from a pickled model")

spark.ask(question) — Ask Anything

Maintains conversation history so you can have a back-and-forth:

spark.ask("What's the best way to handle class imbalance?")
spark.ask("Show me how to implement SMOTE with that approach")
spark.ask("Now how do I evaluate it properly?")

spark.chat() — Interactive Terminal Session

Full interactive mode with slash commands:

/explore data.csv    — Load and analyze a dataset
/project <desc>      — Design a project
/brainstorm <ctx>    — Generate ideas
/code <request>      — Generate code
/model sonnet        — Switch models
/save conversation.md — Save chat history
/clear               — Reset context
/help                — Show commands
/quit                — Exit

---

Configuration

# Model selection (default: Claude Sonnet)
spark = Spark(model="opus")      # Most capable
spark = Spark(model="sonnet")    # Balanced (default)
spark = Spark(model="haiku")     # Fastest / cheapest

# Longer responses
spark = Spark(max_tokens=8192)

# Debug mode
spark = Spark(verbose=True)

---

Command-Line Usage

# Interactive chat
dataspark

# Explore a dataset
dataspark explore data.csv
dataspark explore data.csv -f "focus on the target variable"

# Quick question
dataspark ask "When should I use Ridge vs Lasso?"

# Project design
dataspark project "Build a fraud detection system"

# Use a specific model
dataspark -m opus explore big_dataset.parquet

---

Jupyter Notebook Tips

from dataspark import Spark
spark = Spark()

# Load data through spark — it profiles automatically
df = spark.load("data.csv")

# Now all questions are context-aware
spark.ask("What feature engineering should I do?")
spark.ask("Write the code for that")

# You can also explore at any point
spark.explore(focus="relationships between price and demand")

---

Architecture

dataspark/
├── __init__.py      # Clean exports
├── core.py          # Spark class — main interface & API calls
├── explorer.py      # DataExplorer — load & profile datasets
├── profiles.py      # DataProfile — statistical profiling
├── prompts.py       # System prompts for each mode
└── cli.py           # Command-line interface

The library works by:

1. Profiling your data locally (pandas — nothing leaves your machine except the summary)
2. Building rich context from the profile (statistics, distributions, quality issues)
3. Sending that context + your question to Claude via the API
4. Maintaining conversation history so follow-ups are contextual

Your raw data never leaves your machine. Only statistical summaries and column metadata are sent to the API.

---

Extending DataSpark

Custom System Prompts

from dataspark import Spark

spark = Spark()
spark._current_system = """You are a financial data science expert.
Focus on: regulatory compliance, risk modeling, backtesting.
Always consider: data leakage, survivorship bias, look-ahead bias."""

spark.ask("How should I backtest this trading strategy?")

Adding Data Context Manually

spark._data_context = """
We have a PostgreSQL database with:
- transactions (50M rows, 3 years)
- customers (2M rows)
- products (10K SKUs)
Business: B2B SaaS, $50M ARR, 15% annual churn
"""
spark.ask("What analyses would have the most business impact?")

---

Cost Awareness

API costs per ~1000 tokens (approximate):

Model	Input	Output
Haiku	$0.001	$0.005
Sonnet	$0.003	$0.015
Opus	$0.015	$0.075

A typical explore() call uses ~2-4K tokens. An interactive session might use 10-50K tokens total. Use spark = Spark(model="haiku") for cost-sensitive workloads.

---

Privacy & Security

• Your raw data stays local. Only statistical summaries (means, distributions, column names, 3 sample rows) are sent to the API.
• API key is yours. Use a personal key — it's billed to your Anthropic account, not tied to any employer.
• No logging by default. Conversations are in-memory only unless you /save them.
• Review what's sent. Call spark.explorer.context_for_llm() to see exactly what goes to the API.

---

License

MIT — use it however you want.