pyds-ai

Data science and LLM work from a notebook, without writing much code.

pip install pyds-ai

import pyds_ai as pyds

pyds-ai is the package name; pyds_ai is the import — they don't match on purpose, because the short names pyds and pyds-cli are already taken by unrelated packages on PyPI. Every example on this page shows both lines together so this is never a guessing game.

Your data doesn't leave the building by default. The model gets column names, types, and summary statistics — never your raw rows — unless you explicitly turn send_data_values on. See Privacy default below.

A real result card — the finding, the chart, and (expanded) the data, code, and trace.

A failed call renders as a card, not a traceback.

The one rule that decides everything: the LLM never produces a number. It writes the query and it writes the words. A deterministic engine (DuckDB) computes the answer. Every result can show you the exact code that produced it via result.code().

Try it with no data and no key

pip install pyds-ai

import pyds_ai as pyds

pyds.demo()  # lists the bundled datasets
pyds.demo("product", offline=True)  # a real card, 5 canned answers, no key needed

pyds.demo("product") / pyds.demo("revenue") load the same bundled datasets as a normal DataSet — everything below works on them exactly as it would on your own data, once you're ready for pyds.setup().

Quickstart

pip install pyds-ai[anthropic]  # or [openai], [gemini] -- Ollama needs no extra

import pyds_ai as pyds

pyds.setup()  # one-time key wizard, writes to ~/.pyds_ai/config.toml

d = pyds.load("sales.csv")
d.suggest()  # ten business-language questions this data can answer — no model call
d.ask("which region grew fastest last quarter?")
d.chart("revenue by month, split by channel")

m = d.model(target="churn")
m.explain()
m.what_if(tenure=12, plan="premium")

k = pyds.knowledge("policies/")
k.ask("what is our refund window for enterprise?")

a = pyds.agent(uses=[d, k])
a.ask("did churn spike in any region where we changed the refund policy?")

e = pyds.evaluate(a, cases="qa_cases.csv")
e.watch()

d.suggest() generates its ten questions from the shape of the data alone (dates, categories, missing values) — no model call — and the top three already show up on d's own card the moment it loads, before you ask anything.

Real-world files, not clean CSVs

Data reaches this audience as an Excel export from someone else's system — a merged title row, blank rows, the real header on row 4, six tabs with the actual data buried on the fifth one, dates that mean something different depending on where the file came from:

pyds.sheets("book.xlsx")  # lists every sheet's size, no loading
d = pyds.load("book.xlsx")  # finds the real header row automatically,
# loads the sheet that looks like the data,
# and says exactly what it assumed
d2 = pyds.load_clipboard()  # pasted a range from Excel? this reads it directly
d3 = pyds.load("export.txt")  # sniffs the delimiter (comma/tab/pipe/semicolon)
pyds.pdf_tables("report.pdf")  # lists every table pdfplumber finds, before you load one
d4 = pyds.load("report.pdf")  # loads the first table found; table=1 picks another

d.caveats (shown right on the card) says which row it used as the header, how many sheets the workbook had and which one it loaded, and — for a date column like 01/02/2026 that's genuinely ambiguous — that it assumed month-first (US) and how to say otherwise: pyds.load("book.xlsx", dayfirst=True) for day-first (European) dates. This isn't just a friendlier message: a date column in a non-ISO format used to come back NULL for every row in every downstream d.compare()/d.trend()/etc. query, silently — load() now converts it to a real date type before anything else touches it. A .pdf's tables are detected heuristically (a PDF has no real table structure the way a spreadsheet does) — every load carries an unconditional caveat saying so, and pyds.pdf_tables(path).get(index) previews one table's raw extraction before you trust load() with it.

Writing to a real database

The other direction: once a table's loaded, d.export_to_sql() writes it to SQL Server, Oracle, Postgres, MySQL, or SQLite, via SQLAlchemy — needs pip install pyds-ai[db] plus the driver for your target database (e.g. pyodbc for SQL Server, oracledb for Oracle). Nothing about this is reachable from d.ask() — it's a completely separate, explicitly- invoked path, unlike everything else in this README.

d = pyds.load("export.csv")
r = d.export_to_sql(
    "mssql+pyodbc://user:pass@myserver/mydb?driver=ODBC+Driver+17+for+SQL+Server",
    "sales_data",
    if_exists="replace",  # pandas' own vocabulary: "fail" (the default), "replace", or "append"
)
print(r.headline)
# "Wrote 12,483 row(s) to 'sales_data' (replace) at mssql+pyodbc://user:***@myserver/mydb..."
r.teach()  # a walkthrough of the dtype mapping, batching, and why it was written this way

The connection string never appears anywhere with credentials — every headline, caveat, and trace entry shows only the redacted form, and even a failed connection's own driver exception gets the literal password scrubbed out before it reaches you. if_exists defaults to "fail" so a repeat call never silently overwrites anything.

Metrics: define it once, everyone gets the same number

Without a definition, "activation rate" gets re-guessed by the model on every call — two people asking the same question can get two different numbers. pyds.define() fixes a metric's formula once and caches it next to your data, in a plain YAML file a human can open and edit:

pyds.define(
    "activation rate",
    "users who completed onboarding divided by signups",
    grain="user",
    owner="you@company.com",
)
pyds.metrics()  # lists what's defined for this dataset
pyds.undefine("activation rate")

This writes (and reads) metrics.yaml next to the dataset's source file — the handoff artifact between the data person and the product owner. It ships with a starter template of ~15 common metrics (activation, retention, churn, conversion, MAU/WAU/DAU, ARPU, CAC, NRR, funnel-step conversion, LTV, and more) with descriptions filled in but no formula yet; pyds.define(name, description) resolves each one for your real data. Resolving a definition is the only place a model touches a metric's arithmetic — one call, cached forever after. Every ask() about a defined metric compiles that cached formula, and the card names which definition it used, in the interpretation line and in result.why(). If a question mentions something close to (but not exactly) a defined name, you get offered the defined metric instead of a fresh guess:

d.ask("what's our activation this quarter?")
# -> a Clarify card: "Did you mean the defined metric 'activation_rate'?"

Ask the questions leadership actually asks

d.compare("activation_rate", period="quarter", vs="target")
d.trend("mau", period="week", window=12)
d.drivers("revenue", period="quarter")  # "why did it fall" -> a ranked, waterfall answer
d.segment("conversion_rate")  # "which groups are different"
d.funnel(steps=["signed_up", "completed_onboarding", "made_purchase"])  # "where do people drop off"
d.cohort(by="signup_month")  # "is the March cohort better than January"
d.before_after("2026-03-01", "revenue", window="4 weeks")  # "did the launch work"
d.experiment("variant", "converted")  # "did the test win"

• d.compare(metric, period="quarter", vs=None) — current period, prior period, absolute and percentage change, direction. vs="target" compares against a target column or a fixed target set on the metric.
• d.trend(metric, period="week", window=12) — a plain-English read on whether the latest move is normal or worth a look ("down 4 percent, which is within the normal week-to-week swing for this metric"), never "standard deviation" or "confidence interval."
• d.drivers(metric, period="quarter") — decomposes a change across every categorical column, ranks the dimensions and categories that actually explain it, and renders a waterfall chart. An additive metric (a sum or count) gets an exact category-by-category contribution; a defined ratio metric gets a mix-vs-rate split (did the mix of segments shift, or did a segment's own rate change).
• d.segment(metric, dimensions=None) — which groups depart most from the overall average, ranked, with the sample size next to each so nobody acts on a segment of four users.
• d.funnel(steps, event_col=None, id_col=None, split_by=None) — step-by-step conversion, counting distinct people at each named step, with per-step and overall conversion and the biggest drop-off (by people lost, not percentage) called out in the headline. Renders as a horizontal funnel chart; split_by breaks the same funnel out by a dimension (channel, plan, ...).
• d.cohort(by="signup_month", metric="retention") — the standard cohort retention table (cohorts down the side, periods since signup across the top), rendered as a heatmap, with a plain read on whether recent cohorts retain better or worse than older ones. by names a date column plus a period grain ("signup_week", "signup_quarter", ...).
• d.before_after(date, metric, window="4 weeks") — the window before a date vs. the window after it, both averages and the change — with an explicit, unconditional caveat that this is a before-and-after comparison, not a controlled test.
• d.experiment(group_col, metric) — an A/B read for someone who's never run one: both group sizes, both metric values, the lift, and a plain-English verdict ("large enough to act on" or "could easily be noise, you'd need roughly N more users per group"), backed by a real statistical test with every piece of statistics vocabulary kept out of the answer (it's still in result.why() for anyone who wants it). Refuses, with a clear reason, when the groups are too small or too imbalanced to read.

metric accepts a defined metric name or a plain column name either way.

Accuracy: pending first run — see benchmark/ to run the 200+-case golden set yourself. It's scored by execution match (does the generated SQL return the same data as a known-correct query), not by comparing SQL text.

Every result is the same shape

Every call returns a Result — never a bare DataFrame, never a dict:

result.explain()  # what was done and why, in sentences
result.code()  # the exact SQL/code that produced it
result.data()  # the underlying DataFrame, full precision — never reformatted
result.next()  # up to three suggested follow-up questions
result.why()  # trace: prompt version, model, cost, row counts
result.summary()  # three to five plain-English sentences for an email or speaker notes
result.teach()  # a walkthrough of the SQL and the stats behind this result, for learning
result.save(path)  # .html, .csv, .docx, .pptx, .xlsx, .pdf

An Answer/Chart card also opens with a one-sentence restatement of how the question was read — which metric (naming the defined metric used, if any), which filter, which grouping, and which time window — built from the plan structure, not a second model call. It's how you catch a wrong reading before it lands in a deck. Every result derived from a dataset — not just ask(), also .model(), .profile(), .compare(), .trend(), and the rest — shows the data's as-of date (result.as_of) on its card and in every export, so a number repeated a week later still says which snapshot it came from. Numbers on the card, in chart labels, and in every export are formatted for presentation (thousands separators, $ when a column looks like money, % when it looks like a rate) — none of that touches result.data() or a CSV export, which stay full precision.

Learning from a real result

Every card deliberately keeps SQL and statistics out of the headline — but they're never hidden, and result.teach() is built for the person who actually wants them:

r = d.experiment("variant", "converted")
print(r.headline)  # "A vs B: converted is 19.6% vs 23.2% ... large enough to act on"

r.teach()
# Names the actual technique behind the headline (a two-proportion
# z-test here, because the outcome is yes/no) and why it fits, walks
# through the SQL clause by clause, connects the headline's numbers
# back to result.why()'s z-score and p-value, and ends with one
# concrete "try it yourself" question tied to a real number already on
# the card (e.g. "what happens to required_n_per_group with half the
# sample size?").

Grounded in the exact same result.code()/result.why() any result already carries — nothing new is computed, and it never describes a technique that isn't actually reflected in the SQL or the trace. A paid call like .summary(), cached separately after the first one; Failure.teach() never calls the model at all.

Presentable output

Anything this audience has to edit before showing it is a failure, so the output is built to go straight into an email, a deck, or a spreadsheet:

result.summary()  # "Revenue grew 12% this quarter, driven mostly by Enterprise/AMER. ..."

pyds.deck(
    [d.compare("revenue"), d.drivers("revenue"), d.segment("revenue")],
    title="Q2 Business Review",
    path="q2_review.pptx",
)

pyds.brand(logo="logo.png", colors=["#8e44ad"], template="corp.potx")

result.save(
    "report.xlsx"
)  # Answer (headline + a live chart), Data (autofiltered), How this was calculated

• result.summary() — three to five plain sentences: the finding, its size, and the one caveat that matters, written by the model from the numbers already on the card — it never computes one itself. A paid call, so it's cached after the first one; Failure.summary() never calls the model at all.
• pyds.deck(results, title=, path="deck.pptx") — a title slide, one large-chart slide per result, a closing summary slide, and an appendix with the full tables, the metric definitions used, and the SQL. Charts are native, editable PowerPoint chart objects built straight from each result's own data — not a picture of the notebook chart — so a d.cohort() result becomes a real colored table instead of a chart that doesn't apply to it.
• pyds.brand(logo=, colors=, template=) — configure once, applied everywhere: every chart's colors, the notebook/HTML card's accent and logo, and the logo/colors on every .docx/.pptx/.xlsx export. template is your own .potx/.pptx file, used as the base for every pptx export and deck instead of the stock layout. A missing logo or template file is skipped quietly — a decorative asset should never block a report.
• .xlsx export — "Answer" (headline, interpretation, as-of date, caveats, a live chart referencing the Data sheet), "Data" (the full table, real numeric cells with a formatted number style, autofiltered), "How this was calculated" (the metric definition, the SQL, the model call and prompt version — or an honest "no model call, computed deterministically" for the eight verbs that never touch the LLM at all).
• Shareable .html — already a single, genuinely self-contained file (the chart is inline SVG, there's no external stylesheet, font, or image — a configured logo is embedded as a base64 data URI, never a file path). Includes a print stylesheet, so printing or saving to PDF from a browser shows the full table/code/trace instead of a collapsed disclosure triangle.

Failures render as a card, not a traceback

A call that hits a real problem — no API key configured, a bad file path, an unknown target column — returns a Failure (also a Result) instead of raising. Failure is falsy, so if not result: is the check:

d = pyds.load("sales.csv")
if not d:
    print(d.explain())  # what went wrong, and what to try next

For a script where nobody is there to read a card, turn this off and get the old raise-through behavior back:

pyds.config.set_strict(True)

Privacy default

send_data_values = false by default: the model gets column names, types and summary statistics, not your raw rows. Your data does not leave the building unless you explicitly opt in. Set it via:

pip install pyds-ai

import pyds_ai as pyds

cfg = pyds.config.load_config()
cfg["privacy"]["send_data_values"] = True
pyds.config.save_config(cfg)

Your API key is stored in your OS credential store automatically when pip install pyds-ai[keyring] is installed. Without it, the key falls back to ~/.pyds_ai/config.toml, written with owner-only file permissions. Either way, pyds.setup() never prints the key back.

Installing extras

pip install pyds-ai[ml]          # AutoML: scikit-learn (+ joblib, shap)
pip install pyds-ai[rag]         # RAG: pdf/docx/html parsing
pip install pyds-ai[rag-chroma]  # persistent vector store (Chroma)
pip install pyds-ai[rag-faiss]   # in-process ANN vector store (FAISS)
pip install pyds-ai[rag-pgvector]# Postgres + pgvector vector store
pip install pyds-ai[report]      # result.save(): .pptx, .docx, .pdf; pyds.deck() (.xlsx and Excel loading need no extra — openpyxl is core)
pip install pyds-ai[anthropic]   # or [openai], [gemini] — Ollama needs no extra
pip install pyds-ai[keyring]     # store your API key in the OS credential store
pip install pyds-ai[db]          # d.export_to_sql(): SQLAlchemy (+ your own DB driver, e.g. pyodbc/oracledb)
pip install pyds-ai[pdf-tables]  # pyds.pdf_tables() / loading tables out of a .pdf
pip install pyds-ai[all]

Importing a missing extra returns a friendly message with the exact pip command, never a bare ImportError. The default pyds.knowledge() vector store is a zero-dependency in-memory hashing embedder — RAG works the moment pyds-ai[rag] is installed, no embedding API call required.

Command line

$ pyds doctor
[X]  Config file
      Fix: run `pyds setup`.
[X]  API key
      Fix: run `pyds setup` (or set the anthropic provider's API key environment variable).
[OK] DuckDB

Run the fix command(s) above, then run `pyds doctor` again.

pyds doctor --json prints the same checks as raw JSON for scripts. pyds setup runs the same interactive key wizard as pyds.setup(), from a terminal instead of a notebook. pyds report <data> "<question>" --out <path> and pyds eval <cases.csv> --docs <folder> run the ask and evaluate pipelines end to end without opening Python.

Plugins

Internal teams can register a company database reader or a private model provider without forking, via entry points:

[project.entry-points."pyds.providers"]
internal = "mycompany.pyds_plugin:InternalLLM"

Anything registered under the pyds.providers group is picked up by pyds.setup(provider="internal") / pyds.config.get_llm_client() the same way the four built-in providers are.

Development

git clone <this repo> && cd pyds-ai
python -m venv .venv && .venv/Scripts/activate   # or source .venv/bin/activate
pip install -e ".[all,test]"
pytest

The test suite runs fully offline: a fake LLM client (tests/support.py) stands in for every provider, so no API key or network access is needed to validate the package.

Publishing (maintainers)

python -m pip install --upgrade build twine
python -m build            # writes dist/*.whl and dist/*.tar.gz
twine check dist/*
twine upload dist/*        # or: twine upload --repository testpypi dist/*

pyds-ai/pyds-ai is a placeholder repo path, used consistently across pyproject.toml, this README's badges and screenshot URLs, mkdocs.yml, CONTRIBUTING.md, and .github/ISSUE_TEMPLATE/config.yml — a single find-and-replace for the real org/repo once it exists on GitHub. Two things depend on it working correctly, not just looking right: the README's two screenshots are absolute raw.githubusercontent.com URLs (a relative path renders fine on GitHub but shows broken on the PyPI project page, which doesn't resolve repo-relative links), and the CI/ docs-deploy workflows trigger on push to main — confirm the real repo's default branch is actually named main before relying on either.

License

Apache-2.0 — see LICENSE.