pyds-ai

Ask a business question in plain English — d.ask("did churn spike in EMEA last quarter?") — and get back a real answer with a chart, the exact code behind it, and a plain-English caveat if the data can't really support the question. Built for someone who knows the business, not someone who knows pandas: no SQL, no coding background — if you can run a cell in a notebook (Jupyter, or Google Colab with nothing to install locally), you already have what it takes.

pip install pyds-ai

import pyds_ai as pyds

pyds-ai is the package name; pyds_ai is the import — they don't match on purpose, because the short names pyds and pyds-cli are already taken by unrelated packages on PyPI. Every example on this page shows both lines together so this is never a guessing game.

Your data doesn't leave the building by default. The model gets column names, types, and summary statistics — never your raw rows — unless you explicitly turn send_data_values on. See Privacy default below.

Pasting a spreadsheet into a chatbot gets you a confident, plausible number with no way to check its arithmetic — the same question asked twice can quietly come back with two different answers, and nobody notices until it's in front of leadership. The one rule that decides everything here: the LLM never produces a number. It writes the query and it writes the words. A deterministic engine (DuckDB) computes the answer — the same way, every time, from the same data. Every result can show you the exact code that produced it via result.code(), so a number never has to be taken on faith.

A real result card — the finding, the chart, and (expanded) the data, code, and trace.

A failed call renders as a card, not a traceback.

Contents

• Try it with no data and no key — see a real result before installing anything of your own
• Quickstart — the five-minute tour: load data, ask, model, chat with docs
• Real-world files, not clean CSVs — messy Excel, CSV, TXT, PDF
• Writing to a real database — SQL Server, Oracle, Postgres, MySQL, SQLite
• Metrics: define it once, everyone gets the same number
• Ask the questions leadership actually asks — the 8 no-LLM-math verbs
• Every result is the same shape
• Learning from a real result — for the person who wants the SQL and the stats
• Presentable output — decks, branded exports, .xlsx/.html
• Failures render as a card, not a traceback
• Privacy default
• Installing extras
• Command line
• Plugins
• One session, one process
• Development

Try it with no data and no key

pip install pyds-ai

import pyds_ai as pyds

pyds.demo()  # lists the bundled datasets
pyds.demo("product", offline=True)  # a real card, 5 canned answers, no key needed

pyds.demo("product") / pyds.demo("revenue") load the same bundled datasets as a normal DataSet — everything below works on them exactly as it would on your own data, once you're ready for pyds.setup().

Quickstart

The tour below is the whole surface area at a glance: load a file, ask a question, get a chart, train a model, ask a question of a folder of documents, then combine both into one agent. Each line works on its own — you don't need the ones before it.

pip install pyds-ai[anthropic]  # or [openai], [gemini] -- Ollama needs no extra

import pyds_ai as pyds

pyds.setup()  # one-time key wizard, writes to ~/.pyds_ai/config.toml

d = pyds.load("sales.csv")
d.suggest()  # ten business-language questions this data can answer — no model call
d.ask("which region grew fastest last quarter?")
d.chart("revenue by month, split by channel")

m = d.model(target="churn", positive_class="yes")  # which value is "churned" — required, not guessed
m.explain()
m.what_if(tenure=12, plan="premium")

k = pyds.knowledge("policies/")
k.ask("what is our refund window for enterprise?")

a = pyds.agent(uses=[d, k])
a.ask("did churn spike in any region where we changed the refund policy?")

e = pyds.evaluate(a, cases="qa_cases.csv")
e.watch()

d.suggest() generates its ten questions from the shape of the data alone (dates, categories, missing values) — no model call — and the top three already show up on d's own card the moment it loads, before you ask anything.

d.model() checks every column against the target for a suspiciously high correlation (> 0.95) — or a name that itself suggests it records the outcome or its aftermath (*_status, reason, cancel_*) — before training. Either shape usually means a column is derived from the target itself (e.g. a status code that's really just churn in disguise, or a cancel_reason that's only ever filled in for people who already churned) rather than a real predictor of it. A flagged column is excluded from training by default, not just warned about, so it can't quietly produce a hollow ~1.0 accuracy/F1/AUC and dominate m.explain()/m.what_if() underneath it; the caveat names which column and why, and m.why()["excluded_leaky"] lists them. A column correlated between 0.7 and 0.95 is kept but still named in a caveat, since that's also a plausible shape for a subtler leak. Pass d.model(target=..., allow_leaky=["that_column"]) if you're sure a flagged column is legitimate and want it back in.

For a two-value (binary) target, d.model() requires positive_class= naming which value is the outcome you're predicting. predict_proba's column order is just alphabetical — fine for {0, 1}, silently backwards for something like {"churned", "retained"} — so every probability, roc_auc, and m.what_if() reading would otherwise be relative to whichever value sorts last, with nothing on the card saying so.

Real-world files, not clean CSVs

Data reaches this audience as an Excel export from someone else's system — a merged title row, blank rows, the real header on row 4, six tabs with the actual data buried on the fifth one, dates that mean something different depending on where the file came from:

pyds.sheets("book.xlsx")  # lists every sheet's size, no loading
d = pyds.load("book.xlsx")  # finds the real header row automatically,
# loads the sheet that looks like the data (skipping a detected cover page),
# and says exactly what it assumed -- pass sheet="..." to pick one yourself
d2 = pyds.load_clipboard()  # pasted a range from Excel? this reads it directly
d3 = pyds.load("export.txt")  # sniffs the delimiter (comma/tab/pipe/semicolon)
pyds.pdf_tables("report.pdf")  # lists every table pdfplumber finds, before you load one
d4 = pyds.load("report.pdf")  # loads the first table found; table=1 picks another

d.caveats (shown right on the card) says which row it used as the header, how many sheets the workbook had and which one it loaded, and — for a date column like 01/02/2026 that's genuinely ambiguous — that it assumed month-first (US) and how to say otherwise: pyds.load("book.xlsx", dayfirst=True) for day-first (European) dates. This isn't just a friendlier message: a date column in a non-ISO format used to come back NULL for every row in every downstream d.compare()/d.trend()/etc. query, silently — load() now converts it to a real date type before anything else touches it. A .pdf's tables are detected heuristically (a PDF has no real table structure the way a spreadsheet does) — every load carries an unconditional caveat saying so, and pyds.pdf_tables(path).get(index) previews one table's raw extraction before you trust load() with it.

Writing to a real database

The other direction: once a table's loaded, d.export_to_sql() writes it to SQL Server, Oracle, Postgres, MySQL, or SQLite, via SQLAlchemy — needs pip install pyds-ai[db] plus the driver for your target database (e.g. pyodbc for SQL Server, oracledb for Oracle). Nothing about this is reachable from d.ask() — it's a completely separate, explicitly- invoked path, unlike everything else in this README.

d = pyds.load("export.csv")
r = d.export_to_sql(
    "mssql+pyodbc://user:pass@myserver/mydb?driver=ODBC+Driver+17+for+SQL+Server",
    "sales_data",
    if_exists="replace",  # pandas' own vocabulary: "fail" (the default), "replace", or "append"
)
print(r.headline)
# "Wrote 12,483 row(s) to 'sales_data' (replace) at mssql+pyodbc://user:***@myserver/mydb..."
r.teach()  # a walkthrough of the dtype mapping, batching, and why it was written this way

The connection string never appears anywhere with credentials — every headline, caveat, and trace entry shows only the redacted form, and even a failed connection's own driver exception gets the literal password scrubbed out before it reaches you. if_exists defaults to "fail" so a repeat call never silently overwrites anything.

Metrics: define it once, everyone gets the same number

Without a definition, "activation rate" gets re-guessed by the model on every call — two people asking the same question can get two different numbers. pyds.define() fixes a metric's formula once and caches it next to your data, in a plain YAML file a human can open and edit:

pyds.define(
    "activation rate",
    "users who completed onboarding divided by signups",
    grain="user",
    owner="you@company.com",
)
pyds.metrics()  # lists what's defined for this dataset
pyds.undefine("activation rate")

This writes (and reads) metrics.yaml next to the dataset's source file — the handoff artifact between the data person and the product owner. It ships with a starter template of ~15 common metrics (activation, retention, churn, conversion, MAU/WAU/DAU, ARPU, CAC, NRR, funnel-step conversion, LTV, and more) with descriptions filled in but no formula yet; pyds.define(name, description) resolves each one for your real data. Resolving a definition is the only place a model touches a metric's arithmetic — one call, cached forever after. Every ask() about a defined metric compiles that cached formula, and the card names which definition it used, in the interpretation line and in result.why(). If a question mentions something close to (but not exactly) a defined name, you get offered the defined metric instead of a fresh guess:

d.ask("what's our activation this quarter?")
# -> a Clarify card: "Did you mean the defined metric 'activation_rate'?"

Ask the questions leadership actually asks

d.compare("activation_rate", period="quarter", vs="target")
d.trend("mau", period="week", window=12)
d.drivers("revenue", period="quarter")  # "why did it fall" -> a ranked, waterfall answer
d.segment("conversion_rate")  # "which groups are different"
d.funnel(steps=["signed_up", "completed_onboarding", "made_purchase"])  # "where do people drop off"
d.cohort(by="signup_month")  # "is the March cohort better than January"
d.before_after("2026-03-01", "revenue", window="4 weeks")  # "did the launch work"
d.experiment("variant", "converted")  # "did the test win"

• d.compare(metric, period="quarter", vs=None) — current period, prior period, absolute and percentage change, direction. vs="target" compares against a target column or a fixed target set on the metric.
• d.trend(metric, period="week", window=12) — a plain-English read on whether the latest move is normal or worth a look ("down 4 percent, which is within the normal week-to-week swing for this metric"), never "standard deviation" or "confidence interval."
• d.drivers(metric, period="quarter") — decomposes a change across every categorical column, ranks the dimensions and categories that actually explain it, and renders a waterfall chart. An additive metric (a sum or count) gets an exact category-by-category contribution; a defined ratio metric gets a mix-vs-rate split (did the mix of segments shift, or did a segment's own rate change).
• d.segment(metric, dimensions=None) — which groups depart most from the overall average, ranked, with the sample size next to each so nobody acts on a segment of four users.
• d.funnel(steps, event_col=None, id_col=None, split_by=None) — step-by-step conversion, counting distinct entities at each named step, with per-step and overall conversion and the largest gap between consecutive steps (by count, not percentage) called out in the headline. Each step is a presence count (who/what ever had that event) rather than a verified chronological path, and the card says so — "drop-off" implies a sequential loss this counting method can't actually confirm. The headline says "people" only when the id column actually looks like one (user_id, customer_id, ...) — a funnel over order_id/session_id reads as orders or sessions, not people. Renders as a horizontal funnel chart; split_by breaks the same funnel out by a dimension (channel, plan, ...).
• d.cohort(by="signup_month", metric="retention") — the standard cohort retention table (cohorts down the side, periods since signup across the top), rendered as a heatmap, with a plain read on whether recent cohorts retain better or worse than older ones. by names a date column plus a period grain ("signup_week", "signup_quarter", ...).
• d.before_after(date, metric, window="4 weeks") — the window before a date vs. the window after it, both averages and the change — with an explicit, unconditional caveat that this is a before-and-after comparison, not a controlled test.
• d.experiment(group_col, metric, control=None) — an A/B read for someone who's never run one: both group sizes, both metric values, the lift, and a plain-English statement of whether the difference is unlikely to be due to chance (backed by a two-proportion z-test or Welch's t-test, every piece of statistics vocabulary kept out of the headline — it's still in result.why()). That's a statement about chance, not a recommendation: it never says a difference is "large enough to act on," since whether to act needs a cost/benefit call this package can't make, and it never estimates "how many more users you'd need" from the observed effect (a post-hoc power calculation, which isn't statistically valid). control="..." names which group is the baseline — without it, the two groups are ordered alphabetically and the card says so, since the lift's sign otherwise carries no guarantee about which one you actually consider the starting point. Every result also carries standing caveats about unit of randomization, peeking, and multiple comparisons, and flags heavy imbalance as a possible sample-ratio mismatch. Refuses, with a clear reason, when the groups are too small or too imbalanced to read.

metric accepts a defined metric name or a plain column name either way.

d.compare(), d.drivers(), d.segment(), and d.before_after() also take filter={"column": ..., "op": ..., "value": ...} (or a list of those) to scope the whole call to a segment — d.drivers("revenue", filter={"column": "region", "op": "=", "value": "EMEA"}) — without pre-filtering in pandas and reloading, or reaching for d.ask() (which needs a model and a key). The card names exactly what was filtered to, so a scoped result never looks identical to an unscoped one.

Accuracy: 186/203 (91.6%), run 2026-08-23 against claude-sonnet-4-6 — see benchmark/ to run the 200+-case golden set yourself, and its README for a full breakdown of what the remaining gap actually is (entirely ambiguous wording in a handful of the golden set's own questions at this point — not silently excluded, read before you trust this number). Scored by execution match (does the generated SQL return the same data as a known-correct query), not by comparing SQL text.

Every result is the same shape

Every call returns a Result — never a bare DataFrame, never a dict:

result.explain()  # what was done and why, in sentences
result.interpretation()  # the one-line "read as: ..." restatement of the question
result.code()  # the exact SQL/code that produced it
result.data()  # the underlying DataFrame, full precision — never reformatted
result.next()  # up to three suggested follow-up questions
result.why()  # trace: prompt version, model, cost, row counts
result.summary()  # three to five plain-English sentences for an email or speaker notes
result.teach()  # a walkthrough of the SQL and the stats behind this result, for learning
result.save(path)  # .html, .csv, .docx, .pptx, .xlsx, .pdf

An Answer/Chart card also opens with a one-sentence restatement of how the question was read — which metric (naming the defined metric used, if any), which filter, which grouping, and which time window — built from the plan structure, not a second model call. It's how you catch a wrong reading before it lands in a deck. A question that asks for a comparison ("...vs last quarter", "...compared to target") gets a real baseline_value computed alongside the answer, not just the current number with the comparison silently dropped. Every result derived from a dataset — not just ask(), also .model(), .profile(), .compare(), .trend(), and the rest — shows the data's as-of date (result.as_of) on its card and in every export, so a number repeated a week later still says which snapshot it came from. Numbers on the card, in chart labels, and in every export are formatted for presentation (thousands separators, $ when a column looks like money, % when it looks like a rate) — none of that touches result.data() or a CSV export, which stay full precision.

Learning from a real result

Every card deliberately keeps SQL and statistics out of the headline — but they're never hidden, and result.teach() is built for the person who actually wants them:

r = d.experiment("variant", "converted", control="A")
print(r.headline)  # "A (control) vs B (treatment): converted is 19.6% vs 23.2% ... unlikely to be due to random chance."

r.teach()
# Names the actual technique behind the headline (a two-proportion
# z-test here, because the outcome is yes/no) and why it fits, walks
# through the SQL clause by clause, connects the headline's numbers
# back to result.why()'s z-score and p-value, and ends with one
# concrete "try it yourself" question tied to a real number already on
# the card (e.g. "what happens to required_n_per_group with half the
# sample size?").

Grounded in the exact same result.code()/result.why() any result already carries — nothing new is computed, and it never describes a technique that isn't actually reflected in the SQL or the trace. A paid call like .summary(), cached separately after the first one; Failure.teach() never calls the model at all.

Presentable output

Anything this audience has to edit before showing it is a failure, so the output is built to go straight into an email, a deck, or a spreadsheet:

result.summary()  # "Revenue grew 12% this quarter, driven mostly by Enterprise/AMER. ..."

pyds.deck(
    [d.compare("revenue"), d.drivers("revenue"), d.segment("revenue")],
    title="Q2 Business Review",
    path="q2_review.pptx",
)

pyds.brand(logo="logo.png", colors=["#8e44ad"], template="corp.potx")

result.save(
    "report.xlsx"
)  # Answer (headline + a live chart), Data (autofiltered), How this was calculated

• result.summary() — three to five plain sentences: the finding, its size, and the one caveat that matters, written by the model from the numbers already on the card — it never computes one itself. A paid call, so it's cached after the first one; Failure.summary() never calls the model at all.
• pyds.deck(results, title=, path="deck.pptx") — a title slide, one large-chart slide per result, a closing summary slide, and an appendix with the full tables, the metric definitions used, and the SQL. Charts are native, editable PowerPoint chart objects built straight from each result's own data — not a picture of the notebook chart — so a d.cohort() result becomes a real colored table instead of a chart that doesn't apply to it.
• pyds.brand(logo=, colors=, template=) — configure once, applied everywhere: every chart's colors, the notebook/HTML card's accent and logo, and the logo/colors on every .docx/.pptx/.xlsx export. template is your own .potx/.pptx file, used as the base for every pptx export and deck instead of the stock layout. A missing logo or template file is skipped quietly — a decorative asset should never block a report.
• .xlsx export — "Answer" (headline, interpretation, as-of date, caveats, a live chart referencing the Data sheet), "Data" (the full table, real numeric cells with a formatted number style, autofiltered), "How this was calculated" (the metric definition, the SQL, the model call and prompt version — or an honest "no model call, computed deterministically" for the eight verbs that never touch the LLM at all).
• Shareable .html — already a single, genuinely self-contained file (the chart is inline SVG, there's no external stylesheet, font, or image — a configured logo is embedded as a base64 data URI, never a file path). Includes a print stylesheet, so printing or saving to PDF from a browser shows the full table/code/trace instead of a collapsed disclosure triangle.

Failures render as a card, not a traceback

A call that hits a real problem — no API key configured, a bad file path, an unknown target column — returns a Failure (also a Result) instead of raising. Failure is falsy, so if not result: is the check:

d = pyds.load("sales.csv")
if not d:
    print(d.explain())  # what went wrong, and what to try next

For a script where nobody is there to read a card, turn this off and get the old raise-through behavior back:

pyds.config.set_strict(True)

Privacy default

send_data_values = false by default: the model gets column names, types and summary statistics, not your raw rows. Your data does not leave the building unless you explicitly opt in. A date column's min/max (used to resolve a relative phrase like "last quarter" against your data's own timeline, not today's real-world date) is one such summary statistic and is sent either way; per-row sample values are the part this flag gates. Set it via:

pip install pyds-ai

import pyds_ai as pyds

cfg = pyds.config.load_config()
cfg["privacy"]["send_data_values"] = True
pyds.config.save_config(cfg)

Your API key is stored in your OS credential store automatically when pip install pyds-ai[keyring] is installed. Without it, the key falls back to ~/.pyds_ai/config.toml, restricted to your account with owner-only file permissions on macOS/Linux — on Windows, os.chmod can't restrict an NTFS ACL, so the file stays plain text there and pyds.setup() says so plainly rather than claiming a protection it can't actually provide; install pyds-ai[keyring] for real protection on Windows. Either way, pyds.setup() never prints the key back.

Installing extras

pip install pyds-ai[ml]          # AutoML: scikit-learn (+ joblib, shap)
pip install pyds-ai[rag]         # RAG: pdf/docx/html parsing
pip install pyds-ai[rag-chroma]  # persistent vector store (Chroma)
pip install pyds-ai[rag-faiss]   # in-process ANN vector store (FAISS)
pip install pyds-ai[rag-pgvector]# Postgres + pgvector vector store
pip install pyds-ai[report]      # result.save(): .pptx, .docx; pyds.deck() (.xlsx and Excel loading need no extra — openpyxl is core)
pip install pyds-ai[report-pdf]  # result.save(...".pdf"); needs Pango/cairo/GDK-pixbuf too (see below), not just pip install
pip install pyds-ai[anthropic]   # or [openai], [gemini] — Ollama needs no extra
pip install pyds-ai[keyring]     # store your API key in the OS credential store
pip install pyds-ai[db]          # d.export_to_sql(): SQLAlchemy (+ your own DB driver, e.g. pyodbc/oracledb)
pip install pyds-ai[pdf-tables]  # pyds.pdf_tables() / loading tables out of a .pdf
pip install pyds-ai[all]         # everything above, all three vector stores included

Importing a missing extra returns a friendly message with the exact pip command, never a bare ImportError. The default pyds.knowledge() vector store is a zero-dependency in-memory hashing embedder — RAG works the moment pyds-ai[rag] is installed, no embedding API call required.

report-pdf is split out from report on purpose: pip install always succeeds for weasyprint itself, but actually rendering a PDF needs Pango/cairo/GDK-pixbuf — native libraries pip can't install (on Windows, a separate GTK3 runtime). result.save("x.pptx")/.docx never need any of that; a .pdf save without those native libraries fails with a message telling you what to install, instead of every other format failing along with it.

Command line

$ pyds doctor
[X]  Config file
      Fix: run `pyds setup`.
[X]  API key
      Fix: run `pyds setup` (or set the anthropic provider's API key environment variable).
[OK] DuckDB

Run the fix command(s) above, then run `pyds doctor` again.

pyds doctor --json prints the same checks as raw JSON for scripts. pyds setup runs the same interactive key wizard as pyds.setup(), from a terminal instead of a notebook. pyds report <data> "<question>" --out <path> and pyds eval <cases.csv> --docs <folder> run the ask and evaluate pipelines end to end without opening Python; add --watch to pyds eval to record the run and flag a regression against the last one.

Plugins

Internal teams can register a company database reader or a private model provider without forking, via entry points:

[project.entry-points."pyds.providers"]
internal = "mycompany.pyds_plugin:InternalLLM"

Anything registered under the pyds.providers group is picked up by pyds.setup(provider="internal") / pyds.config.get_llm_client() the same way the four built-in providers are.

One session, one process

pyds-ai keeps one process-wide session — one DuckDB connection shared by every dataset you load. That fits its intended use (a single notebook kernel, one person at a time) but it is not a concurrency-safe building block: a Streamlit app or web API serving multiple users on different threads would have them all silently sharing the same connection, loaded datasets, and defined metrics, with no isolation between requests. If you're wiring pyds-ai into something like that, give each request/user its own process (or otherwise don't share one import pyds_ai across them) rather than assuming this is handled for you.

Development

git clone <this repo> && cd pyds-ai
python -m venv .venv && .venv/Scripts/activate   # or source .venv/bin/activate
pip install -e ".[all,test]"
pytest

The test suite runs fully offline: a fake LLM client (tests/support.py) stands in for every provider, so no API key or network access is needed to validate the package.

Publishing (maintainers)

python -m pip install --upgrade build twine
python -m build            # writes dist/*.whl and dist/*.tar.gz
twine check dist/*
twine upload dist/*        # or: twine upload --repository testpypi dist/*

jsaimanoj/pyds_ai is the real repo path, used consistently across pyproject.toml, this README's badges and screenshot URLs, mkdocs.yml, CONTRIBUTING.md, and .github/ISSUE_TEMPLATE/config.yml. Two things depend on it working correctly, not just looking right: the README's two screenshots are absolute raw.githubusercontent.com URLs (a relative path renders fine on GitHub but shows broken on the PyPI project page, which doesn't resolve repo-relative links) — they'll go live once the docs/images/*.png files are actually pushed to main; and the CI/ docs-deploy workflows trigger on push to main, this repo's default branch.

License

Apache-2.0 — see LICENSE.