ds-fetch-py-sdk

Python SDK for AI Commons Fetch: read-only SQL over your tenant's data lake, plus natural-language questions through ds-tools. For the command line, see ds-fetch-cli.

uv add ds-fetch-py-sdk            # or: pip install ds-fetch-py-sdk
uv add 'ds-fetch-py-sdk[pandas]'  # adds QueryResult.to_pandas()

Quick start

from ds_fetch_py_sdk import Fetch

with Fetch("dev") as fetch:  # signs in with the browser on first use
    for ds in fetch.datasets():
        print(ds.name, ds.row_count, [c.name for c in ds.columns])

    result = fetch.query("""
        SELECT department, count(*) AS n
        FROM gold."5003105c-8a84-5f77-ab74-5e57003112b8"
        GROUP BY department
    """)
    print(result.records())  # [{'department': 'engineering', 'n': 1520}, ...]

    answer = fetch.ask("Headcount per department?", datasets=[ds.name])
    print(answer.sql, answer.rows)

Environments

Pick one with Fetch("local") or AIC_ENV=local. Override single URLs with AIC_FETCH_URL, AIC_TOOLS_URL and AIC_ISSUER, or pass your own Environment(...). Every URL must use HTTPS, except on localhost, so tokens and passwords never travel in clear text. API docs: prod · dev · local.

Authentication

Local and dev sign in at the dev identity server, prod at prod's. Without arguments the client checks the environment variables in that order, then a cached session, then falls back to browser sign-in.

PasswordLogin posts to the identity server's /auth/login/. For accounts with MFA, pass mfa_code= as a code or as a callable that takes the number of digits and returns the code. It refreshes with /auth/token/refresh/ and caches the session like browser sign-in, but never the password.

Every flow signs in at the environment's issuer, never at one a server names. OAuth flows read its endpoints from RFC 8414 metadata, which must name that issuer and use HTTPS, and bind tokens to Fetch with resource (RFC 8707). Browser sign-in uses authorization code + PKCE on http://127.0.0.1:8976/callback, registers a public client on first use (RFC 7591), and offers whatever SSO your tenant has configured. Sessions are cached in ~/.config/aic/credentials.json, replaced atomically as a mode-600 file. Pass on_url= to show the sign-in URL your own way. Expired tokens are refreshed, and a 401 triggers one renewal and retry.

Querying

• query(sql, fresh=False) runs one SELECT and returns a QueryResult with columns, rows, records(), to_pandas(), truncated, elapsed_ms, cache and request_id. fresh=True reads S3 as it is now, bypassing the index and cache.
• validate(sql) checks one SELECT as query would, without reading data, and returns a Validation with the canonical sql, the datasets it reads and the result columns. Invalid SQL raises QueryError, exactly as a query does.
• datasets(names=..., layers=..., limit=...) iterates every granted dataset across pages, or only the names and layers (gold, silver, bronze) you ask for; dataset(name) returns one. Dataset.ref is the name quoted for SQL, and file_count, row_count and byte_size size the partitions you may read (rows and bytes for gold only). limit (default 100, at most MAX_LIMIT = 500) is the page size: the server describes every dataset on a page within its timeout, so prefer layers=["gold"] over a large page. An unknown layer raises ValueError.
• sql(question, datasets=...) translates a question to SQL with ds-tools; ask(...) also runs it.
• tools.list(), tools.get(id) and tools.invoke(id, input) reach every ds-tools tool with the same sign-in.

validate, datasets and dataset are lookups: they have their own rate-limit budget, never spend query allowance, and take no concurrency slot.

Retries follow the API contract: connection failures, timeouts, 429, 500, 503 and 5xx responses without a JSON body are retried up to 3 attempts with exponential backoff, jitter and Retry-After. Everything else raises at once.

Errors

All errors derive from AICError.

Pass language="nb" for Norwegian error messages.

License

Apache 2.0