hotdata-ibis
Use Ibis to create on-demand databases, upload data, and query with Python expressions — get pandas or Arrow results back without writing SQL.
Requirements: Python 3.10+, ibis-framework ≥12,<13, hotdata ≥0.9,<0.10.
Install
pip install hotdata-ibis
# or: uv pip install hotdata-ibis
Quickstart: create a database and query it
import time
import pandas as pd
import ibis
con = ibis.hotdata.connect(
api_url="https://api.hotdata.dev",
token="YOUR_API_KEY",
workspace_id="ws_...",
)
# 1. Create a database and declare the tables you'll load.
# Hotdata database names are not unique — create_database returns the id
# you'll use for every subsequent operation on this database.
database_id = con.create_database("sales", tables=["orders"])
# 2. Upload a pandas DataFrame (or PyArrow table)
df = pd.DataFrame({
"order_id": [1, 2, 3],
"amount": [9.99, 49.99, 5.00],
"region": ["west", "east", "west"],
})
con.create_table("orders", df, database=(database_id, "main"), overwrite=True)
# 3. Uploads are async — wait briefly before querying
time.sleep(2)
# 4. Query with Ibis expressions
# Managed tables are always accessed with catalog "default"
t = con.table("orders", database=("default", "main"))
result = (
t.group_by("region")
.agg(total=t.amount.sum())
.order_by(ibis.desc("total"))
.execute() # returns a pandas DataFrame
)
# 5. Clean up
con.drop_table("orders", database=(database_id, "main"))
con.drop_database(database_id)
Connect
con = ibis.hotdata.connect(
api_url="https://api.hotdata.dev",
token="YOUR_API_KEY",
workspace_id="ws_...",
# optional
timeout=120.0, # per-request HTTP timeout in seconds
verify_ssl=True, # False to skip TLS verification, or path to CA bundle
default_connection=None, # default catalog (connection id); auto-detected if only one exists
default_schema=None, # default schema; auto-detected if only one exists
database_id=None, # bind an existing instant database id at connect time
async_after_ms=1000, # queries finishing within this return rows in one request;
# slower ones go async. None = always async
poll_interval_s=0.25, # polling interval for async queries
poll_timeout_s=600.0, # max time to wait for a query result
)
api_url, token and workspace_id fall back to the HOTDATA_API_URL,
HOTDATA_API_KEY and HOTDATA_WORKSPACE environment variables (same as the
Hotdata CLI), so with those set this is enough:
con = ibis.hotdata.connect()
URL-style also works, with the same parameters as query string keys:
con = ibis.connect(
"hotdata://api.hotdata.dev/"
"?token=...&workspace_id=ws_..."
"&default_connection=my_conn&default_schema=public"
)
Instant databases
Instant databases are the primary way to bring data into Hotdata with Ibis. Declare a database and its tables, upload data, and query immediately.
Create and load
# Declare the database and all table names up front.
# Hotdata database names are not unique — create_database returns the id
# you'll use for every subsequent operation on this database.
database_id = con.create_database("analytics", tables=["events", "users"])
# Upload from a pandas DataFrame
con.create_table("events", events_df, database=(database_id, "main"), overwrite=True)
# PyArrow tables also work
import pyarrow as pa
table = pa.table({"id": [1, 2], "name": ["alice", "bob"]})
con.create_table("users", table, database=(database_id, "main"), overwrite=True)
# Schema-only (no data): creates an empty table with the declared schema
import ibis.expr.schema as sch
con.create_table(
"staging",
schema=sch.Schema({"id": "int64", "ts": "timestamp"}),
database=(database_id, "main"),
)
Declaring table names in tables= is optional — loading into a table that was not declared creates it automatically as part of the load.
Query
When querying, use "default" as the catalog:
t = con.table("events", database=("default", "main"))
result = (
t.filter(t.event_type == "click")
.group_by("user_id")
.agg(n=t.count())
.execute()
)
Or with raw SQL:
result = con.sql(
'SELECT user_id, COUNT(*) AS n '
'FROM "default"."main"."events" '
'WHERE event_type = \'click\' '
'GROUP BY user_id'
).execute()
Delete
Pass force=True to silently skip errors when the database or table does not exist:
con.drop_table("events", database=(database_id, "main"))
con.drop_table("events", database=(database_id, "main"), force=True) # no-op if missing
con.drop_database(database_id)
con.drop_database(database_id, force=True) # no-op if missing
Addressing summary
| Operation | database= argument |
|---|---|
create_table / drop_table |
(database_id, schema) — the id returned by create_database, not its display name |
drop_database |
the id returned by create_database, not its display name |
con.table(...) when querying |
("default", schema) |
Querying
Ibis expressions
t = con.table("orders", database=("default", "main"))
summary = (
t.filter(t.amount > 10)
.group_by("region")
.agg(total=t.amount.sum(), n=t.count())
.order_by(ibis.desc("total"))
.execute()
)
.execute() returns a pandas DataFrame. .to_pyarrow() returns an Arrow table. .to_pyarrow_batches() returns a RecordBatchReader — note that Hotdata returns a single Arrow IPC payload per query, so this method downloads the full result first and then splits it into local batches.
Raw SQL
base = con.sql(
'SELECT * FROM "default"."main"."orders"',
dialect="postgres",
)
result = base.filter(base.amount > 10).execute()
You can chain Ibis expressions on the result of con.sql(...).
Vector search
ibis_hotdata.vector provides helpers for querying HNSW-indexed vector (embedding)
columns:
from ibis_hotdata.vector import semantic_search, l2_distance
t = con.table("docs", database=("default", "main"))
result = semantic_search(t, "embedding", query_vector, k=10).execute()
# or with a different metric
result = semantic_search(t, "embedding", query_vector, k=10, distance_fn=l2_distance).execute()
semantic_search compiles to ORDER BY <distance>(col, ARRAY[...]) ASC LIMIT k with the
vector column excluded from the output — the SQL shape Hotdata's query engine requires to
route the query through its HNSW index instead of a brute-force scan.
Text queries
For a text column indexed with an embedding provider, pass the query as a string.
Hotdata embeds it with the index's provider (vector_distance(col, 'text')):
result = semantic_search(t, "body", "how do refunds work?", k=10).execute()
The text column stays in the output; vector (array) columns such as the generated
body_embedding are left out.
Managing indexes
Create, list and drop indexes on instant-database tables (database= works as in
create_table):
db = (db_id, "main")
# HNSW (default): fastest search, memory grows with the table
con.create_index("docs", "embedding", name="docs_idx", database=db, metric="cosine")
# IVF: for tables too large for HNSW; tune recall/speed with probe_fraction
con.create_index(
"docs", "embedding", name="docs_idx", database=db,
algorithm="ivf", metric="cosine", probe_fraction=0.1,
)
# Hotdata generates the embeddings from a text column
con.create_index(
"docs", "body", name="docs_body_idx", database=db,
embedding_provider_id="ep_...",
)
con.list_indexes("docs", database=db)
con.drop_index("docs", "docs_idx", database=db)
index_type can also be "bm25" (full-text) or "sorted". The index builds within
one request, so raise timeout= on connect for large tables.
Connecting to existing sources
If you have existing databases or warehouses connected to your Hotdata workspace (Postgres, Snowflake, BigQuery, etc.), you can query them through the same Ibis connection:
con = ibis.hotdata.connect(
api_url="https://api.hotdata.dev",
token="YOUR_API_KEY",
workspace_id="ws_...",
default_connection="<connection_id>", # a connection id, from con.list_catalogs()
default_schema="public",
)
t = con.table("orders") # resolves to <connection_id>.public.orders
Discover what's available. Catalogs are connection ids — the id field of the
workspace's connections, as returned by list_catalogs() — not display names:
con.list_catalogs() # connection ids
conn_id = con.list_catalogs()[0]
con.list_databases(catalog=conn_id) # schemas
con.list_tables(database=(conn_id, "public")) # tables
What's supported
| Feature | Status |
|---|---|
create_database / drop_database (managed) |
✅ |
create_table from pandas / PyArrow / schema-only |
✅ |
drop_table |
✅ |
create_index / list_indexes / drop_index (vector HNSW/IVF, BM25, sorted) |
✅ |
con.table(...) with full schema metadata |
✅ |
| Ibis expressions: filter, select, join, group_by, agg, order_by, limit | ✅ |
con.sql(...) raw SQL |
✅ |
.execute() → pandas, .to_pyarrow(), .to_pyarrow_batches() |
✅ |
list_catalogs, list_databases, list_tables |
✅ |
| Arrow / Parquet column types (timestamp, decimal, list, duration, …) | ✅ |
Vector search (ibis_hotdata.vector.semantic_search, vector or text query) |
✅ |
| Temporary tables | ❌ |
In-memory tables (ibis.memtable(...)) |
❌ |
| Python UDFs | ❌ |
| INSERT / UPDATE / DELETE on external connections | ❌ |
SQL compilation uses Ibis's Postgres dialect. Column types returned by Hotdata's information schema are resolved via PyArrow's type system, so Parquet-loaded tables with Arrow-native types (timestamps with time zones, decimals, lists, durations) are mapped correctly to Ibis types.
Development
uv sync # installs dev group (pytest, ruff, httpx)
uv run pytest
uv run ruff check src tests examples
CI: uv sync --all-groups && uv run pytest -v.
Examples
Set your credentials, then run any example script:
export HOTDATA_API_KEY=...
export HOTDATA_WORKSPACE=...
uv run python examples/01_catalog_introspection.py
uv run python examples/02_execute_sql.py 'SELECT COUNT(*) AS n FROM tpch.tpch_sf1.customer'
uv run python examples/03_connect_via_url.py
uv run python examples/04_ibis_table_workflows.py
uv run python examples/05_roundtrip_demo.py
uv run python examples/06_semantic_search.py
References
Metadata
Release files for hotdata-ibis 0.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| hotdata_ibis-0.7.0.tar.gz | 33.3 kB | With provenance | Details for hotdata_ibis-0.7.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| hotdata_ibis-0.7.0-py3-none-any.whl | Python 3 | none | any | Details for hotdata_ibis-0.7.0-py3-none-any.whl |
Total release size: 55.8 kB
Release files / hotdata_ibis-0.7.0.tar.gz
| Download URL | hotdata_ibis-0.7.0.tar.gz |
|---|---|
| Size | 33.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
883f28e29ad34bac5135fdf20f4709ef12e7985619f91cbf80dc03d9ff17ad27
|
|
BLAKE2b-256 checksum How to use checksums |
5e30bfb75caec385f7571678749f2b2b130612e39e2d8a40372cee17c0401723
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / hotdata_ibis-0.7.0-py3-none-any.whl
| Download URL | hotdata_ibis-0.7.0-py3-none-any.whl |
|---|---|
| Size | 22.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7663bee581da94d6aea870fdb748bc27df5cc2d967124bd6b85d2fc9d494566a
|
|
BLAKE2b-256 checksum How to use checksums |
c80eab33ccf1d6740b54b32a6c8e49e45030bb3bd3ba5fe0273c6b26f0b864e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logSecurity for hotdata-ibis 0.7.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for hotdata_ibis-0.7.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.