Skip to main content

SciDuck

The DuckDB layer under scidb. It owns two things: the codec that turns Python values into queryable DuckDB columns and back, and a locked connection that is safe to share across threads. It creates no tables: scidb owns every table of a project database (scidb.store.tables) and the schema-location ids (scidb.store.sciduck.SciDuck, moved out of sciduckdb in P3). Application code reaches the database only through scidb.

All data, arrays included, is stored in queryable DuckDB types (LIST, nested LIST, JSON), so the database can be inspected with DBeaver or any DuckDB-compatible viewer.

Codec

from sciduckdb import infer_data_columns, value_to_storage_row, restore_types, storage_to_python

col_types, dtype_meta = infer_data_columns(value)   # {column: DuckDB type}, restore metadata
row = value_to_storage_row(value, dtype_meta)        # one row of storage values
# ... after reading the rows back into a DataFrame df:
df = restore_types(df, dtype_meta)                   # Python types restored per column
  • Scalars, numpy arrays (any dtype and shape), lists, and dicts (flat or nested; nested keys become dot-separated columns) store one row per value.
  • A DataFrame value stores one DuckDB row per DataFrame row (dataframe_to_storage_rows), one column per DataFrame column.
  • dtype_meta records what each column was, so a load is lossless: dtypes, shapes and NULL list elements (restored as NaN) survive the round trip.

Connection

DuckConnection(path) opens the file. Every statement runs under one lock, so a concurrent close() cannot land mid-statement. Group paired work with with duck.transaction("label"):; it is re-entrant on the owning thread and rolls back on any exception.

The supported interface of a connection (DuckConnection, which scidb's SciDuck extends). Other packages call these and nothing with a leading underscore:

Method Does
transaction(label) context manager: one transaction around the block
execute(sql, params) a statement that returns no rows; returns None
fetchall / fetchone rows as tuples (few rows) / the first row or None
fetchdf rows as a DataFrame (many rows; NULL floats become NaN)
fetch_table (columns, rows) with NULL kept as None (diagnostics)
fetchdf_if_open fetchdf, or None when the connection is closed
table_exists(name) whether a table or view exists
bulk_insert / bulk_update vectorized multi-row writes (one statement)
add_rollback_listener(fn) call fn() after every outermost ROLLBACK
query(sql) fetchdf under its user-facing name
is_open, close(), reopen() connection lifetime; also a context manager

Every fetch holds the lock across execute and fetch. execute returns nothing so a result can never be fetched after the lock is released.

Modules: codec.py (conversion functions), connection.py (DuckConnection), datetimes.py. Import from sciduckdb, not from the modules.

Schema ids

Moved to scidb (scidb.store.sciduck.SciDuck): _schema, _variables and _variable_groups are project tables like any other.

Datetimes

A datetime value (datetime.datetime, pd.Timestamp, np.datetime64, NaT) is stored as two columns: <col> TIMESTAMP_NS holding the UTC instant (or a naive value's wall clock), and <col>__tz VARCHAR holding that value's zone ("Europe/Berlin", "UTC", "+02:00"; NULL = naive). Zones are per value, so one column may hold several zones and naive values. Loads give back each value in its own zone. The __tz column exists from the column's first save and is never listed in the variable's data columns. In raw SQL, filter on <col>__tz IS NULL to tell naive rows apart. Owner: sciduckdb.datetimes; design: docs/claude/datetime-roundtrip.md.

Metadata

Release files for sciduckdb 0.3.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sciduckdb 0.3.6
File Size Uploaded Provenance Actions
sciduckdb-0.3.6.tar.gz 28.5 kB With provenance Details for sciduckdb-0.3.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sciduckdb 0.3.6
File Interpreter ABI Platform Actions
sciduckdb-0.3.6-py3-none-any.whl Python 3 none any Details for sciduckdb-0.3.6-py3-none-any.whl

Total release size: 57.7 kB

Release files / sciduckdb-0.3.6.tar.gz

Download URL sciduckdb-0.3.6.tar.gz
Size 28.5 kB
Tags Source
SHA-256 checksum
How to use checksums
4a6d4b7fd28b36daf342b597636f8f1bd2b724cacc14ffcd4eb743babc2ed83e
BLAKE2b-256 checksum
How to use checksums
bb98ba8d6e4f59b94563db552c1350cd3ab4e907bf6f822f6d34425ffcaebc3e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Release files / sciduckdb-0.3.6-py3-none-any.whl

Download URL sciduckdb-0.3.6-py3-none-any.whl
Size 29.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1e904e5926232d08a46ca07491929bcfd3c19abc0214652e9249c750198c03f3
BLAKE2b-256 checksum
How to use checksums
291a62f42a521951a395c6b7c12e4196a14c031d8072a5c6eb5bb1d1f307ad40
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Security for sciduckdb 0.3.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for sciduckdb-0.3.6.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.3.6 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page