Skip to main content

SciDuck

The DuckDB layer under scidb. It owns two things: the codec that turns Python values into queryable DuckDB columns and back, and a locked connection that is safe to share across threads. It creates no tables: scidb owns every table of a project database (scidb.store.tables) and the schema-location ids (scidb.store.sciduck.SciDuck, moved out of sciduckdb in P3). Application code reaches the database only through scidb.

All data, arrays included, is stored in queryable DuckDB types (LIST, nested LIST, JSON), so the database can be inspected with DBeaver or any DuckDB-compatible viewer.

Codec

from sciduckdb import infer_data_columns, value_to_storage_row, restore_types, storage_to_python

col_types, dtype_meta = infer_data_columns(value)   # {column: DuckDB type}, restore metadata
row = value_to_storage_row(value, dtype_meta)        # one row of storage values
# ... after reading the rows back into a DataFrame df:
df = restore_types(df, dtype_meta)                   # Python types restored per column
  • Scalars, numpy arrays (any dtype and shape), lists, and dicts (flat or nested; nested keys become dot-separated columns) store one row per value.
  • A DataFrame value stores one DuckDB row per DataFrame row (dataframe_to_storage_rows), one column per DataFrame column.
  • dtype_meta records what each column was, so a load is lossless: dtypes, shapes and NULL list elements (restored as NaN) survive the round trip.

Connection

DuckConnection(path) opens the file. Every statement runs under one lock, so a concurrent close() cannot land mid-statement. Group paired work with with duck.transaction("label"):; it is re-entrant on the owning thread and rolls back on any exception.

The supported interface of a connection (DuckConnection, which scidb's SciDuck extends). Other packages call these and nothing with a leading underscore:

Method Does
transaction(label) context manager: one transaction around the block
execute(sql, params) a statement that returns no rows; returns None
fetchall / fetchone rows as tuples (few rows) / the first row or None
fetchdf rows as a DataFrame (many rows; NULL floats become NaN)
fetch_table (columns, rows) with NULL kept as None (diagnostics)
fetchdf_if_open fetchdf, or None when the connection is closed
table_exists(name) whether a table or view exists
bulk_insert / bulk_update vectorized multi-row writes (one statement)
add_rollback_listener(fn) call fn() after every outermost ROLLBACK
query(sql) fetchdf under its user-facing name
is_open, close(), reopen() connection lifetime; also a context manager

Every fetch holds the lock across execute and fetch. execute returns nothing so a result can never be fetched after the lock is released.

Modules: codec.py (conversion functions), connection.py (DuckConnection), datetimes.py, row_lists.py. Import from sciduckdb, not from the modules.

Schema ids

Moved to scidb (scidb.store.sciduck.SciDuck): _schema, _variables and _variable_groups are project tables like any other.

Datetimes

A datetime value (datetime.datetime, pd.Timestamp, np.datetime64, NaT) is stored as two columns: <col> TIMESTAMP_NS holding the UTC instant (or a naive value's wall clock), and <col>__tz VARCHAR holding that value's zone ("Europe/Berlin", "UTC", "+02:00"; NULL = naive). Zones are per value, so one column may hold several zones and naive values. Loads give back each value in its own zone. The __tz column exists from the column's first save and is never listed in the variable's data columns. In raw SQL, filter on <col>__tz IS NULL to tell naive rows apart. Owner: sciduckdb.datetimes; design: docs/claude/datetime-roundtrip.md.

Metadata

Release files for sciduckdb 0.2.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sciduckdb 0.2.5
File Size Uploaded Provenance Actions
sciduckdb-0.2.5.tar.gz 32.5 kB With provenance Details for sciduckdb-0.2.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sciduckdb 0.2.5
File Interpreter ABI Platform Actions
sciduckdb-0.2.5-py3-none-any.whl Python 3 none any Details for sciduckdb-0.2.5-py3-none-any.whl

Total release size: 66.6 kB

Release files / sciduckdb-0.2.5.tar.gz

Download URL sciduckdb-0.2.5.tar.gz
Size 32.5 kB
Tags Source
SHA-256 checksum
How to use checksums
019bbd6c52e90638aeec3507e7689acbbe295315c7b9e62231352fe9f80215b8
BLAKE2b-256 checksum
How to use checksums
9939cb82ba3d63f4f9ce18f1f7c24256a8cb95c64ffe80ef905eb83545ec0e4f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.

Transparency log

Release files / sciduckdb-0.2.5-py3-none-any.whl

Download URL sciduckdb-0.2.5-py3-none-any.whl
Size 34.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
daf23235056f535752d2c33a3250e690a62eeccba4da2e824ee6fb80cbca8e7e
BLAKE2b-256 checksum
How to use checksums
63ae912301c58bae25dc390c417f1777d5c8f39b4f1ac69ee168d9a555a60c63
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.

Transparency log

Security for sciduckdb 0.2.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for sciduckdb-0.2.5.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.5 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page