Skip to main content

SciDuck

The DuckDB layer under scidb. It owns two things: the codec that turns Python values into queryable DuckDB columns and back, and a locked connection that is safe to share across threads. It creates no tables: scidb owns every table of a project database (scidb.store.tables) and the schema-location ids (scidb.store.sciduck.SciDuck, moved out of sciduckdb in P3). Application code reaches the database only through scidb.

All data, arrays included, is stored in queryable DuckDB types (LIST, nested LIST, JSON), so the database can be inspected with DBeaver or any DuckDB-compatible viewer.

Codec

from sciduckdb import infer_data_columns, value_to_storage_row, restore_types, storage_to_python

col_types, dtype_meta = infer_data_columns(value)   # {column: DuckDB type}, restore metadata
row = value_to_storage_row(value, dtype_meta)        # one row of storage values
# ... after reading the rows back into a DataFrame df:
df = restore_types(df, dtype_meta)                   # Python types restored per column
  • Scalars, numpy arrays (any dtype and shape), lists, and dicts (flat or nested; nested keys become dot-separated columns) store one row per value.
  • A DataFrame value stores one DuckDB row per DataFrame row (dataframe_to_storage_rows), one column per DataFrame column.
  • dtype_meta records what each column was, so a load is lossless: dtypes, shapes and NULL list elements (restored as NaN) survive the round trip.

Connection

DuckConnection(path) opens the file. Every statement runs under one lock, so a concurrent close() cannot land mid-statement. Group paired work with with duck.transaction("label"):; it is re-entrant on the owning thread and rolls back on any exception.

The supported interface of a connection (DuckConnection, which scidb's SciDuck extends). Other packages call these and nothing with a leading underscore:

Method Does
transaction(label) context manager: one transaction around the block
execute(sql, params) a statement that returns no rows; returns None
fetchall / fetchone rows as tuples (few rows) / the first row or None
fetchdf rows as a DataFrame (many rows; NULL floats become NaN)
fetch_table (columns, rows) with NULL kept as None (diagnostics)
fetchdf_if_open fetchdf, or None when the connection is closed
table_exists(name) whether a table or view exists
bulk_insert / bulk_update vectorized multi-row writes (one statement)
add_rollback_listener(fn) call fn() after every outermost ROLLBACK
query(sql) fetchdf under its user-facing name
is_open, close(), reopen() connection lifetime; also a context manager

Every fetch holds the lock across execute and fetch. execute returns nothing so a result can never be fetched after the lock is released.

Modules: codec.py (conversion functions), connection.py (DuckConnection), datetimes.py. Import from sciduckdb, not from the modules.

Schema ids

Moved to scidb (scidb.store.sciduck.SciDuck): _schema, _variables and _variable_groups are project tables like any other.

Datetimes

A datetime value (datetime.datetime, pd.Timestamp, np.datetime64, NaT) is stored as two columns: <col> TIMESTAMP_NS holding the UTC instant (or a naive value's wall clock), and <col>__tz VARCHAR holding that value's zone ("Europe/Berlin", "UTC", "+02:00"; NULL = naive). Zones are per value, so one column may hold several zones and naive values. Loads give back each value in its own zone. The __tz column exists from the column's first save and is never listed in the variable's data columns. In raw SQL, filter on <col>__tz IS NULL to tell naive rows apart. Owner: sciduckdb.datetimes; design: docs/claude/datetime-roundtrip.md.

Metadata

Release files for sciduckdb 0.4.8

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sciduckdb 0.4.8
File Size Uploaded Provenance Actions
sciduckdb-0.4.8.tar.gz 28.5 kB With provenance Details for sciduckdb-0.4.8.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sciduckdb 0.4.8
File Interpreter ABI Platform Actions
sciduckdb-0.4.8-py3-none-any.whl Python 3 none any Details for sciduckdb-0.4.8-py3-none-any.whl

Total release size: 57.7 kB

Release files / sciduckdb-0.4.8.tar.gz

Download URL sciduckdb-0.4.8.tar.gz
Size 28.5 kB
Tags Source
SHA-256 checksum
How to use checksums
874f81458a94d8fb79ef3a2b3c8d037125e22fb773fdef93f2de4b0512e6476c
BLAKE2b-256 checksum
How to use checksums
2181409fabcbdf36dd79c72a034c0372cb30ae775527a10d87779200bfcf906e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Release files / sciduckdb-0.4.8-py3-none-any.whl

Download URL sciduckdb-0.4.8-py3-none-any.whl
Size 29.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
055a97ee81210eb33de11196de9d28718620b223245a58db49a486fa08702e6a
BLAKE2b-256 checksum
How to use checksums
81c0e437a1201236ab9cdcc7a1a807fc76bb95a82c178f0ace4a93af0aadc66e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Security for sciduckdb 0.4.8

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for sciduckdb-0.4.8.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.4.8 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page