Skip to main content

SciDuck

The DuckDB layer under scidb. It owns two things: the codec that turns Python values into queryable DuckDB columns and back, and a locked connection that is safe to share across threads. It creates no tables: scidb owns every table of a project database (scidb.store.tables) and the schema-location ids (scidb.store.sciduck.SciDuck, moved out of sciduckdb in P3). Application code reaches the database only through scidb.

All data, arrays included, is stored in queryable DuckDB types (LIST, nested LIST, JSON), so the database can be inspected with DBeaver or any DuckDB-compatible viewer.

Codec

from sciduckdb import infer_data_columns, value_to_storage_row, restore_types, storage_to_python

col_types, dtype_meta = infer_data_columns(value)   # {column: DuckDB type}, restore metadata
row = value_to_storage_row(value, dtype_meta)        # one row of storage values
# ... after reading the rows back into a DataFrame df:
df = restore_types(df, dtype_meta)                   # Python types restored per column
  • Scalars, numpy arrays (any dtype and shape), lists, and dicts (flat or nested; nested keys become dot-separated columns) store one row per value.
  • A DataFrame value stores one DuckDB row per DataFrame row (dataframe_to_storage_rows), one column per DataFrame column.
  • dtype_meta records what each column was, so a load is lossless: dtypes, shapes and NULL list elements (restored as NaN) survive the round trip.

Connection

DuckConnection(path) opens the file. Every statement runs under one lock, so a concurrent close() cannot land mid-statement. Group paired work with with duck.transaction("label"):; it is re-entrant on the owning thread and rolls back on any exception.

The supported interface of a connection (DuckConnection, which scidb's SciDuck extends). Other packages call these and nothing with a leading underscore:

Method Does
transaction(label) context manager: one transaction around the block
execute(sql, params) a statement that returns no rows; returns None
fetchall / fetchone rows as tuples (few rows) / the first row or None
fetchdf rows as a DataFrame (many rows; NULL floats become NaN)
fetch_table (columns, rows) with NULL kept as None (diagnostics)
fetchdf_if_open fetchdf, or None when the connection is closed
table_exists(name) whether a table or view exists
bulk_insert / bulk_update vectorized multi-row writes (one statement)
add_rollback_listener(fn) call fn() after every outermost ROLLBACK
query(sql) fetchdf under its user-facing name
is_open, close(), reopen() connection lifetime; also a context manager

Every fetch holds the lock across execute and fetch. execute returns nothing so a result can never be fetched after the lock is released.

Modules: codec.py (conversion functions), connection.py (DuckConnection), datetimes.py, row_lists.py. Import from sciduckdb, not from the modules.

Schema ids

Moved to scidb (scidb.store.sciduck.SciDuck): _schema, _variables and _variable_groups are project tables like any other.

Datetimes

A datetime value (datetime.datetime, pd.Timestamp, np.datetime64, NaT) is stored as two columns: <col> TIMESTAMP_NS holding the UTC instant (or a naive value's wall clock), and <col>__tz VARCHAR holding that value's zone ("Europe/Berlin", "UTC", "+02:00"; NULL = naive). Zones are per value, so one column may hold several zones and naive values. Loads give back each value in its own zone. The __tz column exists from the column's first save and is never listed in the variable's data columns. In raw SQL, filter on <col>__tz IS NULL to tell naive rows apart. Owner: sciduckdb.datetimes; design: docs/claude/datetime-roundtrip.md.

Metadata

Release files for sciduckdb 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sciduckdb 0.2.2
File Size Uploaded Provenance Actions
sciduckdb-0.2.2.tar.gz 32.5 kB With provenance Details for sciduckdb-0.2.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sciduckdb 0.2.2
File Interpreter ABI Platform Actions
sciduckdb-0.2.2-py3-none-any.whl Python 3 none any Details for sciduckdb-0.2.2-py3-none-any.whl

Total release size: 66.6 kB

Release files / sciduckdb-0.2.2.tar.gz

Download URL sciduckdb-0.2.2.tar.gz
Size 32.5 kB
Tags Source
SHA-256 checksum
How to use checksums
5637fc8a4a20411b090f3c41151b5463a9c0619d0fe6a4633e18e2f33aa85c28
BLAKE2b-256 checksum
How to use checksums
ee96668d5f2c01e9acc931be658677d449b67a48c4935a1884b4e24bded0d9ca
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

Release files / sciduckdb-0.2.2-py3-none-any.whl

Download URL sciduckdb-0.2.2-py3-none-any.whl
Size 34.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
388bdf9d934dc7dfde9f796cf6ee54485a3e97aa571ac5f87b068c5cf9a7f2b1
BLAKE2b-256 checksum
How to use checksums
da1a13ce7a5f91e9deb65809ac952035bc945fef0c5eb8df7df959e814d0b0fe
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

Security for sciduckdb 0.2.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for sciduckdb-0.2.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.2 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page