Skip to main content

SciDB

Database operations layer for SciStack. Provides abstractions for defining typed variables, configuring the database, and saving/loading data by metadata.

Named Filters (value-based, reusable)

Filter objects are first-class values — assign them to a variable and reuse them in any where= clause, or compose them with & / | / ~:

# Define once (this IS a ColumnFilter — no special wrapper needed)
clean_gr = GAITRiteLoadedCycle["StepLengths_GR"] != 0

# Use anywhere
for_each(mean_change_from_reference,
         inputs={"baseline": Fixed(GAITRiteLoadedCycle["StepLengths_GR"], session="BL"),
                 "value":    GAITRiteLoadedCycle["StepLengths_GR"]},
         outputs=[DeltaStepLength],
         where=clean_gr & (UAStartFoot() == "A"))

# Compose named filters
clean_and_unilateral = clean_gr & (UAStartFoot() == "U")
for_each(..., where=clean_and_unilateral)

MATLAB equivalent:

clean_gr = GAITRiteLoadedCycle("StepLengths_GR") ~= 0;
scidb.for_each(@meanChangeFromReference, ..., where=clean_gr & (UAStartFoot() == "A"));

Permanent Schema-Level Exclusions

For data that should be excluded from every analysis (e.g., a failed recording session), use the exclusion registry instead of per-call filters.

# Mark a specific trial as excluded (persisted in the database)
scidb.exclude_schema(subject=1, trial=2,
                     reason="equipment malfunction during recording")

# Exclude an entire subject (trial omitted = wildcard)
scidb.exclude_schema(subject=3, reason="participant withdrew")

# Inspect currently-excluded combinations
exclusions_df = scidb.list_exclusions()

# Re-include (logged; the original exclusion row is preserved)
scidb.include_schema(subject=1, trial=2,
                     reason="re-reviewed video, recording was valid")

MATLAB equivalent:

scidb.exclude_schema("equipment malfunction", 'subject', 1, 'trial', 2)
scidb.include_schema("re-reviewed, recording was valid", 'subject', 1, 'trial', 2)
tbl = scidb.list_exclusions()

Exclusions live in the __scidb_schema_overrides table (one row per change, with its reason) and are applied automatically by for_each before the iteration loop: an excluded location is not run. They are not part of any record or invocation identity.

from scidb import configure_database, BaseVariable
import numpy as np

db = configure_database("experiment.duckdb", ["subject", "session"])

class RawSignal(BaseVariable):
    schema_version = 1

RawSignal.save(np.array([1, 2, 3]), subject=1, session="A")
raw = RawSignal.load(subject=1, session="A")  # one match -> a BaseVariable; several -> a list
all_versions = list(RawSignal.load(subject=1, session="A", version="all"))

Data Inventory (missing schema-key combinations)

"Which combinations of these schema keys have no data?" The inventory crosses every value each chosen key has anywhere in _schema (in the declared [schema_keys] order) and gives each combination a status: present, missing, or excluded (an active exclude_schema matches it; excluded wins over present). A deeper row counts for its projection onto the chosen keys, and only the database is consulted, never a Variable.

from scidb.provenance.inventory import schema_inventory, write_inventory_csv

inv = schema_inventory(db, ["subject", "session"])   # None = every key
inv.counts            # {"present": 10, "missing": 2, "excluded": 0}
write_inventory_csv(inv, "missing.csv", missing_only=True)
scidb inventory --keys subject,session          # counts + missing table
scidb inventory --all --csv inventory.csv       # every combination + status
scidb inventory --within group                  # one inventory per study arm

within=["group"] (CLI --within group) runs the inventory separately for each group value and crosses only the values found in that arm, so a subject of one arm is never reported missing at the other arm's sessions. Use it for design keys (arm, site, protocol).

Above 1,000,000 combinations the call raises before building the product. The GUI's 📋 Inventory button shows the same table. Details: docs/claude/data-inventory.md.

Logging

Every scidb operation logs through the shared scistacklog facade. configure_database() points both log files beside the database file and writes a run-context header (package versions, Python version, pid), so each log file is self-describing:

  • journal (scistack.log): for the scientist. One block per run: what ran, the counts, how long, and each failure labelled with who can fix it.
  • debug file (scistack-debug.log): for whoever debugs SciStack. Everything, at DEBUG by default (named internal steps with durations, per-iteration [run]/[skip] lines, per-phase [timing] tables), with the originating layer ([scidb], [scifor], [matlab], …) and the run id on every line.
  • console (stderr): the pipeline narrative, at INFO.
from scidb import Log

Log.set_level("INFO", sink="file")   # a quieter debug file
Log.set_level("DEBUG")               # both sinks

or set the SCIDB_LOG_LEVEL environment variable, or pass -v to the scidb CLI.

Metadata

Release files for scistack-db 0.4.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scistack-db 0.4.3
File Size Uploaded Provenance Actions
scistack_db-0.4.3.tar.gz 636.6 kB With provenance Details for scistack_db-0.4.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scistack-db 0.4.3
File Interpreter ABI Platform Actions
scistack_db-0.4.3-py3-none-any.whl Python 3 none any Details for scistack_db-0.4.3-py3-none-any.whl

Total release size: 1.3 MB

Release files / scistack_db-0.4.3.tar.gz

Download URL scistack_db-0.4.3.tar.gz
Size 636.6 kB
Tags Source
SHA-256 checksum
How to use checksums
4c95738da9f9dce0186365b4c2abe625c4ea5a124e46502345420a8fe8008cf6
BLAKE2b-256 checksum
How to use checksums
4b6cf82ed7e5f17ab2741a6d1e2bbd3f5eecfcb7d893aeae7a75e67b6cdc8b60
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Release files / scistack_db-0.4.3-py3-none-any.whl

Download URL scistack_db-0.4.3-py3-none-any.whl
Size 700.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
d901ee059b675def9b754f4432ea1b1221dbaccb40b441062ebce5c0f9f24d59
BLAKE2b-256 checksum
How to use checksums
73bdee9032f4bef4c287ef7c0620a7f82c26dec2c405c56382e670deac04113b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Security for scistack-db 0.4.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for scistack_db-0.4.3.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.4.3 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page