Skip to main content

SciDB

Database operations layer for SciStack. Provides abstractions for defining typed variables, configuring the database, and saving/loading data by metadata.

Named Filters (value-based, reusable)

Filter objects are first-class values — assign them to a variable and reuse them in any where= clause, or compose them with & / | / ~:

# Define once (this IS a ColumnFilter — no special wrapper needed)
clean_gr = GAITRiteLoadedCycle["StepLengths_GR"] != 0

# Use anywhere
for_each(mean_change_from_reference,
         inputs={"baseline": Fixed(GAITRiteLoadedCycle["StepLengths_GR"], session="BL"),
                 "value":    GAITRiteLoadedCycle["StepLengths_GR"]},
         outputs=[DeltaStepLength],
         where=clean_gr & (UAStartFoot() == "A"))

# Compose named filters
clean_and_unilateral = clean_gr & (UAStartFoot() == "U")
for_each(..., where=clean_and_unilateral)

MATLAB equivalent:

clean_gr = GAITRiteLoadedCycle("StepLengths_GR") ~= 0;
scidb.for_each(@meanChangeFromReference, ..., where=clean_gr & (UAStartFoot() == "A"));

Permanent Schema-Level Exclusions

For data that should be excluded from every analysis (e.g., a failed recording session), use the exclusion registry instead of per-call filters.

# Mark a specific trial as excluded (persisted in the database)
scidb.exclude_schema(subject=1, trial=2,
                     reason="equipment malfunction during recording")

# Exclude an entire subject (trial omitted = wildcard)
scidb.exclude_schema(subject=3, reason="participant withdrew")

# Inspect currently-excluded combinations
exclusions_df = scidb.list_exclusions()

# Re-include (logged; the original exclusion row is preserved)
scidb.include_schema(subject=1, trial=2,
                     reason="re-reviewed video, recording was valid")

MATLAB equivalent:

scidb.exclude_schema("equipment malfunction", 'subject', 1, 'trial', 2)
scidb.include_schema("re-reviewed, recording was valid", 'subject', 1, 'trial', 2)
tbl = scidb.list_exclusions()

Exclusions live in the __scidb_schema_overrides table (one row per change, with its reason) and are applied automatically by for_each before the iteration loop: an excluded location is not run. They are not part of any record or invocation identity.

from scidb import configure_database, BaseVariable
import numpy as np

db = configure_database("experiment.duckdb", ["subject", "session"])

class RawSignal(BaseVariable):
    schema_version = 1

RawSignal.save(np.array([1, 2, 3]), subject=1, session="A")
raw = RawSignal.load(subject=1, session="A")  # one match -> a BaseVariable; several -> a list
all_versions = list(RawSignal.load(subject=1, session="A", version="all"))

Data Inventory (missing schema-key combinations)

"Which combinations of these schema keys have no data?" The inventory crosses every value each chosen key has anywhere in _schema (in the declared [schema_keys] order) and gives each combination a status: present, missing, or excluded (an active exclude_schema matches it; excluded wins over present). A deeper row counts for its projection onto the chosen keys, and only the database is consulted, never a Variable.

from scidb.provenance.inventory import schema_inventory, write_inventory_csv

inv = schema_inventory(db, ["subject", "session"])   # None = every key
inv.counts            # {"present": 10, "missing": 2, "excluded": 0}
write_inventory_csv(inv, "missing.csv", missing_only=True)
scidb inventory --keys subject,session          # counts + missing table
scidb inventory --all --csv inventory.csv       # every combination + status
scidb inventory --within group                  # one inventory per study arm

within=["group"] (CLI --within group) runs the inventory separately for each group value and crosses only the values found in that arm, so a subject of one arm is never reported missing at the other arm's sessions. Use it for design keys (arm, site, protocol).

Above 1,000,000 combinations the call raises before building the product. The GUI's 📋 Inventory button shows the same table. Details: docs/claude/data-inventory.md.

Logging

Every scidb operation logs through the shared scistacklog facade. configure_database() points both log files beside the database file and writes a run-context header (package versions, Python version, pid), so each log file is self-describing:

  • journal (scistack.log): for the scientist. One block per run: what ran, the counts, how long, and each failure labelled with who can fix it.
  • debug file (scistack-debug.log): for whoever debugs SciStack. Everything, at DEBUG by default (named internal steps with durations, per-iteration [run]/[skip] lines, per-phase [timing] tables), with the originating layer ([scidb], [scifor], [matlab], …) and the run id on every line.
  • console (stderr): the pipeline narrative, at INFO.
from scidb import Log

Log.set_level("INFO", sink="file")   # a quieter debug file
Log.set_level("DEBUG")               # both sinks

or set the SCIDB_LOG_LEVEL environment variable, or pass -v to the scidb CLI.

Metadata

Release files for scistack-db 0.4.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scistack-db 0.4.2
File Size Uploaded Provenance Actions
scistack_db-0.4.2.tar.gz 635.0 kB With provenance Details for scistack_db-0.4.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scistack-db 0.4.2
File Interpreter ABI Platform Actions
scistack_db-0.4.2-py3-none-any.whl Python 3 none any Details for scistack_db-0.4.2-py3-none-any.whl

Total release size: 1.3 MB

Release files / scistack_db-0.4.2.tar.gz

Download URL scistack_db-0.4.2.tar.gz
Size 635.0 kB
Tags Source
SHA-256 checksum
How to use checksums
b9b1ec4287a125a410dab8f1913d2e08c20f98fc7f828ff6650a93cec94eb496
BLAKE2b-256 checksum
How to use checksums
4a5c45938e9c285c8244a73bd0bfd23d1c669a8aab7b01aa41c023d69e37d18a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / scistack_db-0.4.2-py3-none-any.whl

Download URL scistack_db-0.4.2-py3-none-any.whl
Size 698.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b0a074f04f3040be85cc5b090ecec4b5fc93c36d729c0a5e3d8c38266e923afd
BLAKE2b-256 checksum
How to use checksums
0008dacd90850669e3abfa1cd1fc061b164a4ba68d074c1f117511917b77362e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Security for scistack-db 0.4.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for scistack_db-0.4.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.4.2 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page