Skip to main content

SciDB

Database operations layer for SciStack. Provides abstractions for defining typed variables, configuring the database, and saving/loading data by metadata.

Named Filters (value-based, reusable)

Filter objects are first-class values — assign them to a variable and reuse them in any where= clause, or compose them with & / | / ~:

# Define once (this IS a ColumnFilter — no special wrapper needed)
clean_gr = GAITRiteLoadedCycle["StepLengths_GR"] != 0

# Use anywhere
for_each(mean_change_from_reference,
         inputs={"baseline": Fixed(GAITRiteLoadedCycle["StepLengths_GR"], session="BL"),
                 "value":    GAITRiteLoadedCycle["StepLengths_GR"]},
         outputs=[DeltaStepLength],
         where=clean_gr & (UAStartFoot() == "A"))

# Compose named filters
clean_and_unilateral = clean_gr & (UAStartFoot() == "U")
for_each(..., where=clean_and_unilateral)

MATLAB equivalent:

clean_gr = GAITRiteLoadedCycle("StepLengths_GR") ~= 0;
scidb.for_each(@meanChangeFromReference, ..., where=clean_gr & (UAStartFoot() == "A"));

Permanent Schema-Level Exclusions

For data that should be excluded from every analysis (e.g., a failed recording session), use the exclusion registry instead of per-call filters.

# Mark a specific trial as excluded (persisted in the database)
scidb.exclude_schema(subject=1, trial=2,
                     reason="equipment malfunction during recording")

# Exclude an entire subject (trial omitted = wildcard)
scidb.exclude_schema(subject=3, reason="participant withdrew")

# Inspect currently-excluded combinations
exclusions_df = scidb.list_exclusions()

# Re-include (logged; the original exclusion row is preserved)
scidb.include_schema(subject=1, trial=2,
                     reason="re-reviewed video, recording was valid")

MATLAB equivalent:

scidb.exclude_schema("equipment malfunction", 'subject', 1, 'trial', 2)
scidb.include_schema("re-reviewed, recording was valid", 'subject', 1, 'trial', 2)
tbl = scidb.list_exclusions()

Scoped to one variable (variable=): only that variable is bad or absent here. Every step that reads it skips the location, and the step's outputs read ⊘ "inherited from ". Two presets write the same rows:

scidb.exclude_schema("EMG lead fell off", variable=RawEMG, subject=3, trial=2)
scidb.mark_missing_expected(RawEMG, subject=4)      # scoped exclusion
scidb.mark_present_expected(RawEMG, subject=4)      # scoped include

Exclusions live in the __scidb_schema_overrides table (one row per change, with its reason) and are applied automatically by for_each before the iteration loop: an excluded location is not run. They are not part of any record or invocation identity.

from scidb import configure_database, BaseVariable
import numpy as np

db = configure_database("experiment.duckdb", ["subject", "session"])

class RawSignal(BaseVariable):
    schema_version = 1

RawSignal.save(np.array([1, 2, 3]), subject=1, session="A")
raw = RawSignal.load(subject=1, session="A")  # one match -> a BaseVariable; several -> a list
all_versions = list(RawSignal.load(subject=1, session="A", version="all"))

Did that step work? (step verdict)

from scidb.provenance.status import step_status

v = step_status(db, filter_emg, [FilteredEMG])
v.status_line()   # "✅ Succeeded · 40 of 40 present · +4 new to process"
v.chip()          # the top reason: "✕ 3 failed", "↓ 2 missing input", "○ not run", ...

Every planned location (the product of each iteration key's levels, taken within each stratum of the declared between keys) gets one state: ✓ present, ⚠ warning (an anomaly finding the run stored), ↻ outdated, ✕ failed, ↓ missing input, + new, ○ not run, ⊘ excluded. scidb status FN prints the same. Try a function without saving anything: scidb.run.try_step.try_step(fn, inputs, outputs, locations=...). Details: docs/claude/pipeline-status.md.

Data Inventory (missing schema-key combinations)

"Which combinations of these schema keys have no data?" The inventory crosses every value each chosen key has anywhere in _schema (in the declared [schema_keys] order) and gives each combination a status: present, missing, or excluded (an active exclude_schema matches it; excluded wins over present). A deeper row counts for its projection onto the chosen keys, and only the database is consulted, never a Variable.

from scidb.provenance.inventory import schema_inventory, write_inventory_csv

inv = schema_inventory(db, ["subject", "session"])   # None = every key
inv.counts            # {"present": 10, "missing": 2, "excluded": 0}
write_inventory_csv(inv, "missing.csv", missing_only=True)
scidb inventory --keys subject,session          # counts + missing table
scidb inventory --all --csv inventory.csv       # every combination + status
scidb inventory --within group                  # one inventory per study arm

within=["group"] (CLI --within group) runs the inventory separately for each group value and crosses only the values found in that arm, so a subject of one arm is never reported missing at the other arm's sessions. Use it for design keys (arm, site, protocol).

Above 1,000,000 combinations the call raises before building the product. The GUI's 📋 Inventory button shows the same table. Details: docs/claude/data-inventory.md.

Logging

Every scidb operation logs through the shared scistacklog facade. configure_database() points both log files beside the database file and writes a run-context header (package versions, Python version, pid), so each log file is self-describing:

  • journal (scistack.log): for the scientist. One block per run: what ran, the counts, how long, and each failure labelled with who can fix it.
  • debug file (scistack-debug.log): for whoever debugs SciStack. Everything, at DEBUG by default (named internal steps with durations, per-iteration [run]/[skip] lines, per-phase [timing] tables), with the originating layer ([scidb], [scifor], [matlab], …) and the run id on every line.
  • console (stderr): the pipeline narrative, at INFO.
from scidb import Log

Log.set_level("INFO", sink="file")   # a quieter debug file
Log.set_level("DEBUG")               # both sinks

or set the SCIDB_LOG_LEVEL environment variable, or pass -v to the scidb CLI.

Metadata

Release files for scistack-db 0.4.13

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scistack-db 0.4.13
File Size Uploaded Provenance Actions
scistack_db-0.4.13.tar.gz 694.3 kB With provenance Details for scistack_db-0.4.13.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scistack-db 0.4.13
File Interpreter ABI Platform Actions
scistack_db-0.4.13-py3-none-any.whl Python 3 none any Details for scistack_db-0.4.13-py3-none-any.whl

Total release size: 1.5 MB

Release files / scistack_db-0.4.13.tar.gz

Download URL scistack_db-0.4.13.tar.gz
Size 694.3 kB
Tags Source
SHA-256 checksum
How to use checksums
8b89bc7b03cc7b085121b23b47138a0eae870f3cef99f59af4d364ac4a24d265
BLAKE2b-256 checksum
How to use checksums
f63c61a1569329de26b763d094b8c05f4e9efbc3e41d48016cc12d6451d686e6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Release files / scistack_db-0.4.13-py3-none-any.whl

Download URL scistack_db-0.4.13-py3-none-any.whl
Size 764.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3ad26c9762cf29d494753849d1aaffc37b91ccd9e79d203a148ea64d444d1fb2
BLAKE2b-256 checksum
How to use checksums
98279f0bcdc9845247318c07b76ce72f2e6a6e1b8609ebdddc121c96cfdfc43c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Security for scistack-db 0.4.13

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for scistack_db-0.4.13.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.4.13 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page