Skip to main content

SciDB

Database operations layer for SciStack. Provides abstractions for defining typed variables, configuring the database, and saving/loading data by metadata.

Named Filters (value-based, reusable)

Filter objects are first-class values — assign them to a variable and reuse them in any where= clause, or compose them with & / | / ~:

# Define once (this IS a ColumnFilter — no special wrapper needed)
clean_gr = GAITRiteLoadedCycle["StepLengths_GR"] != 0

# Use anywhere
for_each(mean_change_from_reference,
         inputs={"baseline": Fixed(GAITRiteLoadedCycle["StepLengths_GR"], session="BL"),
                 "value":    GAITRiteLoadedCycle["StepLengths_GR"]},
         outputs=[DeltaStepLength],
         where=clean_gr & (UAStartFoot() == "A"))

# Compose named filters
clean_and_unilateral = clean_gr & (UAStartFoot() == "U")
for_each(..., where=clean_and_unilateral)

MATLAB equivalent:

clean_gr = GAITRiteLoadedCycle("StepLengths_GR") ~= 0;
scidb.for_each(@meanChangeFromReference, ..., where=clean_gr & (UAStartFoot() == "A"));

Permanent Schema-Level Exclusions

For data that should be excluded from every analysis (e.g., a failed recording session), use the exclusion registry instead of per-call filters.

# Mark a specific trial as excluded (persisted in the database)
scidb.exclude_schema(subject=1, trial=2,
                     reason="equipment malfunction during recording")

# Exclude an entire subject (trial omitted = wildcard)
scidb.exclude_schema(subject=3, reason="participant withdrew")

# Inspect currently-excluded combinations
exclusions_df = scidb.list_exclusions()

# Re-include (logged; the original exclusion row is preserved)
scidb.include_schema(subject=1, trial=2,
                     reason="re-reviewed video, recording was valid")

MATLAB equivalent:

scidb.exclude_schema("equipment malfunction", 'subject', 1, 'trial', 2)
scidb.include_schema("re-reviewed, recording was valid", 'subject', 1, 'trial', 2)
tbl = scidb.list_exclusions()

Exclusions live in the __scidb_schema_overrides table (one row per change, with its reason) and are applied automatically by for_each before the iteration loop: an excluded location is not run. They are not part of any record or invocation identity.

from scidb import configure_database, BaseVariable
import numpy as np

db = configure_database("experiment.duckdb", ["subject", "session"])

class RawSignal(BaseVariable):
    schema_version = 1

RawSignal.save(np.array([1, 2, 3]), subject=1, session="A")
raw = RawSignal.load(subject=1, session="A")  # one match -> a BaseVariable; several -> a list
all_versions = list(RawSignal.load(subject=1, session="A", version="all"))

Data Inventory (missing schema-key combinations)

"Which combinations of these schema keys have no data?" The inventory crosses every value each chosen key has anywhere in _schema (in the declared [schema_keys] order) and gives each combination a status: present, missing, or excluded (an active exclude_schema matches it; excluded wins over present). A deeper row counts for its projection onto the chosen keys, and only the database is consulted, never a Variable.

from scidb.provenance.inventory import schema_inventory, write_inventory_csv

inv = schema_inventory(db, ["subject", "session"])   # None = every key
inv.counts            # {"present": 10, "missing": 2, "excluded": 0}
write_inventory_csv(inv, "missing.csv", missing_only=True)
scidb inventory --keys subject,session          # counts + missing table
scidb inventory --all --csv inventory.csv       # every combination + status
scidb inventory --within group                  # one inventory per study arm

within=["group"] (CLI --within group) runs the inventory separately for each group value and crosses only the values found in that arm, so a subject of one arm is never reported missing at the other arm's sessions. Use it for design keys (arm, site, protocol).

Above 1,000,000 combinations the call raises before building the product. The GUI's 📋 Inventory button shows the same table. Details: docs/claude/data-inventory.md.

Logging

Every scidb operation logs through the shared scistacklog facade. configure_database() points both log files beside the database file and writes a run-context header (package versions, Python version, pid), so each log file is self-describing:

  • journal (scistack.log): for the scientist. One block per run: what ran, the counts, how long, and each failure labelled with who can fix it.
  • debug file (scistack-debug.log): for whoever debugs SciStack. Everything, at DEBUG by default (named internal steps with durations, per-iteration [run]/[skip] lines, per-phase [timing] tables), with the originating layer ([scidb], [scifor], [matlab], …) and the run id on every line.
  • console (stderr): the pipeline narrative, at INFO.
from scidb import Log

Log.set_level("INFO", sink="file")   # a quieter debug file
Log.set_level("DEBUG")               # both sinks

or set the SCIDB_LOG_LEVEL environment variable, or pass -v to the scidb CLI.

Metadata

Release files for scistack-db 0.2.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scistack-db 0.2.5
File Size Uploaded Provenance Actions
scistack_db-0.2.5.tar.gz 545.6 kB With provenance Details for scistack_db-0.2.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scistack-db 0.2.5
File Interpreter ABI Platform Actions
scistack_db-0.2.5-py3-none-any.whl Python 3 none any Details for scistack_db-0.2.5-py3-none-any.whl

Total release size: 1.1 MB

Release files / scistack_db-0.2.5.tar.gz

Download URL scistack_db-0.2.5.tar.gz
Size 545.6 kB
Tags Source
SHA-256 checksum
How to use checksums
6d6717b83da5244b7902ef7c0df9c904561da996b701054da0a34a679f5afafb
BLAKE2b-256 checksum
How to use checksums
a870723a027b1dbf31c86492f9ff46fedf3df2ea63f060db23db2c2ccb805ef7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.

Transparency log

Release files / scistack_db-0.2.5-py3-none-any.whl

Download URL scistack_db-0.2.5-py3-none-any.whl
Size 591.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b1448a4ddad8141335cb5541889cf35bbfe7433ef5a5fec5d9e56a7c42b17a3b
BLAKE2b-256 checksum
How to use checksums
2b688912d10006813264bd90894be1426d4aefe91ec11722079e4f1b05a3b3c3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.

Transparency log

Security for scistack-db 0.2.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for scistack_db-0.2.5.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.5 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page