Skip to main content

SciDB

Database operations layer for SciStack. Provides abstractions for defining typed variables, configuring the database, and saving/loading data by metadata.

Named Filters (value-based, reusable)

Filter objects are first-class values — assign them to a variable and reuse them in any where= clause, or compose them with & / | / ~:

# Define once (this IS a ColumnFilter — no special wrapper needed)
clean_gr = GAITRiteLoadedCycle["StepLengths_GR"] != 0

# Use anywhere
for_each(mean_change_from_reference,
         inputs={"baseline": Fixed(GAITRiteLoadedCycle["StepLengths_GR"], session="BL"),
                 "value":    GAITRiteLoadedCycle["StepLengths_GR"]},
         outputs=[DeltaStepLength],
         where=clean_gr & (UAStartFoot() == "A"))

# Compose named filters
clean_and_unilateral = clean_gr & (UAStartFoot() == "U")
for_each(..., where=clean_and_unilateral)

MATLAB equivalent:

clean_gr = GAITRiteLoadedCycle("StepLengths_GR") ~= 0;
scidb.for_each(@meanChangeFromReference, ..., where=clean_gr & (UAStartFoot() == "A"));

Permanent Schema-Level Exclusions

For data that should be excluded from every analysis (e.g., a failed recording session), use the exclusion registry instead of per-call filters.

# Mark a specific trial as excluded (persisted in the database)
scidb.exclude_schema(subject=1, trial=2,
                     reason="equipment malfunction during recording")

# Exclude an entire subject (trial omitted = wildcard)
scidb.exclude_schema(subject=3, reason="participant withdrew")

# Inspect currently-excluded combinations
exclusions_df = scidb.list_exclusions()

# Re-include (logged; the original exclusion row is preserved)
scidb.include_schema(subject=1, trial=2,
                     reason="re-reviewed video, recording was valid")

MATLAB equivalent:

scidb.exclude_schema("equipment malfunction", 'subject', 1, 'trial', 2)
scidb.include_schema("re-reviewed, recording was valid", 'subject', 1, 'trial', 2)
tbl = scidb.list_exclusions()

Exclusions live in the __scidb_schema_overrides table (one row per change, with its reason) and are applied automatically by for_each before the iteration loop: an excluded location is not run. They are not part of any record or invocation identity.

from scidb import configure_database, BaseVariable
import numpy as np

db = configure_database("experiment.duckdb", ["subject", "session"])

class RawSignal(BaseVariable):
    schema_version = 1

RawSignal.save(np.array([1, 2, 3]), subject=1, session="A")
raw = RawSignal.load(subject=1, session="A")  # one match -> a BaseVariable; several -> a list
all_versions = list(RawSignal.load(subject=1, session="A", version="all"))

Data Inventory (missing schema-key combinations)

"Which combinations of these schema keys have no data?" The inventory crosses every value each chosen key has anywhere in _schema (in the declared [schema_keys] order) and gives each combination a status: present, missing, or excluded (an active exclude_schema matches it; excluded wins over present). A deeper row counts for its projection onto the chosen keys, and only the database is consulted, never a Variable.

from scidb.provenance.inventory import schema_inventory, write_inventory_csv

inv = schema_inventory(db, ["subject", "session"])   # None = every key
inv.counts            # {"present": 10, "missing": 2, "excluded": 0}
write_inventory_csv(inv, "missing.csv", missing_only=True)
scidb inventory --keys subject,session          # counts + missing table
scidb inventory --all --csv inventory.csv       # every combination + status
scidb inventory --within group                  # one inventory per study arm

within=["group"] (CLI --within group) runs the inventory separately for each group value and crosses only the values found in that arm, so a subject of one arm is never reported missing at the other arm's sessions. Use it for design keys (arm, site, protocol).

Above 1,000,000 combinations the call raises before building the product. The GUI's 📋 Inventory button shows the same table. Details: docs/claude/data-inventory.md.

Logging

Every scidb operation logs through the shared scistacklog facade. configure_database() points both log files beside the database file and writes a run-context header (package versions, Python version, pid), so each log file is self-describing:

  • journal (scistack.log): for the scientist. One block per run: what ran, the counts, how long, and each failure labelled with who can fix it.
  • debug file (scistack-debug.log): for whoever debugs SciStack. Everything, at DEBUG by default (named internal steps with durations, per-iteration [run]/[skip] lines, per-phase [timing] tables), with the originating layer ([scidb], [scifor], [matlab], …) and the run id on every line.
  • console (stderr): the pipeline narrative, at INFO.
from scidb import Log

Log.set_level("INFO", sink="file")   # a quieter debug file
Log.set_level("DEBUG")               # both sinks

or set the SCIDB_LOG_LEVEL environment variable, or pass -v to the scidb CLI.

Metadata

Release files for scistack-db 0.3.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scistack-db 0.3.6
File Size Uploaded Provenance Actions
scistack_db-0.3.6.tar.gz 585.1 kB With provenance Details for scistack_db-0.3.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scistack-db 0.3.6
File Interpreter ABI Platform Actions
scistack_db-0.3.6-py3-none-any.whl Python 3 none any Details for scistack_db-0.3.6-py3-none-any.whl

Total release size: 1.2 MB

Release files / scistack_db-0.3.6.tar.gz

Download URL scistack_db-0.3.6.tar.gz
Size 585.1 kB
Tags Source
SHA-256 checksum
How to use checksums
9af343017a6249b4b649596cc37e19d1e9bb179bbe5a445c58a17fd46e9bb9d7
BLAKE2b-256 checksum
How to use checksums
a099c1f218a3313bb049b467e577bdc2a9783bdfb745f06030207168cca62fb3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Release files / scistack_db-0.3.6-py3-none-any.whl

Download URL scistack_db-0.3.6-py3-none-any.whl
Size 634.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
fef7d308d2585b747d0a0e04c5acc3067a8b25214bad470318e2768168f410c8
BLAKE2b-256 checksum
How to use checksums
1b416f7f3da048971395596ab475e26b32c0d5b28d2cbd21d7a0b3745d17429d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Security for scistack-db 0.3.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for scistack_db-0.3.6.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.3.6 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page