sollertia-forgery

Provides tools for processing and managing the data acquired using the Sollertia data acquisition platform.

---

Detailed Description

This library is part of the Sollertia AI-assisted scientific data acquisition and processing platform, built on the Ataraxis framework and developed in the Sun (NeuroAI) lab at Cornell University. It is the processing counterpart to sollertia-experiment, which acquires the sessions this library reads. It processes the raw data of a recorded session into multiple per-session intermediate data tables and forges the processed sessions into the multi-session datasets consumed by the downstream analysis assets.

Every processing pipeline runs the same way for every acquisition system. Each system contributes its own parsers and workers as data, through the dispatch registries described in Acquisition Systems, so every command infers its acquisition system from the session instead of taking one from the caller. The library plans each unit's jobs, sizes the cores and memory used by every job, and dispatches the resulting batches onto this machine's process pool or onto a SLURM compute server.

The preferred way for using the library runs through AI agents. Every operation except slf mcp, which starts the MCP server itself, and slf omp, which links the macOS OpenMP runtime, is exposed as a Model Context Protocol tool, and the Claude Code skills described in AI-Assisted Development orchestrate those tools. The slf CLI serves the same operations to a human operator and to the job scripts the compute server runs.

---

Features

• Supports Windows, Linux, and macOS.
• Processes camera, microcontroller, acquisition-runtime, and two-photon imaging data for supported acquisition systems.
• Forges the processed sessions of a project into analysis-ready multi-session datasets.
• Sizes every job from the data it reads and dispatches it onto a local process pool or onto a SLURM scheduler.
• Exposes every operation but the slf mcp launcher and the slf omp macOS OpenMP setup through an MCP server.
• Apache 2.0 License.

---

Table of Contents

• Dependencies
• Installation
  • Source
  • pip
• Usage
  • Acquisition Systems
  • Processing Pipelines
  • Jobs, Trackers, and Batches
  • Processed Data Structure
  • Forged Datasets
  • CLI Commands
  • Configuring Server Access
  • Running a Remote Batch
  • Recovering from Interruptions
• API Documentation
• AI-Assisted Development
  • MCP Server
  • Skills
  • Client Registration
• Developers
  • Installing the Project
  • Additional Dependencies
  • Development Automation
  • Adding a New Acquisition System
  • Adding a New Session Type
  • Adding a New Processing Stage
  • Adding a New Processing Pipeline
  • Adding an MCP Tool
  • Automation Troubleshooting
• Versioning
• Authors
• License
• Acknowledgments

---

Dependencies

For users, all library dependencies are installed automatically by all supported installation methods. For developers, see the Developers section for information on installing additional development dependencies.

On macOS, this library uses Numba's OpenMP threading layer, because the Numba macOS wheel carries no TBB support. The OpenMP runtime (libomp.dylib) ships with neither Numba nor macOS itself, so it is installed separately.

Run slf omp to report the runtimes present on the host, and slf omp --yes to make one loadable. The command finds runtimes installed by Homebrew or MacPorts, present in the active conda environment, or carried inside an installed Python package, and links the one it finds into the library directory of the interpreter that runs it. That directory belongs to the interpreter, so a conda environment needs no elevated permission while a system-wide interpreter does. Keep the same interpreter on an elevated run, because the link target follows the interpreter that runs the command. Install one with brew install libomp when the command finds none. Without a loadable runtime, the two-photon and forging pipelines fail once they reach one of cindra's parallelized stages. Linux and Windows run the TBB threading layer, which needs no additional steps, so slf omp errors when run on them.

---

Installation

Source

Note, installation from source is highly discouraged for anyone who is not an active project developer.

1. Download this repository to the local machine using the preferred method, such as git-cloning. Use one of the stable releases that include precompiled binary and source code distribution (sdist) wheels.
2. If the downloaded distribution is stored as a compressed archive, unpack it using the appropriate decompression tool.
3. cd to the root directory of the prepared project distribution.
4. Run pip install . to install the project and its dependencies.

pip

Use the following command to install the library and all of its dependencies via pip: pip install sollertia-forgery

---

Usage

Acquisition Systems

An acquisition system is the collection of hardware and software that records a session, and it is defined by sollertia-shared-assets as a member of the AcquisitionSystems enumeration. Every session on disk names the system that recorded it, so the pipelines in this library resolve the system from the session's metadata. The current reference system is Mesoscope-VR, which pairs a 2-Photon Random Access Mesoscope with a Unity virtual reality task.

The processing packages themselves are system-agnostic. A category package such as video, microcontrollers, runtime, two_photon, or forging owns one pipeline that runs identically for every system, and everything a specific system contributes lives in a per-system package such as mesoscope_vr. The two meet in the library's registries.py module, which maps each AcquisitionSystems member to the parsers, resolvers, locators, and workers donated by that member:

Donation	What the system supplies
Microcontroller parsers	One parser per hardware module, keyed by the module's type and identifier
Microcontroller event codes	The event codes each parsed module reads, which build the extraction filter
Microcontroller eligibility	The modules a given session configured for use
Runtime binding	The source identifier of the runtime log, paired with the parser that decodes its payloads
Pose prediction locator	The externally-produced pose-prediction file the system's tracking pass reads
Video tracking	The pass that reads the session's pose predictions and writes its tracking outputs
Two-photon data locator	The raw imaging directory the two-photon pipeline hands to cindra
Cindra configuration resolvers	The single-recording and multi-recording configurations passed to cindra
Forging assembler	The per-session worker that assembles the session's data.feather, plus its columns
Assembly geometry resolver	The heights at which the system's own assembler holds a session's frame and its sources
Assembly source resolver	The height at which that assembler holds each source it reads for a session
Forging admission policy	The pipelines a session of each type completes before it joins a dataset
Multi-recording session types	The session types the system tracks across recordings

A coverage check runs when registries.py is imported and raises a RuntimeError naming the registry where it stopped and the systems missing from it, so a partially wired system fails at import time. The dependency runs one way, because a per-system package never imports an agnostic category package.

Note, adding a system to the platform spans three repositories. The enumeration member and the session records belong to sollertia-shared-assets, the acquisition runtime belongs to sollertia-experiment, and the donations belong here. See Adding a New Acquisition System for the steps this library requires.

Processing Pipelines

Each pipeline reads one class of acquired data and writes its outputs beside the session. A pipeline is made of ordered stages, and each stage contributes one or more independently schedulable jobs:

Pipeline	Stages	Unit
checksum	checksum_resolution	Session
runtime	runtime_processing	Session
microcontroller	microcontroller_data_extraction, module_parsing	Session
video	camera_timestamp_extraction, camera_timestamp_rename, pose_tracking, motion_energy	Session
two_photon	binarization, registration, processing, combination	Session
forging	multiday_discovery, multiday_extraction, session_data_assembly	Dataset
manifest	manifest_generation	Project

The microcontroller_data_extraction, camera_timestamp_extraction, and two-photon stages are owned by this library's upstream dependencies. This library resolves the inputs for those stages, calls their job bindings in-process, and reuses each library's own exported job-name constant, so the identifiers recorded by a tracker stay aligned with the library that produced the work. The forging pipeline's multiday_discovery and multiday_extraction stages are likewise owned by cindra and dispatched in-process, but the forging tracker records them under this library's own job names, because it interleaves them with the per-session assembly stage this library owns. Every other stage is implemented here.

A stage resolves its own job universe from the acquisition data. The video pipeline reads the camera manifest, so it declares one timestamp job and one motion-energy job per registered camera regardless of which archives happen to be on disk. The microcontroller pipeline declares one parse job for every module used by the processed session that has an assigned parser. The forging pipeline declares one assembly job per session in the dataset. Because every pipeline resolves its universe this way, a completed tracker already accounts for every source recorded by the session.

Jobs, Trackers, and Batches

Work reaches a host as a job, and every pipeline models its jobs the same way.

1. Plan. slf plan reads a unit's acquisition data, registers on its processing tracker every job that unit is able to run, and records each job's cores, its two memory figures, and its upstream jobs into a per-unit job_plan.yaml. A unit runs only the jobs registered on its tracker.
2. Prepare. Preparation joins the tracker state to those records into one descriptor per job, and registers the result under a batch identifier. A job counts as blocked when preparation can neither queue its upstream stage nor confirm that the stage already succeeded.
3. Execute. Dispatch runs the prepared batch on one of two backends. The local engine admits jobs against a budget of cores and anonymous memory, then dispatches them onto a shared process pool in dependency order. The remote engine submits one SLURM allocation per job, each requesting the job's resident memory and sequenced through an afterok dependency.
4. Close. Closure snapshots the state recorded by a finished batch's jobs, while the batch is still tracked.

Every job carries two memory figures, and the same sizing pass resolves both. The anonymous figure is the memory the job allocates, and the resident figure adds the pages the job maps and the shared library image its processes hold in common, above a margin. The local engine budgets against the anonymous figure, because anonymous pages are the ones a host cannot reclaim under pressure. A SLURM allocation requests the resident figure, because the scheduler's memory cgroup charges a job for every page it holds. A stage carries a non-zero mapped term when it holds a map open at its peak instead of reading through one and releasing it, which covers the two-photon binarization, registration, and processing stages and the forging pipeline's multiday_extraction and session_data_assembly stages. Every other job's two figures differ by the shared image and the margin alone.

Each job holds one of four statuses on its tracker, and a rerun resolves only the work still outstanding:

Status	Meaning
SCHEDULED	The job is registered and has not started
RUNNING	The job is executing, and the record names its executor
SUCCEEDED	The job completed and its output is on disk
FAILED	The job raised, and a reset returns it to SCHEDULED

Trackers are per-unit YAML files written under a file lock, so the state of a unit travels with the unit's data. Project-level artifacts roll that state up for a submitting host that holds none of the data:

Artifact	Contents
<project>/<project>_manifest.feather	One row per session, snapshotting the project's state
<project>/<project>_jobs.feather	One row per tracked job of every per-session pipeline
<project>/<project>_plan.feather	Every plan cache under the project, collected into one table
<dataset>/dataset_state.feather	One dataset's forging job state, in a shippable form
<working directory>/remote_state/	The artifacts mirrored from the server and this host's own batch and submission records

Processed Data Structure

The acquisition side owns a session's raw_data directory, and every session pipeline except checksum writes only under processed_data:

Session/
├── raw_data/                                     <- Written by sollertia-experiment
│   ├── ax_checksum.txt                           <- Verified and regenerated by the checksum pipeline
│   ├── checksum_processing_tracker.yaml
│   ├── behavior_data/                            <- Every log archive, plus the shared camera manifest
│   ├── camera_data/                              <- The camera recordings and the tracking pass's pose predictions
│   └── ...                                       <- The session's descriptors, configurations, and metadata
└── processed_data/
    ├── job_plan.yaml                             <- Every job's cores, memory figures, and upstream jobs
    ├── runtime_data/
    │   ├── runtime_processing_tracker.yaml
    │   └── ...                                   <- The system's parsed runtime state and trial tables
    ├── microcontroller_data/
    │   ├── microcontroller_processing_tracker.yaml
    │   ├── extraction_configuration.yaml         <- The per-controller filter the extraction stage runs
    │   ├── controller_{id}_module_{type}_{id}.feather
    │   └── ...                                   <- One parsed table per module, named by the system
    ├── video_data/
    │   ├── video_processing_tracker.yaml
    │   ├── camera_{source_id}_timestamps.feather <- One per camera, written by the extraction stage
    │   ├── {camera_name}_timestamps.feather      <- The manifest-named hardlink the rename stage publishes
    │   ├── {camera_name}_energy.feather          <- One per camera, written by the motion-energy stage
    │   └── ...                                   <- Whatever the system's tracking pass writes
    └── cindra/                                   <- Written by cindra's single-recording pipeline
        ├── single_recording_tracker.yaml
        └── ...                                   <- The registered stacks, detected ROIs, and traces

The checksum tracker sits under the acquired data, because that pipeline verifies the acquired data in place. Every other pipeline records beside the output it produces. The elided entries are named by the acquisition system's donated parsers and workers, so their filenames and column schemas belong to that system instead of the pipeline. The agnostic entries are stable across systems, and every camera table is positional, carrying one row per acquired frame.

Forged Datasets

A dataset aggregates the processed sessions of one session type, recorded by one acquisition system, across many animals of one project, and it lives under the project root beside the animal directories. Defining a dataset requires every named session to report a success for every pipeline its acquisition system requires for that session type, and one under-processed session aborts the definition before any of the hierarchy is built. The definition bakes that system's column meanings into the dataset at definition time:

Project/
└── Dataset/                          <- A sibling of the project's animal directories
    ├── dataset.yaml                  <- The dataset marker, which names its project, session type, and system
    ├── data_descriptions.feather     <- Every column the system's assembler can emit, with its meaning
    ├── dataset_state.feather         <- The dataset's forging job state, written by 'slf dataset-state'
    ├── forging_tracker.yaml
    ├── job_plan.yaml
    └── Animal/
        ├── surgery_metadata.yaml
        ├── multi_recording_configuration.yaml   <- Written for an animal the system tracks across recordings
        └── Session/
            ├── data.feather          <- The assembled per-session data the analysis layer reads
            └── ...                   <- The session's acquisition snapshots, re-exported beside its data

The forging pipeline runs the cross-recording stages first, one discovery job per animal whose system resolves a multi-recording configuration and one extraction job per that animal's session, then assembles one data.feather per session. Extending an existing dataset materializes the animals introduced by the call, the animals it names for recreation, and any animal the dataset already holds whose configuration is missing from disk. An identical call therefore heals a definition that failed partway. A large project forges in passes, while part of its source data lives elsewhere.

CLI Commands

This library provides the slf CLI. Every command infers the acquisition system from the data it opens, so no command takes a system selector:

Command	Description
plan session	Records what every processing job of each named session costs
plan dataset	Records what every forging job of each named dataset costs
plan project	Projects every plan cache under the project into one table at the project root
process video	Extracts camera frame timestamps, processes pose predictions, and measures motion energy
process microcontroller	Extracts the microcontroller log archives and parses them into behavior feathers
process runtime	Decodes the acquisition runtime log archive into the session's runtime behavior feathers
process two-photon	Runs the single-recording two-photon (calcium-imaging) processing pipeline for a session
forge	Forges a dataset by assembling per-session data from a project's processed sessions
checksum	Resolves the data integrity checksum for the target session's 'raw_data' directory
manifest create	Creates the .feather file capturing the snapshot of the target project's state
manifest print	Prints the requested data from the project's manifest as a formatted table
dataset-state	Snapshots each named dataset's forging job state into a shippable table
reset	Returns tracked jobs of the named units to the scheduled state
clean	Removes a pipeline's output and processing tracker for the named units
server configure	Creates the server configuration file in the working directory's configuration subdirectory
server print	Displays the remote server's SLURM queue status or job data as a formatted table
server discover	Discovers and prints the sessions stored under the project's directory on the server
server batches	Reports the batches outstanding on the server's scheduler and resolves every allocation
server retire-batch	Remediates the named batches and drops them from this machine's submission ledger
server pull	Copies a file or directory off the compute server onto this machine
mcp	Starts the agentic Model Context Protocol server using the requested transport
omp	Links the OpenMP runtime Numba loads on macOS into a directory the loader searches

Use slf --help or slf SUBCOMMAND --help for detailed usage information.

The process group parses the session path, the job identifier, the worker budget, and the progress flag, so those options come before the subcommand name, as -pp does on the manifest group. Without a job identifier, a subcommand runs every job the session's data supports on this host, re-running the ones that already succeeded, and with one it runs exactly the job named by that identifier. Resolving only the outstanding work belongs to batch preparation instead of the direct invocation. That is how a scheduler drives cross-job parallelism, by dispatching each identifier as its own allocation. The runtime subcommand uses no identifier, since its single-job pipeline has no remote-dispatch job.

Note, on macOS the Numba threading layer resolves its OpenMP runtime through the rpath dependency that the omppool extension records, which the loader expands against the entries the running interpreter carries. The Dependencies section covers linking one. Every pipeline that dispatches a parallel worker pool verifies the runtime before it starts a job.

Configuring Server Access

To access the remote compute server, first author the server configuration. The slf server configure command creates the configuration and stores it as the 'server_configuration.yaml' file, under the configuration subdirectory of the Sollertia platform working directory. It records the username, the password, the host, the absolute path to the server's data root, and the name of the shared conda environment that every remote job activates. That environment supplies the remote half of every job, so it holds this library and the processing libraries it drives.

Running a Remote Batch

The remote path runs the same prepared jobs that the local batch engine runs, so one job graph, one core table, and one memory model serve both. The local engine schedules on the anonymous figure and the remote engine on the resident one. A run has three steps, each exposed as a Model Context Protocol tool and carried out on the server by the slf CLI:

1. Prepare. prepare_batch_tool with host='remote' refreshes the plan and state tables on the server and pulls them, which is one state table per project for a session batch and one per named dataset for a dataset batch. Planning reads each unit's acquisition data, writes the runnable jobs of that unit onto its processing tracker, and records each job's cores, both of its memory figures, and its upstream jobs. It refreshes the recorded status every time and re-estimates only the figures a unit's cache does not already hold, so a submission's sizing never changes underneath it. State generation turns those trackers into a table. A job absent from that table is a job that the unit is unable to run. The submitting host therefore resolves a batch from the tables alone, without opening anything on the server. The plan and state records join on the job identifier and become one descriptor per job, registered under a batch identifier.
2. Submit. execute_jobs_tool renders each job as its own shell script, transfers it over SFTP, and submits it as its own SLURM allocation, requesting the resident memory the job's own estimate reports, in dependency order. It takes no host of its own, because a batch runs where it was prepared. Each script carries an SBATCH directive block, activates the configured conda environment, and then runs the slf command for its job. Each job declares an afterok dependency on the allocations of the upstream jobs held by the batch, so the scheduler sequences the graph and the batch finishes with nothing running locally.
3. Read. get_processing_status_tool with host='remote' reports what the scheduler observed while a run is in flight. generate_project_manifest_tool and generate_dataset_state_tool, each with host='remote', regenerate the project's manifest, job table, and dataset state on the server. Every read tool called with host='remote' then mirrors the project's artifacts into <working directory>/remote_state/<project>/ and reads them exactly as it reads a local project. The plan projection is mirrored as the server last wrote it, since replanning belongs to preparation.

Because the scheduler owns the run once it accepts the jobs, every accepted allocation is recorded in a submission ledger at <working directory>/remote_state/submission_ledger.yaml, guarded by a file lock, as is every other shared artifact this library keeps. The ledger keeps concurrent batches queryable, keeps a batch findable after this process exits, and preserves the record of the accepted allocations when the scheduler rejects a later job of the same batch. A batch leaves the ledger once every one of its allocations resolves to the plain drop remediation and its closure has snapshotted what its jobs recorded. That rule is derived from the same resolution the read publishes, so a batch that the read reports as still held never closes. A batch holding an allocation for which the resolution answers any other way stays outstanding for an explicit remediation, and so does a batch whose closure failed.

Every status read also resolves each outstanding allocation against three records, which are the scheduler's accounting, the scheduler's queue, and the processing tracker of the job the allocation carries. The allocation is held when the queue carries it or accounting reports a state it has yet to leave, and settled when accounting reports a state it never leaves and the queue does not carry it. It is gone only when accounting returns no row for it and the queue does not carry it either. A missing row is not evidence of a purge, because accounting answers the same way for a submission it has not registered yet, and nothing observable separates the two. So gone licenses no claim that the allocation ever ran or that a later query answers alike. What it does license is narrower. This reading did not observe the allocation settle, so the batch holding it never settles on its own, and only an explicit remediation releases it.

That state and the job's own tracker together carry the allocation's verdict. running means the scheduler still holds the allocation or the one the job's tracker claims, or that tracker names an executor outside the scheduler, for which neither record can answer. Nothing is done to a running allocation. finished and failed mean the job recorded an outcome, which its tracker keeps. abandoned means nothing claims the job, so it is already runnable. stranded means the job's tracker still claims to be running while no allocation is, whether it names a scheduler allocation the records no longer hold or names no executor at all. That is the one case whose remediation writes to a tracker, returning the job to SCHEDULED, because the claim is what no rerun can otherwise clear. Each batch carries a verdict of its own beside them, progressing while any of its allocations resolves running, stalled when none does and at least one is gone, and awaiting_closure otherwise. That last verdict is the state of a batch holding an entry the automatic closure may not drop, and of a batch whose closure failed.

slf server batches and get_processing_status_tool with host='remote' report every one of those values and name the remedy, and slf server retire-batch and retire_remote_batches_tool carry it out, resetting each stranded job and snapshotting what the batch's jobs recorded before dropping the ledger entry. Remediating refuses a batch holding an allocation that resolves as running, and refuses a drop whose snapshot failed, until the caller waives each guarantee with its own flag.

The run reports a job as blocked rather than submitted when it can neither queue that job's upstream stage nor confirm that the stage already succeeded. A local batch treats the same job the same way.

Critical! A remote job runs the same slf command that a local run would run, so test a pipeline on one session locally before dispatching a project-wide batch. A command that works on the local machine behaves the same way on the server.

Recovering from Interruptions

Every pipeline records its progress on a per-unit tracker, so an interrupted run resumes instead of restarting from scratch when possible. What to run depends on what the interruption left behind:

Situation	Recovery
A batch is still running and is no longer wanted	cancel_processing_tool stops the local batch or the outstanding allocations
Jobs report FAILED after a fixable cause	slf reset or reset_processing_jobs_tool returns them to SCHEDULED
A pipeline's output is suspect and must be rebuilt	slf clean or clean_processing_output_tool removes the output and the tracker
A run was interrupted, leaving jobs stuck in RUNNING	A local batch clears every job's record before it dispatches, and a remote batch adopts the allocation the scheduler still holds for the job
A remote job's tracker claims a non-scheduler executor	Neither record answers for that executor, so nothing adopts or resets the job. slf reset or reset_processing_jobs_tool clears the claim, and the next status read closes its batch
A remote batch settled while this host was offline	The next status read closes the settled batch and retires it from the ledger
A remote batch is reported as stalled	slf server retire-batch -b <id> or retire_remote_batches_tool resolves it, remediates it, and drops its ledger entry
A remote job is reported as stranded	The same command returns that job to SCHEDULED on its own tracker, which is what lets a later batch run it again
Remediation refuses because an allocation is running	Wait for it, or cancel it with cancel_processing_tool, then remediate again. --force cancels each one first instead
Remediation refuses because its snapshot failed	Restore access to the server and remediate again, or pass --drop-without-outcome to drop the entries regardless

Note, slf clean removes processed output. The checksum pipeline verifies the acquired data in place and owns no output directory, so cleaning that pipeline removes its tracker alone. Cleaning discards work, and clean_processing_output_tool refuses it while a local batch is running, so prefer a reset whenever the failure cause was external.

---

API Documentation

See the API documentation for the detailed description of the methods and classes exposed by components of this library.

---

AI-Assisted Development

The library ships one MCP server, exposed through the slf mcp command, and a set of Claude Code skills distributed through the sollertia marketplace.

MCP Server

The server exposes the planning, processing, forging, and project management pipelines. The sollertia-shared-assets, ataraxis-video-system, ataraxis-communication-interface, and cindra libraries each serve their own assets through their own MCP server, so this server leaves those tools to them. assets:working-directory sets the data root, and the assets:project-hierarchy, assets:session-discovery, and assets:datasets skills produce the paths named below.

Every tool names a filesystem path by what that path holds. On the batch tools, session_paths names a processing unit root, which is a session root for every session pipeline and a dataset root for forging. A dataset_path on the forging and planning tools names one forged dataset's root, and a project_path names a project root under the data root. Most tools take a host, which is local for the data on this machine and remote for the data on the compute server. Nine tools take none. execute_jobs_tool takes none because a batch runs where it was prepared, forget_prepared_batches_tool and read_resource_model_tool answer for this machine alone, the two server-configuration tools are always local, and discover_remote_project_tool, read_scheduler_jobs_tool, pull_remote_path_tool, and retire_remote_batches_tool always address the compute server. On list_prepared_batches_tool the host is a filter, and omitting it lists the batches prepared against either host.

Starting the Server

Start the MCP server using the CLI:

slf mcp

The -t/--transport option selects the transport. The default stdio serves a local agent client, while sse and streamable-http serve the same tools over the network to a client that reaches a processing host.

Available Tools

The server registers twenty-eight tools across six modules.

The dataset tools compose forged datasets and report their forging job state:

Tool	Description
define_forging_dataset_tool	Creates or extends a forged dataset hierarchy and materializes its per-animal configurations
generate_dataset_state_tool	Snapshots each named dataset's forging job state into a shippable feather file at the dataset root
read_dataset_state_tool	Reads a dataset's forging job state out of its stored snapshot, in three widening stages
list_project_datasets_tool	Lists a project's forged datasets and which of them hold a given session or animal

The management tools read and regenerate a project's state artifacts:

Tool	Description
generate_project_manifest_tool	Regenerates the target project's manifest and job artifacts
read_project_manifest_tool	Reads a project's sessions out of its stored manifest, in three widening stages
read_project_jobs_tool	Reads a project's tracked jobs out of its stored job artifact, in three widening stages
get_manifest_status_tool	Reports the state of the project's last state-artifact generation from its processing tracker

The planning tools record the cost of every job and collect those records into one table:

Tool	Description
plan_session_jobs_tool	Records what every processing job of one or more sessions costs, caching the figures beside each session
plan_dataset_jobs_tool	Records what every forging job of one or more datasets costs, caching the figures at each dataset root
generate_project_plan_tool	Projects every plan cache under a project into one table at the project root
read_project_plan_tool	Reads the planned cores and memory of a project's jobs out of that stored projection

The processing tools resolve, dispatch, and recover a batch:

Tool	Description
prepare_batch_tool	Resolves a pipeline's dispatchable jobs for one or more units, on this machine or on the compute server
inspect_job_resources_tool	Reports the cores and memory a pipeline's outstanding jobs need, running none of them
execute_jobs_tool	Dispatches prepared batches onto this machine's process pool or onto the server's scheduler
get_processing_status_tool	Reports the live status of the active batch, resolving each remote allocation to one verdict
cancel_processing_tool	Cancels the active local batch, or the outstanding allocations of the remote batches
retire_remote_batches_tool	Applies each allocation's resolved remediation, then drops the named batches from the submission ledger
reset_processing_jobs_tool	Resets tracked jobs to SCHEDULED across one or more units, so a later execute reruns only them
clean_processing_output_tool	Removes a pipeline's output and processing tracker for one or more units

The orchestration tools read what this host has already prepared and what a job type costs before any unit is planned:

Tool	Description
list_prepared_batches_tool	Lists the batches this host has prepared, which is how a lost batch identifier is recovered
forget_prepared_batches_tool	Removes what this host records for the named batches, the prepared document and the recorded outcome alike
read_resource_model_tool	Reports the width, the ceiling, and the reservation every job type declares, alongside this machine's capacity

The server tools author the compute server's credentials and read what that server holds:

Tool	Description
read_server_configuration_tool	Loads the server configuration from the working directory's configuration subdirectory, password masked
write_server_configuration_tool	Creates or replaces the server configuration YAML in that same configuration subdirectory
discover_remote_project_tool	Enumerates the sessions and forged datasets a project holds on the compute server, with their server paths
read_scheduler_jobs_tool	Reads the compute server's own record of its allocations, queued or already settled
pull_remote_path_tool	Copies a file or directory off the compute server onto this machine, taking a directory whole

Skills

The forging plugin ships the skills that orchestrate the tools above:

Skill	Description
pipeline	Orders the processing lifecycle end to end and decides which forging skill runs next
batch-processing	Orchestrates batch processing across all six batch pipelines
job-planning	Sizes every runnable job of a session or a dataset before a batch or a submission
dataset-definition	Composes forged dataset hierarchies and reports their forging job state
dataset-forging	Runs the forging pipeline, which processes the entire dataset
remote-execution	Runs work on the configured SLURM compute server and reads its scheduler records
server-configuration	Authors the ServerConfiguration YAML that authorizes SSH and SLURM execution
project-state	Generates and queries the session manifest and the job table published beside it
processing-input-format	Documents the on-disk inputs each batch pipeline requires
processing-results	Documents what each pipeline writes to disk and how an agent verifies it
data-processing-design	Documents the agnostic-worker and per-system-donation design behind the pipelines
library-extension	Adds an acquisition system, session type, processing stage, pipeline, or MCP tool
cli-reference	Documents every slf command and option and the MCP tool each one maps to
forging-mcp-environment-setup	Diagnoses MCP connectivity and owns the response envelope every forging tool returns

Operators working on a specific acquisition system additionally install that system's plugin, which ships the skills for the processing donations that system makes to this library. The mesoscope plugin covers the Mesoscope-VR system, whose mesoscope:mesoscope-vr-module-parsing, mesoscope:mesoscope-vr-trial-decomposition, mesoscope:mesoscope-vr-fluorescence-alignment, mesoscope:mesoscope-vr-video-tracking, mesoscope:mesoscope-vr-imaging-configuration, mesoscope:mesoscope-vr-processing-schema, and mesoscope:mesoscope-vr-dataset-assembly skills document the donations through which the agnostic pipelines above dispatch for that system.

Client Registration

The forging plugin of the sollertia marketplace distributes this library's Claude Code skills and the registration for its MCP server. Installing that plugin registers the MCP server with compatible clients and makes every associated skill available.

Operators additionally install the assets plugin from the same marketplace, because every batch workflow above starts from the session_paths list produced by its assets:session-discovery skill, which is served by the separate slsa mcp server.

Contributors additionally install the automation plugin from the ataraxis marketplace. That plugin provides the skills that enforce the Sollertia coding conventions of this repository, together with the codebase exploration and audit tools.

---

Developers

This section provides installation, dependency, and build-system instructions for the developers that want to modify the source code of this library.

Installing the Project

Note, this installation method requires mamba version 2.3.2 or above. Currently, all automation pipelines require that mamba is installed through the miniforge3 installer.

1. Download this repository to the local machine using the preferred method, such as git-cloning.
2. If the downloaded distribution is stored as a compressed archive, unpack it using the appropriate decompression tool.
3. cd to the root directory of the prepared project distribution.
4. Install the core development dependencies into the base mamba environment via the mamba install tox uv tox-uv command.
5. Use the tox -e create command to create the project-specific development environment followed by tox -e install command to install the project into that environment as a library.
6. macOS only, activate the environment created by step 5 and run slf omp --yes to link an OpenMP runtime into it. Run brew install libomp first when slf omp reports that the host holds none. The environment carries no OpenMP runtime of its own, and the py314-test task fails without one.

Additional Dependencies

In addition to installing the project and all user dependencies, install the following dependencies:

1. Python distributions, one for each version supported by the developed project. Currently, this library supports Python 3.14 only. It is recommended to use a tool like pyenv to install and manage the required versions.

Development Automation

This project uses tox for development automation. The following tox environments are available:

Environment	Description
lint	Runs ruff formatting, ruff linting, and mypy type checking
stubs	Generates py.typed marker and .pyi stub files
{py314}-test	Runs the test suite via pytest and aggregates coverage data
coverage	Aggregates test coverage and applies the 100% coverage gate
docs	Builds the API documentation via Sphinx
build	Builds sdist and wheel distributions
upload	Uploads distributions to PyPI via twine
deploy	Uploads the built documentation to the Netlify site
install	Builds and installs the project into its mamba environment
uninstall	Uninstalls the project from its mamba environment
create	Creates the project's mamba development environment
remove	Removes the project's mamba development environment
provision	Recreates the mamba environment from scratch
export	Exports the mamba environment as a .yml file
import	Creates or updates the mamba environment from a .yml file

Run any environment using tox -e ENVIRONMENT. For example, tox -e lint.

Note, all pull requests for this project have to successfully complete the tox task before being merged. To expedite the task's runtime, use the tox --parallel command to run some tasks in parallel.

Adding a New Acquisition System

A new acquisition system reaches the agnostic pipelines by donating one entry to each registry in registries.py. The Mesoscope-VR package is the reference for both the module split and the contents.

Step 1: Add the enumeration member upstream

AcquisitionSystems is owned by sollertia-shared-assets, and assets:library-extension owns this step. Add the member there first, along with the system's hardware state, experiment configuration, and raw-data dataclasses. In that library's SYSTEM_SESSION_TYPES, pair the member with the session types the system records. Until the steps below are complete, importing any pipeline in this library raises, because the coverage check refuses a partially wired system.

Step 2: Create the system package

Create a <system>/ package alongside mesoscope_vr/, and export every donated symbol from its __init__.py. The package supplies:

1. A parser for every microcontroller hardware module the system records, plus the accessor that returns the event codes read by each parser and the accessor that returns the modules a given session configured for use.
2. The identifier of the runtime log source, together with the parser that interprets the decoded runtime payloads.
3. The locator that resolves the session's externally-produced pose-prediction file, and the video-tracking function that writes the session's tracking outputs. That function no-ops for a system that performs no tracking.
4. The locator that resolves the session's raw two-photon imaging directory, and the resolvers that build the single-recording and multi-recording cindra configurations.
5. The per-session forging assembler, the mapping from each emitted column to that column's meaning, the admission policy that names the pipelines required for each session type, and the session types the system tracks across recordings.
6. The two resolvers that report the shape of the system's own assembly. One gives the heights at which the assembler holds a session's frame and its sources, and the other gives the height at which it holds each source it reads. The sizing pass charges its per-sample terms against those heights.

A system that produces none of a given data class still donates an entry for that class. A no-op tracking function, a pose-prediction locator that returns None, and a multi-recording resolver that returns None are the donations a system makes for a class it never produces. The coverage check therefore stays a check on wiring rather than on capability.

Critical! A system package must never import an agnostic category package. The category pipelines import registries.py and registries.py imports every system package, so the reverse import is a circular import that must be avoided.

Step 3: Register the donations

In registries.py, import the new symbols and add the system's entry to every registry. Then import the library. The coverage check raises a RuntimeError on the first gap it reaches, whether that gap is a registry still missing the system or a parseable module without event codes. The same check fires on a declared multi-recording session type absent from the upstream pairing, and on an admission entry naming a session type the system does not record. One import surfaces one gap, so run the import again after every fix.

Nothing under orchestration/ or interfaces/ changes. The CLI and the MCP tools resolve the system from the session or dataset they open, so a fully wired system reaches every command already exposed.

Step 4: Cover the new package

Add the system's automodule block to docs/source/api.rst, add its tests under tests/, and add the system to the per-system assertions in tests/registry_coverage_test.py, which name each registered system explicitly. That file's donor registry list names the registries themselves instead of systems, so a new system adds no entry to it. The suite gates on 100% statement and branch coverage.

Step 5: Update the sibling libraries

Coordinate with sollertia-experiment, which acquires the sessions the new system records, and confirm that every artifact a pipeline reads is an artifact the acquisition runtime writes. experiment:library-extension owns the seams that half composes. That library carries no import-time coverage check of its own, so a system left unwired in it surfaces only when an operator runs its configure command.

Adding a New Session Type

A session type is owned by sollertia-shared-assets and reaches this library through the recording system's package.

1. Add the SessionTypes member and its descriptor dataclass in sollertia-shared-assets, the step assets:library-extension owns. In SYSTEM_SESSION_TYPES, pair the member with every acquisition system that records it.
2. Add the type to the recording system's admission policy, which names the pipelines a session of that type completes before it joins a forged dataset. A type left out of the policy joins no dataset, which is the deliberate opt-out.
3. Route the type in the system's forging assembler and write the branch that assembles its data. An unrouted type raises when the forging pipeline reaches it.
4. Route the type in the system's assembly-source resolver, which reports the height at which the assembler holds each source it reads. A type that resolver does not cover is refused during the sizing pass, so the type's assembly job is never planned.
5. Add the type to the system's multi-recording session types when the system tracks its animals across recordings, and widen the system's own multi-recording configuration resolver to match. The import check validates the declared types against the upstream pairing alone, so it never catches a resolver that still declines the new type.
6. Give every new dataset column both its column member and its description entry. The system's metadata module raises at import when a column carries no description, because a dataset publishes that mapping alongside its data.

Adding a New Processing Stage

A stage inside an existing pipeline rides the routing that pipeline already has, so it needs no dispatch, CLI, or MCP change.

1. Define the stage's job name in the pipeline module, or reuse the constant the upstream library exports when that library's job binding backs the stage, and export the name from the category package.
2. Emit the job from the pipeline's job discovery, into the universe every time and into the possible set when the session's data supports it. Choose the specifier convention next. A stage that runs once per source carries the source identifier, and a stage that runs once per session carries an empty specifier.
3. Order the job in the pipeline's prerequisite mapping, where an independent stage maps to no upstream job.
4. Execute the stage inside the pipeline entry point, under the branch that the job identifier selects.
5. Declare the cores one of its jobs occupies. The resolver refuses to admit a job type that declares none, and says so.
6. Add its sizing model and route the job type to that model. A job with no resolvable resource figures is a hard error rather than a job that runs at a default size.
7. Declare a concurrency ceiling only when the job type's throughput plateaus before its cores do, and a reservation only when the type should hand capacity back. The core and memory budgets alone bound a type named in neither.

Adding a New Processing Pipeline

A pipeline that reads a new class of acquired data is agnostic, so it becomes a category package.

1. Add the tracker filename to ProcessingTrackers and the output directory to ProcessedData in sollertia-shared-assets, because a per-session pipeline records beside the output it produces.
2. Add the ProcessingPipelines member and, for a per-session pipeline, its tracker location. The session pipeline listing derives from that mapping, so the two never disagree about the pipelines a session carries.
3. Give a per-session pipeline its project manifest column. The pipeline-to-column mapping is checked at import against the session pipeline listing, so a pipeline that declares no column fails the moment the manifest module loads, and a pipeline that processes no single session declares none. Five further rosters name the manifest's columns, which are the artifact schema, the row accumulator the generation pass fills, the column list the printed summary selects, and the breakdown axes and listing fields the manifest MCP tool reads. That same import check holds all five rosters to the declared status columns and to the schema, so a roster missing the new column fails, and so does one still naming a column the schema no longer declares.
4. Create the category package. Its __init__.py exports the pipeline entry point, every stage job name, and the job discovery and prerequisite callables the orchestration layer binds.
5. Add the pipeline to the batch pipeline set, then give it a dispatch entry. That entry supplies its unit kind, unit loader, job discovery, batch worker, prerequisites, tracker path, output path, unit name, sizing pass, remote command renderer, any priming hook, and any external output paths. An import-time check fails on either half alone, and it also refuses an entry declaring an unknown unit kind.
6. Declare the core allocation and the sizing model of every job type the pipeline resolves, following Adding a New Processing Stage.
7. Add the pipeline's CLI surface, which is a slf process subcommand for a per-session processing pipeline of that shape and a root-level command otherwise, and its MCP surface. Then add the pipeline to the admission policy of every system whose sessions complete it before forging.

Adding an MCP Tool

Place the tool in the *_tools.py module under src/sollertia_forgery/interfaces/ that owns its domain, decorate it with @mcp.tool(), and return through the shared ok_response and error_response helpers. Document the response key shape in the tool's Returns docstring section, since it is part of the public contract. The server discovers tool modules by their filename suffix, so a new module needs no edit to the server itself. Add a new module to the coverage omit list in pyproject.toml, because tool modules reach infrastructure that only a live MCP session supplies.

Automation Troubleshooting

Note, on macOS, a test run in which most tests fail with ValueError: No threading layer could be loaded holds no loadable OpenMP runtime. The hint that error carries names intel-openmp, which the macOS threading layer does not use. Run slf omp --yes from the activated development environment instead.

Many packages used in tox automation pipelines (uv, mypy, ruff) and tox itself may experience runtime failures. In most cases, this is related to their caching behavior. If an unintelligible error is encountered with any of the automation components, deleting the corresponding cache directories (.tox, .ruff_cache, .mypy_cache, etc.) manually or via a CLI command typically resolves the issue.

---

Versioning

This project uses semantic versioning. See the tags on this repository for the available project releases.

---

Authors

• Ivan Kondratyev (Inkaros)
• Kushaan Gupta (kushaangupta)
• Natalie Yeung

---

License

This project is licensed under the Apache 2.0 License: see the LICENSE file for details.

---

Acknowledgments

• All individuals who contributed to the development of this library, directly or indirectly.
• The creators of all other dependencies and projects listed in the pyproject.toml file.