SciStack

pip install scistack # Installs tools for Python and MATLAB

Better Research Tools, Better Research Outcomes

SciStack is a suite of tools built by scientists for scientists of nearly any discipline to enhance data analysis pipelines. SciStack focuses on the common case of embarrassingly parallel workflows used to process nested datasets. It has three core goals:

1. Minimalism: Minimize wasted time and effort, with a focus on reducing boilerplate code and offering advanced capabilities.
2. Reusability: With a near-zero boilerplate format and support for cookie-cutter pipelines, SciStack pipelines are highly interoperable between projects. Thus, SciStack encourages scientific code reuse, an essential yet underutilized scientific work product.
3. Openness: Minimize lock-in to the SciStack framework. SciStack code never touches scientific code, so you can easily change your data processing pipeline architecture at any time.

Minimal Example

Imagine that I am running a study involving a dataset containing Subjects each performing multiple Conditions, and each Condition contains multiple Trials (a common setup in my home field of biomechanics). Below is an example minimal data processing pipeline showcasing SciStack's features.

import scifor
import numpy as np
import pandas as pd

schema = ["subject", "condition", "trial"] # The structure of the dataset
scifor.set_schema(schema) # Tell SciStack about the schema

def example_loading_function(filepath: str) -> np.ndarray:
  """Data loading logic here."""
  return np.ndarray(pd.read_csv(filepath))

def example_processing_function(val: np.ndarray, const: float) -> np.ndarray:
  """Do some processing on some inputs."""
  return val + const

# Load every file matching "path/to/{subject}/{condition}/{trial}_data.ext"
# Returns a df with columns ["subject", "condition", "trial", "loaded_variable"]
loaded_df = scifor.for_each(example_loading_function,
  filepath=scifor.PathInput("path/to/{subject}/{condition}/{trial}_data.ext"),
  outputs=["loaded_variable"]
)

# Process every subject, condition, & trial combination.
# Returns a df with columns ["subject", "condition", "trial", "procesed_variable"]
processed_df = scifor.for_each(example_processing_function,
  val=scifor.ColumnSelection(loaded_df, "variable_to_process"), const=5,
  subject=[], condition=[], trial=[], # Tell SciStack that `my_processing_function` should receive only one trial's data at a time.
  outputs=["processed_variable"]
)

SciStack Design

For any project, it's essential to pick the right tool for the job. SciStack - as the name implies - contains a suite of tools in a "stack" of increasing complexity and weight. There are three main tools in this stack:

• scifor: The lightest-weight level. Syntactical sugar around nested for loops. Operates on in-memory variables only (no file IO) just like standard functions.
• scidb: Wraps scifor, adds a SQL database for data save/load and an auditable data processing history
• scistack GUI: Wraps scidb, adds a GUI to manage complex pipelines.

scifor: syntactic sugar around for loops

Imagine you are conducting a study of human subjects walking. Each Subject comes in to the lab for multiple Sessions, and in each Session they perform multiple Trials of walking. During each Trial, you measure their speed every 0.1 seconds and store that data to one .csv file for each trial.

Example scifor Pipeline Data Loading Step

import scifor
import pandas as pd

# Tell `scifor` about the structure of this dataset
scifor.set_schema(["subject", "session", "trial"]) # ordered one-to-many

def load_data(file_path: str) -> pd.DataFrame:
    """Example logic to load the data"""
    return pd.read_csv(file_path)

path_template = scifor.PathInput("path/to/data/{subject}/{session}/{trial}.csv")
loaded_df = scifor.for_each(load_data,
    file_path=path_template,
    subject=[], session=[], trial=[],
    output_names=["Loaded"]
)

In this example step, after defining a basic load_data() function, we:

1. Defined a scifor.PathInput, providing a template to load all of the files of interest.
2. Invoked the main command scifor.for_each(), providing load_data as the function, file_path as the input variable, and specifying to run load_data once over every combination of subject, session, and trial that match the scifor.PathInput path template.
3. loaded_df is a pd.DataFrame with one row per subject, session, and trial combination. The data is stored into the "Loaded" field specified in the optional output_names parameter (default output name: "value").

Example scifor Pipeline Data Processing Step

import numpy as np

def process_data(speed: np.ndarray) -> np.ndarray:
    """Square every data point"""
    return np.square(speed)

squared_df = scifor.for_each(process_data,
    speed=loaded_df,
    subject=[], session=[], trial=[]
)

scifor.for_each automatically parses loaded_df, repeatedly inputting only the speed values for one combination of subject, session, trial, allowing process_data() to remain very simple and ignore the structure of this project's dataset.

To see more, refer to the scifor docs.

scidb

scistack GUI