PyATCLI

The analysis library, engine and command line behind PyAT (Python Analysis Tool) — file parsing, signal processing, curve fitting, group statistics, saved state and provenance records, with no GUI and no plotting code of its own: it computes and writes numbers, and PyAT draws them. Any interface (a script, a notebook, python -m PyATCLI) can sit on top of it.

Coming from PhysicsLibrary? This is the same project under a new name, from version 2026.10.10 (PyPI: ZaksPhysicsLibrary → PyATCLI). Change import PhysicsLibrary to import PyATCLI, python -m PhysicsLibrary to python -m PyATCLI and the physicslib command to pyatcli; nothing else about the top-level names changed. CHANGELOG.md has the list, including the modules that moved into folders.

What it does

• Loads lab data from three instrument formats plus generic tabular files:
  • TDT (Tucker-Davis Technologies) fibre photometry tanks
  • Oxysoft / Artinis (Oxymon, OctaMon, PortaMon …) NIRS .txt exports
  • Terranova Prospa .pt2 EFNMR/MRI 2D images
  • Generic Excel / CSV / TSV / plain text, with automatic sub-table detection for side-by-side data layouts on one sheet
• Processes signals — RANSAC-robust motion correction, bleach correction, denoising, Z-score PETH slicing, FFT with peak annotation, slope/segment analysis
• Fits curves — linear, single/double exponential, exponential rise, Gaussian, sinusoidal, and a photon-entanglement visibility model, all via scipy.optimize.curve_fit
• Analyses text field studies — one JSON file per subject with several free-text fields; pick any pair of fields to compare directly (e.g. does the answer to one question track another for the same subject). Word counts, data-quality flagging, sentence-transformers embeddings, an optional delta-vector magnitude between two fields, and an optional paired-similarity metric per pair with a permutation test and a word-count confound check. Domain-agnostic — field names and which pairs to compare are supplied by the caller, nothing is hardcoded to one study
• Validates the similarity metric statistically — Benjamini-Hochberg FDR-corrected permutation-test p-values, Cohen's d effect size, a word-count-controlled OLS regression (statsmodels), a bootstrap confidence interval on the mean, and a leave-one-out sensitivity check, one row per field pair, with docstrings explaining what each statistic means
• Opens a recording the way PyAT does, without the GUI — a TDT block processed to dF/F with its saved splices replayed and its saved markers on top (load_tdt_recording), splicing that re-fits the dF/F on what is left, events grouped by name, and the whole group analysis (run_group_analysis) repeatable from the settings file it saves. A script gets the numbers the app shows
• Runs every analysis by name, and reproduces any result — run_analysis("event_peth", cache, {...}) returns tables whose columns match PyAT's CSV exports. Every export PyAT writes comes with a .provenance.json beside it (software versions, a fingerprint of the recording, its splices and signal names, the pipeline's own settings, the parameters; never recording data), and python -m PyATCLI reproduce <record> runs the analysis again without the app and checks the numbers against the export. run, batch and group do the same from a shell, so any figure from the GUI can be regenerated by a script
• Works in float64, always — TDT hands its streams over as float32, and numpy is not safe on single-precision data of a realistic length (np.polyfit silently loses the slope past ~3 minutes at 1 kHz). Every file load returns float64 and every public function that takes a signal widens its input, so nothing depends on a caller remembering to
• Reports progress on anything slow — loaders, the TDT pipeline, PETH and peak searches and the text-study pipelines take progress=True (a tqdm bar on the console) or a callback receiving (fraction, message), which is how PyAT shows a live percentage; silent by default

Structure

PyATCLI/
  __init__.py              Public API — see below
  numeric.py                The float64 rule: every load, and every public function that takes a signal, works in float64
  models.py                  Parametric model functions and the curve-fit registry (CURVE_FIT_MODELS)
  progress.py                 tqdm-based progress reporting shared by the slow functions (Plan, track)
  _version.py                  Which library version is running (a source checkout reports its own)
  sources/                     Reading a recording from disk
    load.py                      load_dataset(), load_dataset_file(): detect the format, read it, return a Dataset (load_tdt, load_oxysoft ...)
    dataset.py                    Dataset struct, DataFormat enum, format detection
    processing_TDT.py              TDT tank reading, motion + bleach correction, dF/F, denoising, event markers
    processing_oxysoft.py           Oxysoft .txt reading: the Legend-block parser, folder and single-file readers
    tables.py                        Generic Excel/CSV/TSV/text parser with sub-table detection (load_any_file)
    pt2.py                            .pt2 EFNMR/MRI image parser
  recording.py                 A recording as PyAT holds it: saved state, building / splicing / loading it headlessly
  splice.py                     Non-destructive trim / cut-and-stitch of signals and markers
  markers.py                     How event markers are named and grouped by name
  saved_state.py                  Writing the saved state (markers, splices, signal names, highlights, text) — what Save Changes writes
  outputs.py                       Where results go under an Output folder: <type>/<recording> (Analysis)/, never overwriting
  analysis/                         One file per tool, each a pure function of arrays
    shared.py  zscore_peth.py  event_peth.py  auc.py  fft.py  peak_finder.py  curve_fit.py  intervals.py
    group/                            group analysis: the spec (__init__.py), the trials, the mixed models, the report,
                                      the scan of what each recording holds, and the headless run with its settings file
      trials.py  stats.py  report.py  scan.py  run.py
  analyses.py                   Every analysis by name: run_analysis(kind, cache, parameters) -> tables that mirror PyAT's CSV exports
  provenance.py                  How a result was produced, as data: software, the recording's fingerprint and state, pipeline settings, parameters
  reproduce.py                    Run an analysis again from its provenance record and compare it with an export
  results.py, viewplan.py          Reading a finished analysis back from its files (open_results), and what the main plot shows of a recording, as data (plan_view)
  cli/, __main__.py                 The command line: python -m PyATCLI (and the pyatcli command); docs.py writes its reference and man page
  ops/                               The operations the command line and the engine share: JSON in, JSON out
  engine/                            `serve`: the operations over a pipe, with recordings kept open (wire.py, store.py, session.py, server.py), and contract.py: what its callers can count on (ENGINE_CONTRACT.json)
  fieldstudy/                         The text-field study
    study.py                           Grouped-text-field study pipeline (embeddings, delta vector, paired similarity)
    validation.py                       Statistical validation for the paired-similarity metric (permutation test, Cohen's d, regression, bootstrap CI, leave-one-out)

The layers: sources/ reads a format and computes (processing_*), and sources/load.py is the only place that ties format detection to the right reader and wraps its result into a Dataset; recording.py and analysis/ work on plain arrays and dicts — the same functions PyAT calls, which is why a script reproduces what the app shows. There is no GUI and no plotting code in the library: it computes and writes numbers, and PyAT draws them (its figures package), so the library needs no matplotlib.

Installation

pip install PyATCLI

That also installs the pyatcli command (python -m PyATCLI is the same thing). For the development version:

pip install git+https://github.com/zakgm2/PyATCLI.git

As a dependency of another project: PyATCLI>=2026.10.10 in its requirements.txt or pyproject.toml.

Requirements

• Python 3.10+
• numpy, scipy, tdt, pandas, sentence-transformers, statsmodels, scikit-learn, tqdm (installed automatically)
• sentence-transformers pulls in torch/transformers as transitive dependencies — a genuinely heavy install (hundreds of MB) if you only need the signal-processing side; only actually loaded when you call embed_text_fields/run_field_study_pipeline
• openpyxl — only needed for .xlsx/.xls files; imported lazily with a clear error if missing when you actually try to load Excel

Usage

import PyATCLI as pl

# Detect + load a TDT tank or Oxysoft export folder
fmt     = pl.detect_format(folder_path)
dataset = pl.load_dataset(folder_path, fmt)

# Or load a single Oxysoft .txt file directly
dataset = pl.load_dataset_file(file_path)

# Every loader returns the same universal Dataset struct
dataset.source_format   # "TDT" | "Oxysoft"
dataset.sample_rate      # Hz
dataset.signals            # (num_channels, num_samples)
dataset.channel_names        # list[str]
dataset.events                # [{'label': str, 'sample': int}, ...]

# Anything slow takes progress=: True draws a tqdm bar, a callable gets (fraction 0-1, message)
result = pl.process_tdt_folder(folder_path, progress=True)
result = pl.process_tdt_folder(folder_path, progress=lambda fraction, message: print(f"{fraction:4.0%}  {message}"))

# Generic tabular data (Excel/CSV/TSV/text) — returns one GenericTable per
# detected sub-table, since a single sheet can contain several side-by-side
tables = pl.load_any_file(path)
table  = tables[0]
table.headers   # list[str]
table.data      # (n_rows, n_cols) float64, NaN for missing

# Terranova .pt2 EFNMR/MRI image — returns a raw 2D array, not a Dataset
img = pl.load_pt2(path)   # (n, n) float64 (stored as float32 in the file)

# A recording as PyAT holds it: processed to dF/F, its saved splices replayed, its saved markers on top
loaded = pl.load_tdt_recording(folder_path, regression_method="ols")
cache  = loaded["cache"]                        # the same dict the app keeps in ctx.cache
events = pl.group_markers_by_name(cache)         # {"L1P¹": [12.3, 45.6, ...], "Clap": [20.0], ...}

# Splice it — a TDT splice re-fits the dF/F on what is left, exactly as the app does
cut, applied = pl.apply_splices(cache, [{"mode": "cut_out", "start": 100.0, "end": 130.0}])

# A saved group analysis, run again from its settings file
spec, _ = pl.read_group_settings(group_folder)
results = pl.run_group_analysis(spec, out_dir=new_folder)

# Any analysis PyAT offers, by name — the tables have the columns of PyAT's CSV exports
cache  = pl.load_recording(folder_path)["cache"]                 # TDT folder, Oxysoft .txt, or a table file
result = pl.run_analysis("event_peth", cache, {"event": "L1P¹", "pre_s": 10, "post_s": 20})
result.tables["mean_sem"]          # time_from_event_s, mean, sem
result.parameters                   # as run: defaults filled in, the event's times written out
pl.describe_kinds()                  # every analysis and every parameter

# Run one again from the record PyAT wrote beside an export, and check the numbers
rep = pl.reproduce_record("Event PETH_L1P_103522.provenance.json", compare=["Event PETH_L1P_103522.csv"])
rep.ok, rep.warnings

python -m PyATCLI run event_peth "D:/rats/Rat7-250101" --param event="L1P¹" --param pre_s=10 --out out
python -m PyATCLI reproduce "out/Rat7_event_peth.provenance.json" --compare "out/Event PETH.csv"
python -m PyATCLI batch jobs.json --out batch-out        # many analyses over many recordings

run_cli.py in the repository folder starts the same command line without remembering the module name. Run it with nothing (or double-click it) and it opens Git Bash in that folder, shows the commands and leaves the prompt open, as right-click > "Git Bash Here" would; run it with arguments (python run_cli.py info) and it is the command line itself in the terminal you are in (--terminal and --inline choose; see the file's own docstring).

opened = pl.open_results("out/Rec_event_peth.provenance.json")   # a finished analysis read back from its files: a record, a folder, a group's folder, a .csv
opened.kind, opened.parameters, opened.tables["trials"]       # what made it, what it was asked, and its tables (the library draws nothing; PyAT does)

python -m PyATCLI run event_peth "D:/rats/Rat7-250101" --param event="L1P¹" --out out
python -m PyATCLI run event_peth "D:/rats/Rat7-250101" --param event="L1P¹" --output-folder Output   # into Output/TDT/Rat7-250101 (Analysis)/, as PyAT's exports

What the app saves with Save Changes — markers, splices, signal names, highlights, text — a script can save too, as the same files:

pl.saved_state.add_marker("D:/rats/Rat7-250101", 312.5, label="Lever")
pl.saved_state.add_splice("D:/rats/Rat7-250101", "cut_out", 40.0, 55.0)       # stacks on the saved ones; replayed on every open

python -m PyATCLI saved marker "D:/rats/Rat7-250101" --time 312.5 --label Lever
python -m PyATCLI saved show "D:/rats/Rat7-250101"
python -m PyATCLI group setup "D:/cohort" --name Sucrose                  # lists the markers the recordings share, as the wizard does
python -m PyATCLI group setup "D:/cohort" --name Sucrose --marker "L1P¹" --marker "L2P¹" --output-folder Output --run

Every command takes --json (one document on standard output, so ... --json | jq always works), --plain (tab-separated rows), -q, -v, --no-input and --events (progress as JSON lines on standard error); standard output is only ever the result and errors go to standard error with an exit status. python -m PyATCLI with no arguments lists the commands, and each command's --help has examples. CLI_REFERENCE.md and the man page man/pyatcli.1 are generated from the program (python -m PyATCLI docs --format markdown|man) and kept in step by a test. python -m PyATCLI serve runs the same operations as a long-lived engine speaking JSON lines on standard input and output, with recordings kept open between requests: it is what PyAT's window talks to, so none of the numerical work runs in the window's process (the engine).

See API_REFERENCE.md for the analyses, the provenance record's layout and every command, and Drawing results for where the figures went. The command line itself computes and writes files and never draws or opens a window: drawing is PyAT's.

# Analysis
x_seg, z = pl.get_zscore_slice(time_array, signal, center_t, window=30)
freqs, power, seg_x, seg_y = pl.compute_fft_slice(time_array, signal, center_t, fs)
peaks = pl.find_fft_peaks(freqs, power)                  # [{"freq_hz", "power", "bpm"}, ...] — drawing them is up to you

# Curve fitting
result = pl.fit_model_to_segment(x_seg, y_seg, pl.single_exponential_model, p0_fn)
result["popt"], result["r2"], result["y_fit"]

# Text field study — one JSON file per subject, e.g. P-0001.json. Pick
# pairs of fields to compare directly; no grouping concept needed.
fields = pl.peek_fields(folder_path)              # see what fields exist before picking pairs
df = pl.run_field_study_pipeline(
    folder_path,
    text_fields=["q1", "q2", "q3", "q4"],
    delta_pair=("q1", "q2"),                        # optional: how much did q2 change from q1
    paired_fields=[("q1", "q3", "pair1")],           # optional: does q1 track q3
)
# df has one row per subject: wordcount_<field>, low_quality_<field>, delta_magnitude,
# sim_<pair>, pvalue_<pair>, effect_size_<pair>, wc_confound_r_<pair>, ...

# Statistical validation of the paired-similarity metric — one row per pair
summary = pl.run_validation_pipeline(
    folder_path,
    text_fields=["q1", "q3"],
    paired_fields=[("q1", "q3", "pair1")],
)
# summary: p_value, p_value_fdr, cohens_d, wc_coef_a/b + wc_pvalue_a/b,
# regression_r_squared, ci_lower/ci_upper, n_flagged_loo, flagged_participant_ids

See FIELD_STUDY_METHODOLOGY.md for why each statistic in the validation step is a sound, standard technique — useful if anyone asks.

See CHANGELOG.md for the version history.

Public API

Everything importable from PyATCLI directly:

License

AGPL-3.0 — see LICENSE for details.