CARWatch — Python tools for processing CARWatch and saliva data

CARWatch supports the processing of app-recorded sampling logs and their integration with saliva biomarkers. It is designed for ambulatory sampling studies in which researchers need auditable sampling times, protocol deviations, manual diary fallbacks, and biomarker features.

• Import CARWatch logs from CSV files, ZIP archives, and participant folders.
• Reconstruct registration-aware study days and sampling positions.
• Review conversion anomalies in a structured, editable two-pass issue report.
• Patch missing timestamps from a wide-format manual measurement diary.
• Load Study Manager exports and simplified study results.
• Merge saliva measurements by physical sample ID or scheduled sample position.
• Correct documented tube swaps and compute cortisol response features.

Documentation: User guides and API reference

Working with coding agents: AGENTS.md and the repository-local skills provide task-specific guidance for raw-log processing, saliva analysis, and package development. See the agent-assisted workflow guide.

What CARWatch produces

CARWatch makes sampling adherence visible at the participant-day level. The timeline shows the recorded sampling times, protocol targets, app-updated targets, timing deviations, and the source of each timestamp.

Conversion anomalies can be resolved in a spreadsheet or in an optional, interactive notebook editor. The editor limits decisions to actions valid for the selected issue and refreshes the queue after upstream corrections.

Installation

CARWatch requires Python 3.10 or newer.

Using uv in an existing Python project:

uv add carwatch

Using pip in an existing virtual environment:

pip install carwatch

Installing the latest version from GitHub

The current development version can be installed directly from GitHub:

uv add "carwatch @ git+https://github.com/carwatch-tools/carwatch-python.git"

The main branch can contain unreleased or unstable changes. To work on the source code itself:

git clone https://github.com/carwatch-tools/carwatch-python.git
cd carwatch-python
uv sync

Not familiar with Python? Don't worry!

The beginner setup tutorial explains how to install everything you need to use CARWatch on macOS, Linux, and Windows, including installing uv, creating an isolated CARWatch analysis environment, installing the package, and running Jupyter notebooks. No existing Python installation or manual environment activation is required.

Typical workflow

The complete workflow separates raw-log reconstruction, researcher decisions, and downstream saliva analysis. The examples below build on the variables created in the preceding subsection.

1. Load raw CARWatch logs

For the common layout with one folder per participant, map the study-specific folder names to the participant IDs used in the CARWatch filenames:

from pathlib import Path

import carwatch as cw

participant_folders = {
    "vp01": Path("data/carwatch/vp01"),
    "vp02": Path("data/carwatch/vp02"),
}

raw_logs, source_audit = cw.io.load_raw_logs_from_participant_folders(
    participant_folders,
    create_report=True,
)

source_audit lists every selected or excluded CSV or ZIP source, the reason for that choice, and the raw-event count for each selected source. If the relevant files are already known, load them directly instead:

raw_logs = cw.io.load_raw_logs(
    ["data/carwatch/vp01.csv", "data/carwatch/vp02.zip"],
)

2. Reconstruct the study and create an issue report

The first conversion pass reconstructs registrations, canonical study days, awakening times, and scheduled samples. In warning mode it returns usable results while reporting unresolved anomalies:

initial_results, conversion_report = (
    cw.logs.convert_raw_logs_to_study_manager_summary(
        raw_logs,
        errors="warn",
        create_report=True,
    )
)

conversion_report["issues"].to_csv("conversion_issues.csv")
print(conversion_report["summary"])

The exported report is a resolution queue. Its predefined accept decisions are proposals only and are not applied during this first pass.

3. Review and apply issue decisions

Review conversion_issues.csv in a spreadsheet editor. Keep accept to execute the proposed action, select another documented decision, or clear the cell to leave the issue unresolved.

If missing timestamps should be taken from a manual measurement diary, load the wide diary before the second conversion:

decisions = cw.logs.load_conversion_issue_report("conversion_issues.csv")
manual_diary = cw.io.load_manual_diary("manual_diary.csv")
checker = cw.compliance.SamplingComplianceChecker(
    awakening_delay_tolerance_min=5,
    sampling_delay_tolerance_min=5,
    absolute_time_tolerance_min=15,
)

study_results, final_report = (
    cw.logs.convert_raw_logs_to_study_manager_summary(
        raw_logs,
        errors="raise",
        create_report=True,
        issue_decisions=decisions,
        manual_diary=manual_diary,
        compliance_checker=checker,
    )
)

Omit manual_diary when no accepted decision uses it. With errors="raise", the second pass stops if a submitted decision is invalid or an issue remains unresolved. Sampling compliance is checked by default. Relative samples are checked against awakening or the preceding sample; fixed-time samples are checked against their registered clock time. Supply sampling_schedule when the registration contains no complete timing schedule, or set check_compliance=False to skip this assessment.

4. Inspect and save canonical study results

Create focused day-level and sample-level tables for quality control:

study_days = cw.logs.extract_day_summary_from_summary(study_results)
study_samples = cw.logs.extract_sample_events_from_summary(study_results)
analysis_results = cw.compliance.drop_non_compliant_samples(study_results)

cw.io.save_study_results(study_results, "study_results.csv")

study_days contains dates, awakening information, registration context, and day-level compliance. study_samples contains actual sampling times, minutes since awakening, scheduled and recorded sample IDs, sample positions, timing deviations, sample-level compliance, and mismatch indicators. drop_non_compliant_samples() preserves the canonical wide results format. It clears the complete participant-day when any sample failed. Pass drop_entire_day=False to clear only failed sample observations, or drop_unassessed=True to also clear unassessed data.

Visual inspection uses the same final long tables. The timeline is most useful when reviewing an individual record; the other plots provide cohort-level quality control and cortisol-response checks:

fig, ax = cw.plotting.plot_sampling_timeline(
    study_results,
    participant="vp01",
    day="D1",
)

# In Jupyter, inspect any participant-day interactively.
# Requires: uv add "carwatch[interactive]"
from IPython.display import display

timeline_widget = cw.plotting.interactive_sampling_timeline(study_results)
display(timeline_widget)

compliance_summary = cw.compliance.summarize_compliance(study_results)
fig, ax = cw.plotting.plot_compliance_overview(study_results)
fig, ax = cw.plotting.plot_timing_deviation(study_results)

Resolve conversion issues interactively

The structured conversion report can be reviewed directly in Jupyter instead of exporting it to a spreadsheet. Install the interactive extra, create the initial report, and use Refresh remaining issues after changing decisions. The editor retains accepted upstream decisions while showing only issues that still require review.

_, conversion_report = cw.logs.convert_raw_logs_to_study_manager_summary(
    raw_logs, errors="warn", create_report=True
)
editor = cw.logs.interactive_conversion_issue_report(
    raw_logs,
    conversion_report["issues"],
    manual_diary=manual_diary,
)
display(editor.widget)

# After resolving and refreshing the issue queue:
study_results = cw.logs.convert_raw_logs_to_study_manager_summary(
    raw_logs,
    errors="raise",
    issue_decisions=editor.decisions,
    manual_diary=manual_diary,
)

CSV export and cw.logs.load_conversion_issue_report() remain available when decisions need to be reviewed outside Jupyter.

The cohort-level protocol and, when necessary, the detailed registration schedule can optionally be inspected with:

protocol = cw.logs.summarize_protocol(raw_logs)
registration_schedule = (
    cw.logs.extract_registration_schedule_from_raw_logs(raw_logs)
)

5. Restore results in a later analysis

Reload package-generated results:

study_results = cw.io.load_study_results(
    "study_results.csv",
)

Use simple=True only for display or quick inspection. It is intentionally rejected by merge, quality-control, plotting, and feature functions; load the complete result for those operations. If the starting point is a flat export from the CARWatch Study Manager rather than package-generated results, use:

study_results = cw.io.load_study_manager_export(
    "study_manager_export.csv",
)

6. Load or prepare laboratory saliva data

When the laboratory export contains the physical tube IDs registered in CARWatch, use the standard long format participant,sample,cortisol:

saliva = cw.io.load_saliva("cortisol.csv")

When the laboratory export identifies samples by study day and sampling position, perform the study-specific column renaming first and create the required index:

import pandas as pd

saliva = (
    pd.read_csv("cortisol_by_position.csv")
    .rename(columns={"vp_nr": "participant"})
    .set_index(["participant", "day", "sample_position"])
)

All non-index measurement columns must be numeric. Additional metadata such as condition can be retained as an additional named index level.

7. Merge sampling and laboratory data

Merge by physical tube ID:

merged_results = cw.merge.merge_saliva(
    study_results,
    saliva,
    match_on="sample",
    correct_swaps=True,
)

For position-based laboratory data, use:

merged_results = cw.merge.merge_saliva(
    study_results,
    saliva,
    match_on="position",
    correct_swaps=True,
)

The result remains canonical wide Study Results. It retains CARWatch timing, laboratory values at sample level, day-level metadata such as condition, and explicit laboratory availability and tube-swap flags. Save it with cw.io.save_study_results() and load it with cw.io.load_study_results().

Plot the aligned cortisol response using actual CARWatch sampling times:

fig, ax = cw.plotting.plot_saliva_curve(merged_results, value="cortisol")

8. Compute saliva response features

The CARWatch adapter groups the merged samples by participant and day, orders them by sample_position, and uses the actual time_min values:

cortisol_features = cw.saliva.compute_features_from_carwatch(
    merged_results,
    saliva_type="cortisol",
)

The output contains AUCg, AUCi, initial value, maximum value, maximum increase, and slope features. For generic long-format saliva data that did not originate from cw.merge.merge_saliva, use cw.saliva.compute_features() and specify its grouping and sample levels when required.

See the raw-log processing guide for the complete two-pass workflow, manual diary integration, and validation views.

Citation

Report the CARWatch package version used in the analysis. For research using the CARWatch framework, cite:

Richer, R., Abel, L., Küderle, A., Eskofier, B. M., & Rohleder, N. (2023). CARWatch — A smartphone application for improving the accuracy of cortisol awakening response sampling. Psychoneuroendocrinology, 151, 106073. https://doi.org/10.1016/j.psyneuen.2023.106073

The installed package version is available as:

import carwatch

print(carwatch.__version__)

Contributing

Bug reports, feature requests, and reproducible examples belong in the GitHub issue tracker. Changes should include tests and documentation for the affected research workflow.

License

CARWatch is published under the MIT License.

For developers

Install uv, clone the repository, and synchronize the project environment:

git clone https://github.com/carwatch-tools/carwatch-python.git
cd carwatch-python
uv sync

The main development commands are:

uv run poe format      # Format and automatically fix source files.
uv run poe ci_check    # Check formatting and linting.
uv run poe test        # Run the test suite with coverage.
uv run poe docs        # Build the Sphinx documentation.
uv run poe docs_preview

The package dependencies and development dependencies are managed through pyproject.toml. uv sync resolves and installs the versions recorded by the project environment.