xpcs-correlator

Table of contents

• About
• Documentation
• Features
• Requirements
• Installation
• Quickstart
• Configuration / Logging
• Tests
• Contributing
• License
• Contact

About

This package consolidates ongoing development of correlators for XPCS data analysis at ESRF, with a focus on the ID02 and ID10-coh beamlines.

Documentation

The documentation is hosted online: https://mj.gitlab-pages.esrf.fr/xpcs_developments/xpcscorr/

Features

• Dense frames data reference and chunked correlator implementations.
• Calculates g2, g2 errors, and ttcf (2-time correlation function).
• The ttcf calculations support linear binning for t1,t2 format and hybrid linear log binning for age,lag format.
• Designed to handle large frame stacks via chunked (partitioned) processing.
• Supports Dask for both cluster (SLURM) and local parallel execution

Requirements

• Python 3.10+ (recommended: 3.10, 3.11, 3.12)
• numpy
• numba
• dask
• dask_jobqueue
• h5py
• hdf5plugin
• threadpoolctl

Installation

Install in editable/develop mode (recommended during development):

pip install -e .

Install with development extras (for running tests and linters):

pip install -e .[dev]

Install from PyPI the package for regular use:

pip install xpcs-correlator

Quickstart

For a step-by-step walkthrough with examples and runnable code, see the Tutorial in the online documentation: Tutorial.

Basic usage example — adapt to your data shape and correlator options:

import numpy as np
from xpcscorr import correlator_dense_reference, correlator_dense_chunked

# Replace with your frames array; shape here is (n_frames, nx_pixels, ny_pixels)
frames = np.random.random((100,512, 512))
roimask= np.ones((512,512), dtype=bool)

# Run reference  correlator
result_ref = correlator_dense_reference(frames, roimask)

# Run chunked correlator (handles large data in chunks)
extra_options = {'chunks_N': 3}
result_chunked = correlator_dense_chunked(frames, roimask, extra_options=extra_options)

print(type(result_ref), type(result_chunked))

Notes:

• Replace the synthetic frames with your real dataset (HDF5 dataset or numpy array).
• Check correlator function docstrings for exact argument names and options.

Configuration / Logging

Standard Approach (Recommended)

xpcscorr follows Python library best practices and does not configure logging handlers by default. This means your application controls all logging configuration:

import logging
import xpcscorr

# Configure logging in your application
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s %(name)s %(levelname)s: %(message)s'
)

# Now xpcscorr logs will appear according to your configuration
result = xpcscorr.correlator_dense_reference(frames, roimask)

Convenience Function with Dask Worker Support

For quick setup that also configures Dask worker logging, use the setup_logging() convenience function:

import xpcscorr

# Enable both file and console logging (including Dask workers)
xpcscorr.setup_logging(log_to_file=True, log_to_cli=True)

# Or let it read from environment variables (if not set, uses defaults below)
xpcscorr.setup_logging()

Important: setup_logging() sets environment variables that Dask workers inherit, enabling logging in worker processes. Without calling this function (or manually setting environment variables), worker processes will not log.

Environment Variables

The following environment variables control logging behavior:

• XPCSCORR_LOG_TO_FILE: {'0', '1', 'true', 'false', 'yes', 'no'}
  Enable file logging. Default when calling setup_logging() without arguments: '1' (enabled)

• XPCSCORR_LOG_TO_CLI: {'0', '1', 'true', 'false', 'yes', 'no'}
  Enable console logging to stdout. Default when calling setup_logging() without arguments: '0' (disabled)

• XPCSCORR_DEBUG_MEMORY: {'0', '1', 'true', 'false', 'yes', 'no'}
  Enable memory profiling with tracemalloc. Default: '0' (disabled)

You can set these before running your script:

export XPCSCORR_LOG_TO_CLI=1
export XPCSCORR_LOG_TO_FILE=0
export XPCSCORR_DEBUG_MEMORY=1
python my_script.py

Or in your Python code before importing:

import os
os.environ['XPCSCORR_LOG_TO_CLI'] = '1'
import xpcscorr

Log File Locations

When file logging is enabled, log files are created in platform-specific locations:

• Linux: ~/.cache/xpcscorr/xpcscorr.log
• macOS: ~/Library/Logs/xpcscorr/xpcscorr.log
• Windows: %LOCALAPPDATA%\xpcscorr\xpcscorr.log

Dask workers on LocalCluster create additional log files in ./dask-worker-local-logs/worker_{pid}.log when XPCSCORR_LOG_TO_FILE=1. SLURM workers log to stdout (captured in SLURM .out files).

Tests

Run tests with pytest:

pip install -e .[dev]
pytest -q

There are unit tests under tests/ that exercise correlator behavior and core utilities.

Contributing

• Open issues for bugs or feature requests.
• Fork the repo, create a feature branch, add tests, and submit a pull request.
• Keep changes small, document API changes, and add tests for new behavior.

License

This project is licensed under the MIT License — see the LICENSE file for details.

Contact

Maintainer: Maciej Jankowski — maciej.jankowski@esrf.fr