Manticore Data Module

A Python module for loading and managing data from an S3 bucket.

ATTENTION (April 30th 2026): BACKEND CHANGED. PLEASE UPDATE TO MANTICORE_DATA >=1.8.0

Overview

The manticore_data module is designed to load and manage data from an S3 bucket. It provides functionalities for handling different types of data products such as velocity fields, monofonic data, submit scripts, and whitenoise data.

Installation

To install the module, clone the repository and install the required dependencies:

pip install git+https://git.aquila-consortium.org/Aquila-Consortium/manticore_data/

Usage

Initialization

First, import the module and initialize the ManticoreDataLoader class. The campaign generation is chosen once, in the constructor, and the loader is bound to it for its whole lifetime — no method takes a generation argument. Create a second loader to read a different generation.

from manticore_data import ManticoreDataLoader, ManticoreGeneration

loader = ManticoreDataLoader(ManticoreGeneration.Local)

Loading Data

To load Halo catalog data from the S3 bucket:

data = loader.get_soap_catalog_files(resolution=1024, snapshot_number=70)

To retrieve a file element of the main chain:

mcmc = loader.get_product_files("chain", as_file_objects=True)

Be aware that these files are about 400 MB big each.

One

Configuration

The module uses a Settings class to manage configuration settings. You can configure the mirror URL and cache directory by modifying the .env file or setting environment variables.

from manticore_data.settings import settings

print(settings.manticore_data_mirror)
print(settings.cache_dir)

Cache management

To clear the cache entirely:

from manticore_data import ManticoreFile

ManticoreFile.clear_cache()

To get information on the cache

ManticoreFile.get_cache_info()

License

This project is licensed under the GPL License - see the LICENSE file for details.