Skip to main content

ToolVIPER

Tools and utilities for optimized radio astronomy processing using the VIPER framework.

Python 3.12 3.13 3.14 Linux Tests macOS Tests ipynb Tests Version Status

ToolVIPER provides a suite of high-level tools designed to simplify Dask cluster management, logging, profiling, and data distribution for radio astronomy applications. It is part of the VIPER (Very Large Interferometry Post-processing and Exploratory Research) ecosystem.

Features

  • Dask Cluster Management: Easily create and manage local or SLURM-based Dask clusters with optimized configurations.
  • Unified Logging: Structured, colorized, and worker-aware logging system.
  • Resource Profiling: Built-in decorators to monitor CPU and memory usage of functions.
  • Interactive Utilities: Enhanced data display for Jupyter notebooks, including interactive JSON and HTML views for complex dictionaries.
  • Data Management: Simple interface for downloading and managing external datasets from Cloudflare.
  • Parameter Validation: Decorator-based system for validating function arguments using JSON schemas.
  • Graph Distribution: Tools to distribute functions across datasets along specific axes using Dask.

Installation

ToolVIPER is tested on Python 3.12, 3.13 and 3.14 (3.11 is still allowed but no longer tested).

Basic Installation

pip install toolviper

With Optional Dependencies

ToolVIPER offers several optional dependency sets:

  • test: For running tests and linting (pytest, pre-commit, etc.)
  • interactive: For Jupyter/IPython support (jupyterlab, matplotlib, ipympl, etc.)
  • docs: For building documentation (sphinx, nbsphinx, etc.)
  • all: Installs all optional dependencies.

Example:

pip install "toolviper[interactive]"

Quick Start

1. Dask Client Management

Example Notebook

ToolVIPER simplifies setting up a Dask environment.

from toolviper.dask.client import local_client

# Start a local Dask client with 4 workers
client = local_client(cores=4, memory_limit="8GB")
print(client.dashboard_link)

For SLURM clusters:

from toolviper.dask.client import slurm_cluster_client

client = slurm_cluster_client(
    workers_per_node=1,
    cores_per_node=4,
    memory_per_node="16GB",
    number_of_nodes=2,
    queue="normal",
    interface="ib0",
    python_env_dir="/path/to/env",
    dask_local_dir="/tmp/dask",
    dask_log_dir="dask_logs",
)

2. Logging

Example Notebook

ToolVIPER provides a pre-configured logger that can be used throughout your application.

from toolviper.utils import logger

logger.info("Initializing process...")
logger.warning("Low memory detected", verbose=True)

3. Profiling

Monitor your functions' resource consumption using decorators.

from toolviper.utils.profile import cpu_usage


@cpu_usage(filename="my_profile.csv")
def heavy_computation():
    # Your code here
    pass


heavy_computation()

4. Interactive Data Exploration

In a Jupyter notebook, you can use DataDict for better visualization of nested dictionaries.

from toolviper.utils.display import DataDict

my_data = {"level1": {"level2": {"key": "value"}}}
dd = DataDict(my_data)
dd.display()  # Renders an interactive JSON tree in Notebooks

5. Data Management

Access and download external datasets easily.

import toolviper

# List available files
toolviper.utils.data.list_files()

# Get a python list available files
toolviper.utils.data.get_files()

# Download a specific dataset
toolviper.utils.data.download("example_dataset.zip", folder="./data")

6. Parameter Validation

A detailed write-up can be found here: Parameter Validation.

Ensure your functions receive correct arguments using the `@validate` decorator.

from toolviper.utils.parameter import validate


@validate()
def my_function(param1: int, param2: str):
    # This function will automatically validate arguments
    # against a corresponding .param.json schema.
    pass

Development and Testing

To contribute or run tests locally:

  1. Clone the repository:

    git clone https://github.com/casangi/toolviper.git
    cd toolviper
    
  2. Install in editable mode with test dependencies:

    pip install -e ".[test]"
    
  3. Install the pre-commit git hooks (ruff linting/formatting, nbstripout, repo hygiene checks — also enforced by CI):

    pre-commit install
    
  4. Run tests:

    pytest tests
    

License

ToolVIPER is licensed under the terms of the LICENSE file included in the root directory.

Metadata

Release files for toolviper 0.1.14

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for toolviper 0.1.14
File Size Uploaded Provenance Actions
toolviper-0.1.14.tar.gz 67.4 kB Without provenance Details for toolviper-0.1.14.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for toolviper 0.1.14
File Interpreter ABI Platform Actions
toolviper-0.1.14-py3-none-any.whl Python 3 none any Details for toolviper-0.1.14-py3-none-any.whl

Total release size: 120.5 kB

Release files / toolviper-0.1.14.tar.gz

Download URL toolviper-0.1.14.tar.gz
Size 67.4 kB
Tags Source
SHA-256 checksum
How to use checksums
36c2b5ef2d8c2461d90e83fe9c3a821f86a7dd9a7e8595fecc39590c246ba21c
BLAKE2b-256 checksum
How to use checksums
ce467c0f37b2765d34ac8f29bf12c067e8685527146fead6fafc309280df3c35
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / toolviper-0.1.14-py3-none-any.whl

Download URL toolviper-0.1.14-py3-none-any.whl
Size 53.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6effd8a771134f3007904141f26a6c5be3a43053e523912d8054ea056425bc1e
BLAKE2b-256 checksum
How to use checksums
5bfdc4e82262b80ae1ac1889be1c459cc7f6bf571b41ce5aec8b43ac4e9b079a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Security for toolviper 0.1.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page