cisl 0.2.3

Ensemble of tools for CISL data

cisl

This project is managed with uv.

Setup

Install the project and development dependencies:

uv sync --dev

uv will create a local virtual environment in .venv and use the Python version declared in .python-version (3.12).

Test Data

Some tests depend on sample datasets under tests/samples/, and those files are tracked with Git LFS rather than regular git blobs.

After cloning the repository, fetch them with:

git lfs install
git lfs pull

If files under tests/samples/ look like small pointer text files instead of real binary data, Git LFS content has not been downloaded yet.

Common commands

Run the CLI:

uv run repack <path-to-dataset>

Run a module or script with the project environment:

uv run python -m cisl.scripts.repack --help

Install a Jupyter kernel for the synced environment:

uv run python -m ipykernel install --user --name cisl-py

ENVI I/O

The ENVI reader supports both eager and lazy access.

from cisl.io import envi

eager = envi.open("dataset.hdr")
lazy = envi.open("dataset.hdr", lazy=True)
chunked = envi.open(
    "dataset.hdr",
    chunks={"wavenumber": 16, "y": 512, "x": 512},
)

Implementation detail:

• spectral is still used for ENVI header parsing and file creation.
• The data path uses a direct numpy.memmap backend.
• Lazy reads wrap that memmap with dask.array, so slices and reductions can stay out-of-core until you call .compute().

Writes are also chunk-friendly:

envi.save("out.hdr", chunked)

If the input is dask-backed, envi.save writes one chunk at a time into an ENVI memmap instead of materializing the full cube first.

Dask Memory Notes

Dask delays work, but it does not make large arrays free. The common ways to still run out of memory are:

• calling .values, .to_numpy(), or .compute() on the full cube
• using chunks that are too large for the operation
• triggering implicit rechunking or other operations that need big temporary arrays

Practical guidance:

• choose chunks when opening the file
• slice or reduce before computing
• avoid converting the whole array to NumPy unless you really need it
• keep chunk sizes moderate and aligned with how you plan to process the data

Notebook Demo

See notebooks/demos/Read core_2x2 ENVI.ipynb for a small example that opens the sample core_2x2.hdr dataset lazily and reads a spectrum plus a single-band image.