lsst-daf-butler 30.2025.5000

An abstraction layer for reading and writing astronomical data to datastores.

lsst-daf-butler

LSST Data Access framework described in arXiv:2206.14941. Please cite the SPIE paper when using this software.

This is a Python 3 only package (we assume Python 3.11 or higher).

• SPIE paper from 2024: Converting Rubin Observatory's data butler to a client/server architecture (Tech note)
• SPIE Paper from 2022: The Vera C. Rubin Observatory Data Butler and Pipeline Execution System (arXiv) [Primary reference]
• ADASS paper from 2019: Abstracting the Storage and Retrieval of Image Data at the LSST.
• Early design note: DMTN-056

PyPI: lsst-daf-butler

This software is dual licensed under the GNU General Public License (version 3 of the License, or (at your option) any later version, and also under a 3-clause BSD license). Recipients may choose which of these licenses to use; please see the files gpl-3.0.txt and/or bsd_license.txt, respectively.

Arrow Memory Leaks

From version 18 of arrow we have seen significant memory leaks when accessing parquet files using the default memory allocator. If you see such leaks the workaround is to set the ARROW_DEFAULT_MEMORY_POOL environment variable to jemalloc following the advice from apache/arrow#45882. For EUPS users this variable is automatically set.