fits2tmp 0.4.0

A way of loading FITS files into temporary filesystems to make them available in memory.

fits2tmp

This project allows you to take a large FITS file and render it as a set of smaller FITS files stored in-memory in a temporary directory, with appropriately updated headers. The program returns a list of the location of these files, which can then be processed, by other programs,perhaps by a parallel processing framework such as Dask, or by using Python's subprocess module to allow feeding those files to a non-Python program with a command line interface (CLI).

The program also allows you to turn a zarr file containing your FITS data into small FITS files, provided you have extracted the FITS file header to a text file.

This can also be done for hdf5 files (again assuming FITS file header data is saved to a text file). Note that if you attempt to do this with Dask running on multiple nodes, it will currently fail, because of a bug in Dask.

Utilities in the program can be used to write headers out to or read them from a text file.

TODO:

• Add a utility to convert FITS files to zarr, including saving the header out to a text file.
• Add hdf5 support, using more native hdf5 metadata storage solutions.
• Add xradio/xarray support.

Installing fits2tmp

The program is available from pypi so can be installed with pip: pip install fits2tmp.

Example Usage

import fits2tmp as f2t

a = f2t.fitstmpfiles("fits", "/path/to/input/data", "/path/to/temporary/filesystem", "path/to/header/", (100, 100, 100), chunks=(20,20,20))

fitstmpfiles takes a large input data set in one of the supported formats, and returns smaller FITS files to a specified directory.

The parameters for fitstmpfiles are:

• filetype: a string describing the filetype; currently supports "fits" and "zarr" fully; "hdf5" only supported on single node runs.
• inputpath: path to the input data
• destination: path to destination directory. It is intended that temporary directories are used, to try to keep the data in-memory, rather than doing potentially expensive disk writes; however, if you wish to write to more permanent storage, that is possible. You will probably want to clean up manually once the run is done, though.
• headerloc: defaults to None. If loading from a FITS file, this isn't required; if you are loading from zarr or hdf5, you need to provide the path to a text file containing the FITS headers. It is recommended that you use the fits2tmp utilities to prepare the header file.
• chunks: defaults to None. You will want to set this to some value that makes sense for the resources on which you're processing the data. Dask recommend that you have at least 10 times as much RAM as the size of the chunk you're processing. You will also want to consider how much overlap you are using when setting the chunk size. If you are particularly concerned about file loading performance, then consider setting the chunk size to a multiple of the block size used in the storage.
• overlap: defaults to None. You will want to set this to some size that makes sense for the data processing you're doing. You probably want to set this to approximately the size of the largest Gaussian kernel you plan to use to detect objects/features.

Utilities

These small methods are provided to make it easy to write out a FITS header to a file, to allow provision of FITS metadata when using other storage solutions. To create a header from an existing FITS file:

from astropy.io import fits
import fits2tmp as f2t

with fits.open('path/to/fits/file') as f:
    header = f[0].header
    d = f2t.header_dict(header)
    f2t.dict2file(d, 'path/to/where/you/save/header')

Note that this process strips most of the history from the header; if you wish to update the history, use some of the astropy.io.fits tools to accomplish this. This tool is purely to provide a minimal amount of metadata to allow storing FITS data in zarr and similar file formats, and also to allow processing of sections of FITS files where it's not desirable to have all processes reading from the same file in storage.

To create a header from text file containing a json dump of a dictionary created from a FITS header:

import fits2tmp as ft2

loaded_header_dict = f2t.file2dict('/path/to/header/txt/file')
header = dict2header(loaded_header_dict)

This leaves you with a valid FITS header