Package to efficiently read large fits arrays in object by object
Project description
Package to sequentially efficiently read large fits arrays in object by object
Free software: MIT license
Features
Sequentially read in large fits files, within a given fixed memory limit
Quick usage
The following example shows an example of reading in a large fits hdu within a memory limit of 2048MB, assuming light curves are stored in rows:
import numpy as np
import fitsio
import fitsiochunked as fc
with fitsio.FITS(filename) as infile:
hdu = infile['flux']
napertures = hdu.get_info()['ndim'][0]
mean_flux = np.zeros(napertures)
for chunk in fc.chunks(hdu, memory_limit_mb=2048):
# `chunk` is a namedtuple with `.data` and `.slice` properties
chunk_data = chunk.data
print('Data shape:', chunk_data.shape)
print('Data dtype:', chunk_data.dtype)
chunk_slice = chunk.slice
print('Chunk starting from aperture:', chunk_slice.start)
print('Chunk up to:', chunk_slice.stop)
chunk_mean = np.average(chunk_data, axis=1)
mean_flux[chunk_slice] = chunk_mean
Installation
Install with pip:
pip install git+https://github.com/mindriot101/fitsiochunked
or download and run the setup file:
git clone https://github.com/mindriot101/fitsiochunked
cd fitsiochunked
python setup.py install
When this code matures, it will be put up on pip.
Details
The high level interface is the chunks function, which builds a ChunkedAdapter object wrapping a fitsio.ImageHDU object.
The ChunkedAdapter wraps a fitsio HDU object. When constructed, it becomes a callable which yields the image data in that hdu in chunks.
The chunksize can be set either with with the parameter chunksize which simply yields chunksize rows each time, or with memory_limit_mb which tries (no promises!) to automatically calculate the number of lightcurves that will fit into memory_limit_mb megabytes of memory.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for fitsiochunked-0.0.2.macosx-10.5-x86_64.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6239a95f6e2da03e2a9810b6487167df1c97bd4e883b51de0a7bc1ab2d0455ac |
|
MD5 | 884d3c3fae37f1afbe9dc3cd92dde77f |
|
BLAKE2b-256 | 9a5e5b49b13642cd5fd78c2edcbb89a98ec3f22ec1e40aba00605f70fce53bb7 |
Hashes for fitsiochunked-0.0.2-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5eb649b3dc6778a79b3a9cdae10f45c8ec54c2e5c8ca0112f7356988dc214089 |
|
MD5 | b01d6af166b62b0a7f53c4e69eb27586 |
|
BLAKE2b-256 | afbda84b4c408c5c3cbfcd27acc63c7721b0139c5b1c09207007a4c078e89050 |