This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

Package to sequentially efficiently read large fits arrays in object by object

  • Free software: MIT license

Features

  • Sequentially read in large fits files, within a given fixed memory limit

Quick usage

The following example shows an example of reading in a large fits hdu within a memory limit of 2048MB, assuming light curves are stored in rows:

import numpy as np
import fitsio
import fitsiochunked as fc

with fitsio.FITS(filename) as infile:
    hdu = infile['flux']
    napertures = hdu.get_info()['ndim'][0]
    mean_flux = np.zeros(napertures)

    for chunk in fc.chunks(hdu, memory_limit_mb=2048):

        # `chunk` is a namedtuple with `.data` and `.slice` properties
        chunk_data = chunk.data
        print('Data shape:', chunk_data.shape)
        print('Data dtype:', chunk_data.dtype)


        chunk_slice = chunk.slice
        print('Chunk starting from aperture:', chunk_slice.start)
        print('Chunk up to:', chunk_slice.stop)

        chunk_mean = np.average(chunk_data, axis=1)
        mean_flux[chunk_slice] = chunk_mean

The library copes with an aribtrary number of hdus:

import numpy as np
import fitsio
import fitsiochunked as fc

with fitsio.FITS(filename) as infile:
    hjd_hdu = infile['hjd']
    flux_hdu = infile['flux']
    fluxerr_hdu = infile['fluxerr']

    napertures = flux_hdu.get_info()['ndim'][0]
    mean_flux = np.zeros(napertures)

    for chunks in fc.chunks(hjd_hdu, flux_hdu, fluxerr_hdu, memory_limit_mb=2048):
        # chunks is a tuple of chunks
        hjd_chunk, flux_chunk, fluxerr_chunk = chunks

        # `chunk` is a namedtuple with `.data` and `.slice` properties
        flux_chunk_data = flux_chunk.data
        print('Data shape:', flux_chunk_data.shape)
        print('Data dtype:', flux_chunk_data.dtype)

        # and so on

Note: if multiple hdus are supplied, then the memory_limit_mb and chunksize arguments to chunks apply to each HDU i.e. three HDUs and a memory limit of 2048MB will lead to 3x2048 = 6144MB of memory used.

Installation

Install with pip:

pip install fitsiochunked
# or get the latest development version from github
pip install git+https://github.com/mindriot101/fitsiochunked

or download and run the setup file:

git clone https://github.com/mindriot101/fitsiochunked
cd fitsiochunked
python setup.py install

Details

The high level interface is the chunks function, which builds a ChunkedAdapter object wrapping a fitsio.ImageHDU object.

The ChunkedAdapter wraps a fitsio HDU object. When constructed, it becomes a callable which yields the image data in that hdu in chunks.

The chunksize can be set either with with the parameter chunksize which simply yields chunksize rows each time, or with memory_limit_mb which tries (no promises!) to automatically calculate the number of lightcurves that will fit into memory_limit_mb megabytes of memory.

Release History

Release History

0.2.0

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.0.5

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.0.4

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.0.3

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.0.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
fitsiochunked-0.2.0-py2.py3-none-any.whl (5.9 kB) Copy SHA256 Checksum SHA256 py2.py3 Wheel Oct 22, 2016
fitsiochunked-0.2.0.tar.gz (6.3 kB) Copy SHA256 Checksum SHA256 Source Oct 22, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting