Skip to main content

A pythonic interface for radio astronomy interferometry data

Project description

pyuvdata

CircleCI testing codecov DOI

pyuvdata defines a pythonic interface to interferometric data sets. Currently pyuvdata supports reading and writing of miriad, uvfits, CASA measurement sets and uvh5 files and reading of FHD (Fast Holographic Deconvolution) visibility save files, SMA Mir files and MWA correlator FITS files.

Documentation of the API, conventions used throughout the package, and a tutorial is hosted on ReadTheDocs.

Motivation

The main goals are:

  1. To provide a high quality, well documented path to convert between file formats.
  2. Support the direct use of interferometric datasets from python with minimal software.
  3. Provide precise data definitions and convention details via both human readable code and high quality documentation.

Package Details

pyuvdata has four major user classes:

  • UVData: supports interferometric data (visibilities) and associated metadata
  • UVCal: supports interferometric calibration solutions (antenna-based) and associated metadata.
  • UVBeam: supports primary beams (E-field or power) and associated metadata. A number of analytic beam objects are also available and the BeamInterface object serves as a unified interface for UVBeam and analytic beam objects.
  • UVFlag: A class to handle the manipulation and combination of flags for data sets. Also can convert raw data quality metrics into flags using thresholding. (This object is somewhat new and experimental. Consider it to be a beta version)

UVData File standard notes

  • Miriad has been thoroughly tested with aipy-style Miriad files and minimally tested with ATCA files. Reading/writing Miriad files is not supported on Windows.
  • UVFITS conforms to AIPS memo 117 (as of March 2020). It is tested against FHD, CASA, and AIPS. However AIPS is limited to <80 antennas and CASA's importuvfits task does not seem to support >255 antennas. Because of this and other limitations to CASA's importuvfits task, we reccommend that users planning to work in CASA avoid using importuvfits and use the measurement set writer instead.
  • CASA measurement sets, primarily conforming to CASA Memo 229, with some elements taken from the proposed v3.0 format documented in CASA Memo 264. Measurement sets are tested against datasets from VLA, MWA (filled via cotter), ALMA, and SMA (filled using the importuvfits task). Extensive loopback testing has been done to verify that pyuvdata written measurement sets are compatible with CASA.
  • UVH5 is an HDF5-based file format defined by the HERA collaboration, details in the uvh5 memo. Note that this is a somewhat new format, so it may evolve a bit but we will strive to make future versions backwards compatible with the current format. It is probably not compatible with other interferometric HDF5 files defined by other groups.
  • FHD (read-only support, tested against MWA and PAPER data)
  • MIR (read-only support, though experimental write functions are available, tested against SMA data)
  • MWA correlator FITS files (read-only support, tested against Cotter outputs and FHD)

UVCal file formats

  • CalH5: a new an HDF5-based file format defined in pyuvdata, details in the calh5 memo. Note that this format is a somewhat new format, so it may evolve a bit but we will strive to make future versions backwards compatible with the current format.
  • Measurement Set calibration files (read and write, gains/delay/bandpass supported, beta version). Tested against a limited set of SMA, LWA, and VLA calibration files generated in CASA.
  • CalFITS: a custom format defined in pyuvdata, details in the calfits memo. Note that this format does not support all possible UVCal objects, so we prefer CalH5 which has full support for all variations of UVCal objects.
  • FHD calibration files (read-only support)

UVBeam file formats

  • BeamFITS: a custom format defined in pyuvdata that supports both regularly gridded beams and beams on a HEALPix grid for both E-field and power beams, details in the beamfits memo.
  • CST beam text files (read only support) with a defined yaml file format for metadata, details here: cst settings file
  • MWA Beams (read only support)

Known Issues and Planned Improvements

  • Incorporating numba to alleviate bottlenecks and to replace some existing cython extensions as appropriate.

See our issue log for a full list.

Community Guidelines

Contributions to this package to add new file formats or address any of the issues in the issue log are very welcome, as are bug reports and feature requests. Please see our guide on contributing

Telescopes

pyuvdata has been used with data from the following telescopes. If you use it on data from a telescope we don't have listed here please let us know how it went via an issue! We would like to make pyuvdata generally useful to the community for as many telescopes as possible.

  • MWA
  • HERA
  • PAPER
  • LWA
  • ALMA
  • VLA
  • ATCA
  • SMA
  • ATA
  • VLBA

Versioning

We use a generation.major.minor version number format. We use the generation number for very significant improvements or major rewrites, the major number to indicate substantial package changes (intended to be released every 3-4 months) and the minor number to release smaller incremental updates (intended to be released approximately monthly and which usually do not include breaking API changes). We do our best to provide a significant period (usually 2 major generations) of deprecation warnings for all breaking changes to the API. We track all changes in our changelog.

History

pyuvdata was originally developed in the low frequency 21cm community to support the development of and interchange of data between calibration and foreground subtraction pipelines. Particular focus has been paid to supporting drift and phased array modes.

Citation

Please cite both of our JOSS papers:

Keating et al., (2025). pyuvdata v3: an interface for astronomical interferometric data sets in Python. Journal of Open Source Software, 10(109), 7482, https://doi.org/10.21105/joss.07482 [ADS Link]

Hazelton et al, (2017), pyuvdata: an interface for astronomical interferometeric datasets in python, Journal of Open Source Software, 2(10), 140, doi:10.21105/joss.00140 [ADS Link]

Installation

Simple installation via conda is available for users, developers should follow the directions under Developer Installation below.

For simple installation, the latest stable version is available via conda (preferred: conda install -c conda-forge pyuvdata) or pip (pip install pyuvdata).

There are some optional dependencies that are required for specific functionality, which will not be installed automatically by conda or pip. See Dependencies for details on installing optional dependencies.

Dependencies

Required:

  • astropy >= 6.0
  • docstring_parser>=0.15
  • h5py >= 3.7
  • numba >= 0.57.0
  • numpy >= 1.23
  • pyerfa >= 2.0.1.1
  • python >= 3.11
  • pyyaml >= 5.4.1
  • scipy >= 1.9
  • setuptools_scm >= 8.1

Optional:

  • astropy-healpix >= 1.0.2 (for working with beams in HEALPix formats)
  • astroquery >= 0.4.4 (for enabling phasing to ephemeris objects using JPL-Horizons)
  • hdf5plugin >= 3.3.1 (for enabling bitshuffle and other hdf5 compression filters in uvh5 files)
  • lunarsky >=0.2.5 (for working with simulated datasets for lunar telescopes)
  • novas and novas_de405 (for using the NOVAS library for astrometry)
  • python-casacore >= 3.5.2 (for working with CASA measurement sets)

The numpy and astropy versions are important, so make sure these are up to date.

We suggest using conda to install all the dependencies. If you want to install python-casacore and astropy-healpix, you'll need to add conda-forge as a channel (conda config --add channels conda-forge).

If you do not want to use conda, the packages are also available on PyPI (python-casacore may require more effort, see details for that package below). You can install the optional dependencies via pip by specifying an option when you install pyuvdata, as in pip install pyuvdata[healpix] which will install all the required packages for using the HEALPix functionality in pyuvdata. The options that can be passed in this way are:

  • astroquery
  • casa
  • cst
  • hdf5_compression
  • healpix
  • lunar
  • novas
  • all
  • test
  • doc
  • dev

The first set (astroquery, casa, hdf5_compression, healpix, lunar, and novas) enable various specific functionality while all will install all of the above to enable all functionality. The last three (test, doc and dev) include everything installed with all plus packages for testing and building the docs which may be useful for developers of pyuvdata.

Installing python-casacore

python-casacore requires the casacore c++ libraries. It can be installed easily using conda (python-casacore on conda-forge).

If you do not want to use conda, the casacore c++ libraries are available for ubuntu through the kern suite. On OSX, casacore is available through the ska-sa brew tap. The python-casacore library (with manual install instructions) is available at https://github.com/casacore/python-casacore

Developer Installation

Clone the repository using git clone https://github.com/RadioAstronomySoftwareGroup/pyuvdata.git

Navigate into the pyuvdata directory and run pip install . (note that python setup.py install does not work). Note that this will attempt to automatically install any missing dependencies. If you use conda or another package manager you might prefer to first install the dependencies as described in Dependencies.

To install without dependencies, run pip install --no-deps .

To compile the binary extension modules such that you can successfully run import pyuvdata from the top-level directory of your Git checkout, run: python setup.py build_ext --inplace

If you want to do development on pyuvdata, in addition to the other dependencies you will need the following packages:

  • pytest >= 8.2
  • pytest-cases >= 3.9.1
  • pytest-cov
  • cython == 3.0
  • coverage
  • pre-commit
  • matplotlib
  • sphinx
  • pypandoc

One other package, pytest-xdist, is not required, but can be used to speed up running the test suite by running tests in parallel. To use it call pytest with the -n auto option.

One way to ensure you have all the needed packages is to use the included environment.yaml file to create a new environment that will contain all the optional dependencies along with dependencies required for testing and development (conda env create -f environment.yaml). Alternatively, you can specify test, doc, or dev when installing pyuvdata (as in pip install pyuvdata[dev]) to install the packages needed for testing (including coverage and linting) and documentation development; dev includes everything in test and doc.

To use pre-commit to prevent committing code that does not follow our style, you'll need to run pre-commit install in the top level pyuvdata directory.

Tests

Uses the pytest package to execute test suite. From the source pyuvdata directory run pytest or python -m pytest.

Testing of UVFlag module requires the pytest-cases plug-in.

API

pyuvdata is organized around objects that contain all the data and metadata required to represent and work with interferometric data, calibration solutions, flags, antenna beams and telescope layouts. Each object has the data and metadata as attributes along with many useful methods for importing and exporting files and manipulating and transforming the data in useful ways. Please see our extensive documentation on ReadTheDocs for tutorials and complete API details.

Maintainers

pyuvdata is maintained by the RASG Managers, which currently include:

  • Adam Beardsley (Winona State University)
  • Bryna Hazelton (University of Washington)
  • Garrett "Karto" Keating (Smithsonian Astrophysical Observatory)
  • Daniel Jacobs (Arizona State University)
  • Matt Kolopanis (Arizona State University)
  • Paul La Plante (University of Nevada, Las Vegas)
  • Jonathan Pober (Brown University)

Please use the channels discussed in the guide on contributing for code-related discussions. You can contact us privately if needed at rasgmanagers@gmail.com.

Acknowledgments

Support for pyuvdata was provided by NSF awards #1835421 and #1835120.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyuvdata-3.2.2.tar.gz (46.0 MB view details)

Uploaded Source

Built Distributions

pyuvdata-3.2.2-cp313-cp313-win_amd64.whl (41.8 MB view details)

Uploaded CPython 3.13Windows x86-64

pyuvdata-3.2.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (45.0 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ x86-64

pyuvdata-3.2.2-cp313-cp313-macosx_11_0_arm64.whl (42.8 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

pyuvdata-3.2.2-cp313-cp313-macosx_10_13_x86_64.whl (42.8 MB view details)

Uploaded CPython 3.13macOS 10.13+ x86-64

pyuvdata-3.2.2-cp312-cp312-win_amd64.whl (41.8 MB view details)

Uploaded CPython 3.12Windows x86-64

pyuvdata-3.2.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (45.1 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ x86-64

pyuvdata-3.2.2-cp312-cp312-macosx_11_0_arm64.whl (42.8 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

pyuvdata-3.2.2-cp312-cp312-macosx_10_13_x86_64.whl (42.8 MB view details)

Uploaded CPython 3.12macOS 10.13+ x86-64

pyuvdata-3.2.2-cp311-cp311-win_amd64.whl (41.8 MB view details)

Uploaded CPython 3.11Windows x86-64

pyuvdata-3.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (45.1 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

pyuvdata-3.2.2-cp311-cp311-macosx_11_0_arm64.whl (42.8 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

pyuvdata-3.2.2-cp311-cp311-macosx_10_9_x86_64.whl (42.8 MB view details)

Uploaded CPython 3.11macOS 10.9+ x86-64

File details

Details for the file pyuvdata-3.2.2.tar.gz.

File metadata

  • Download URL: pyuvdata-3.2.2.tar.gz
  • Upload date:
  • Size: 46.0 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for pyuvdata-3.2.2.tar.gz
Algorithm Hash digest
SHA256 a25586b9178a2e0c9fd4ef64f557419d23b80342ef1e3cda2746c9efa35fde93
MD5 78bfc93e024591d56a23690f3ffcc1cd
BLAKE2b-256 7c87a43faf302df8cf6ff52a2af5e30d0afee657172ecb94a1b218f392c7df4a

See more details on using hashes here.

File details

Details for the file pyuvdata-3.2.2-cp313-cp313-win_amd64.whl.

File metadata

  • Download URL: pyuvdata-3.2.2-cp313-cp313-win_amd64.whl
  • Upload date:
  • Size: 41.8 MB
  • Tags: CPython 3.13, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for pyuvdata-3.2.2-cp313-cp313-win_amd64.whl
Algorithm Hash digest
SHA256 c964c3a235c2dec9e4047a752de2b2792587146e4d4dcebe3e8a449e6acf3d72
MD5 4cc65ec0a477a431cd734e8fced289d3
BLAKE2b-256 8943cda9c415c71bac702f270dc61e6ca3f751c41afb6a294f661884cf286123

See more details on using hashes here.

File details

Details for the file pyuvdata-3.2.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pyuvdata-3.2.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 33b02b7156ebc12bf2f9b2a385332f359ae8581f64dafa5921782f84a6a450d7
MD5 e43dfb455553802b0a696c2ddbbd2496
BLAKE2b-256 1b3769b7262e61a6606bfb1876412450f293c72ffdf523901f6840b12580ade2

See more details on using hashes here.

File details

Details for the file pyuvdata-3.2.2-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for pyuvdata-3.2.2-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 aa532e73b652c39fb2eb878b4c90ebcde9cf0e721333c6e667999ce29b681563
MD5 7a905cbc77e0235f615c5f392e496f08
BLAKE2b-256 d13d428b8c5f2ed3cd6952c5b0b5f69e7a194a8be19ff0373ab77bfa05c6b0c6

See more details on using hashes here.

File details

Details for the file pyuvdata-3.2.2-cp313-cp313-macosx_10_13_x86_64.whl.

File metadata

File hashes

Hashes for pyuvdata-3.2.2-cp313-cp313-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 5865e341e6eed85f3631fa9d1ccd22cd4bd6251703ea4b7e83db113974dd7483
MD5 5b6684c768596d5dac3c0178de60bb16
BLAKE2b-256 f5873817a8877d9cca37d8d7a5f9e9f2d6eccb883586c364c47502c1e5f17b67

See more details on using hashes here.

File details

Details for the file pyuvdata-3.2.2-cp312-cp312-win_amd64.whl.

File metadata

  • Download URL: pyuvdata-3.2.2-cp312-cp312-win_amd64.whl
  • Upload date:
  • Size: 41.8 MB
  • Tags: CPython 3.12, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for pyuvdata-3.2.2-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 7a482f8d4ba99c457d292aa98e4341baff6ef449264e3158e3433c515263b675
MD5 898884796585305c49ccf3b5b6bbe93f
BLAKE2b-256 7704394527c39e5c18173f297698177eee8f78d6ee5a31f046c91fba99ac5224

See more details on using hashes here.

File details

Details for the file pyuvdata-3.2.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pyuvdata-3.2.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2e4852c93ea87b445c983c1aacdb5d9e73d152c59498703cc33f0d13d9faee93
MD5 da9acf28c6637763384aee1f275e16cb
BLAKE2b-256 d96e2b074b7d7a0aa41e91131914876e55ff854c8c86598181c2918318ad7a42

See more details on using hashes here.

File details

Details for the file pyuvdata-3.2.2-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for pyuvdata-3.2.2-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 fa66e2474d0213a3f86380add3314a05b6769f89bb3123b6d5424154e4148ff5
MD5 851d1304e38c5717695191c5495227ce
BLAKE2b-256 8347527b644d502d312ae7149e1958c1b1369431ea26f7dba7745f28b1be01be

See more details on using hashes here.

File details

Details for the file pyuvdata-3.2.2-cp312-cp312-macosx_10_13_x86_64.whl.

File metadata

File hashes

Hashes for pyuvdata-3.2.2-cp312-cp312-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 59fb0d0dea4dd8cc5113d0c80ea4a565cb08881b8eb8e8af7c083a171e23bd85
MD5 9570b2f721d30d383788545b99e98d74
BLAKE2b-256 5ead0718e5eb189fa3b5799d28027c150d2be81f5867e3a6bb8254d9bc2e0903

See more details on using hashes here.

File details

Details for the file pyuvdata-3.2.2-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: pyuvdata-3.2.2-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 41.8 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for pyuvdata-3.2.2-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 9040d33c53df39060102d3496098e2442e58d75156c3543b7a3e45768a10e839
MD5 802f0c79e5223e00eb108f9140c3209e
BLAKE2b-256 3faba54219054701061eedc75a1b4aa531958c50893fdcfef464d6f086801cdd

See more details on using hashes here.

File details

Details for the file pyuvdata-3.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pyuvdata-3.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 da92b2fd98b50c9e6f4c2d5104049833e3597b2301e5d9e7ae0d21e440c304cb
MD5 0f4f0cf5be241616cb9c6f9dff8ff601
BLAKE2b-256 42810d882c2000587627544c9fff9ae7797bf84de201728bd850589aa9319517

See more details on using hashes here.

File details

Details for the file pyuvdata-3.2.2-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for pyuvdata-3.2.2-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 4ba192373e813a2f821a28ad72e0c33232ed9509c3dede31e25ebbcb7ad96395
MD5 fac619ae7534a6b4c1c3a45d026465ca
BLAKE2b-256 e385b7e5b55776dd03cee1d5fd350133b68918c7ef3db3b680f78445a8f06f79

See more details on using hashes here.

File details

Details for the file pyuvdata-3.2.2-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for pyuvdata-3.2.2-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 5e36303be4604c6d5ece755abcd8eb05f591c7818e28a2c6373072ddba9e0038
MD5 e2c3e79b446ee3f5873f0106ad21e02b
BLAKE2b-256 1d703a98b802d4822e47b7629fbc41f9f0daa3673f9d62a93217208a5ae3e983

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page