msutils

Everyday Measurement Set operations for radio-astronomy pipelines — inspect, subset, average, manage columns and flags. No calibration, no imaging.

Academic attribution

Please acknowledge this project and its contributors when using the work in research, and cite the associated publications and software release where applicable. This is a scholarly request, not an additional licence condition. Citation information can be found in CITATION.cff.

import msutils

info = msutils.msinfo("obs.ms")
print(info.render())

info.fields["PKS1934-638"].scan_numbers  # [1, 2]
info.spws[0].chan_width[0]  # 10000000.0
info.antennas["m003"].latitude  # -30.71053

$ msutils info obs.ms
Measurement Set: /data/obs.ms
  format      : MSv2    size: 271.4 MiB    rows: 595,200
  telescope   : MEERKAT      observer: sphe          project: TEST-2024-01
  observed    : 2024-01-01T00:00:00.000 -> 2024-01-01T05:12:44.000  (5h12m44s)
  totals      : 32 antennas, 496 baselines, 3 fields, 60 scans, 1 SPWs, 8 channels
  integration : 8s
  max baseline: 4,701.2 m

Fields (3):
  ID  Name         RA            Dec           Frame  Intents             Scans    Rows
  --  -----------  ------------  ------------  -----  ------------------  -----  ------
   0  PKS1934-638  01h20m12.845  -29d47m37.70  J2000  CALIBRATE_BANDPASS     20  198,400
   1  J0217+0144   02h51m53.240  -32d39m30.94  J2000  CALIBRATE_PHASE        20  198,400
   2  DEEP_2       04h23m33.635  -35d31m24.18  J2000  OBSERVE_TARGET         20  198,400
...

Install

Requires Python ≥ 3.11. The base install is just numpy + python-casacore

• click, and covers everything except averaging, plots and MSv4.

pip install msutils                    # msinfo, columns, subset, flags, flagstats, diagnostics
pip install "msutils[plots]"           # + PNG summary plots (matplotlib)
pip install "msutils[average]"         # + time/channel averaging (codex-africanus)
pip install "msutils[msv4]"            # + read MSv4 processing sets (xarray, zarr)
pip install "msutils[xarray-ms]"       # + read an MSv2 through the MSv4 schema
pip install "msutils[reconstruction]"  # + experimental arcae reconstruction probes
pip install "msutils[convert]"         # + write MSv2 -> MSv4 (xradio)
pip install "msutils[all]"             # everything

msinfo

msinfo returns a typed, JSON-serialisable MSInfo whose collections are addressable by name as well as by id — because real work says "the bandpass calibrator", not "row 1".

info = msutils.msinfo("obs.ms")

info.observation.telescope  # 'MEERKAT'
info.fields["DEEP_2"].intents  # ['OBSERVE_TARGET#ON_SOURCE']
info.fields["DEEP_2"].ra_hms  # '04h23m33.635'
info.scans[7].duration  # 24.0  (includes the final integration)
info.spws["SPW0"].centre_freq  # 1415000000.0
info.data_descriptions[1].spw_id  # DDID -> SPW, explicitly
info.columns["DATA"].shape  # [4, 4]

info.to_dict()  # stable, versioned JSON
info.render(verbose=True)  # the listobs-style report above

Three cost levels, so inspecting a 100 GB MS doesn't have to read 100 GB:

"meta" is independent of MS size (~10 ms); "full" is one TaQL GROUPBY, flat in the number of scans.

MSv4 and engines

msinfo reads MSv2 tables and MSv4 processing sets into the same MSInfo:

msutils.msinfo("obs.ms")  # MSv2, via casacore/TaQL
msutils.msinfo("obs.zarr")  # MSv4, via xarray + zarr
msutils.msinfo("obs.ms", engine="xarray-ms")  # MSv2 through the MSv4 schema

The default MSv2 engine is casacore/TaQL: it needs nothing beyond the base install, it is several times faster for metadata, and it will open a Measurement Set that stricter readers reject — which matters, because a malformed MS is exactly the one you need to inspect. Pass engine="xarray-ms" when you want the n-dimensional MSv4 view of an MS you already have, with no conversion step.

Converting for real needs msutils[convert]:

msutils convert obs.ms obs.zarr

Everything else

# columns
msutils.addcol("obs.ms", "MODEL_DATA", clone="DATA")
msutils.copycol("obs.ms", "DATA", "CORRECTED_DATA")
msutils.sumcols("obs.ms", cols=["DATA", "MODEL_DATA"], outcol="CORRECTED_DATA")
msutils.delcol("obs.ms", "CORRECTED_DATA")  # reclaim the disk
msutils.renamecol("obs.ms", "MODEL_DATA", "OLD_MODEL")
msutils.addnoise("obs.ms", column="MODEL_DATA", sefd=551)

# datasets
msutils.subset("obs.ms", "target.ms", fields=["DEEP_2"], spws=[0])
msutils.subset("obs.ms", "parallel.ms", corrs=["XX", "YY"])  # names or zero-based indices
msutils.subset("obs.ms", "target.ms", fields=["DEEP_2"], chan_bin=4)  # select + average
msutils.average("obs.ms", "avg.ms", time_bin=8.0, chan_bin=4)  # [average]

# flags
stats = msutils.flagstats("obs.ms")
stats.by_correlation["XY"].percent
stats.by_channel[0]  # per-channel, per SPW
msutils.flag_backup("obs.ms", name="pre-rfi")
msutils.flag_restore("obs.ms", "pre-rfi")

# diagnostics
msutils.du("obs.ms")  # where the bytes went
msutils.check("obs.ms")  # MSv2 conformance
msutils.taql("SELECT DISTINCT FIELD_ID FROM $1", "obs.ms")

subset takes time_bin/chan_bin too, so selecting and averaging is one pass rather than two (it hands off to average, so that needs the average extra).

Both functions accept corrs=["XX", "YY"] (or repeatable --corr options) to keep correlations by name or zero-based index, in the requested order. Each polarization setup in the selection must contain them; data, flags, weights and polarization metadata follow the selection.

By default subset keeps the original field and SPW ids rather than renumbering them the way CASA split does, so ids in a subset still match the parent MS. Pass reindex=True (--reindex) for split's behaviour: the FIELD, SPECTRAL_WINDOW and DATA_DESCRIPTION rows the output no longer uses are dropped and the survivors are renumbered from 0, in their original order. ANTENNA, POLARIZATION, STATE and OBSERVATION keep every row.

Command line

msutils info      obs.ms [-v] [--level meta|full|data] [--json out.json]
msutils flagstats obs.ms [--plot flags.png] [--json flags.json] [--field DEEP_2]
msutils subset    obs.ms target.ms --field DEEP_2 --spw 0 [--corr XX --corr YY] [--chan-bin 4] [--reindex]
msutils average   obs.ms avg.ms --time-bin 8 --chan-bin 4 [--corr XX --corr YY] [--reindex]
msutils delcol    obs.ms CORRECTED_DATA MODEL_DATA
msutils renamecol obs.ms MODEL_DATA OLD_MODEL
msutils addcol    obs.ms MODEL_DATA --clone DATA
msutils copycol   obs.ms DATA CORRECTED_DATA
msutils sumcols   obs.ms DATA MODEL_DATA --out CORRECTED_DATA
msutils addnoise  obs.ms --column MODEL_DATA --sefd 551
msutils flags     backup|restore|list|delete obs.ms [NAME]
msutils du        obs.ms
msutils check     obs.ms
msutils taql      'SELECT DISTINCT FIELD_ID FROM $1' --ms obs.ms
msutils convert   obs.ms obs.zarr

msutils <command> --help for the full options.

Migrating from 2.x

summary() still works and returns the same dict shape, but is deprecated in favour of msinfo() and now emits a FutureWarning. Several of its values have been corrected — see CHANGELOG.md for the full list and for the removed 1.x aliases.

msutils.weights (MSNoise, SEFD-profile weight estimation) was removed in 3.0: it is data processing rather than an everyday MS operation. Pin msutils<3 if you still need it.

info = msutils.summary("obs.ms")  # deprecated
info = msutils.msinfo("obs.ms")  # use this

License

Apache License 2.0. See LICENSE and NOTICE.