A library for SuperDARN DMAP file I/O

The SuperDARN DMAP file formats are all supported (IQDAT, RAWACF, FITACF, GRID, MAP, and SND) as well as a generic DMAP format that is unaware of any required fields or types (e.g. char, int32) for any fields. For more information on DMAP please see RST Documentation.

Installation

Package manager

This package is registered on PyPI as darn-dmap, you can install the package with your package manager, e.g. pip install darn-dmap.

From source

If you want to build from source, you first need to have Rust installed on your machine. Then:

1. Clone the repository: git clone https://github.com/SuperDARNCanada/dmap
2. Create a virtual environment and source it, then install maturin
3. In the project directory, run maturin develop to build and install the Python bindings. This will make a wheel file based on your operating system and architecture that you can install directly on any compatible machine.

Usage

The basics

The basic code to read and write a DMAP file is:

import dmap

file = "path/to/rawacf_file"
data, _ = dmap.read_rawacf(file)  # returns `tuple[list[dict], Optional[int]]`
outfile = "path/to/outfile.rawacf"
dmap.write_rawacf(data, outfile)  # writes binary data to `outfile`
raw_bytes = dmap.write_rawacf(data)  # returns a `bytes` object

dmap.read_rawacf(...) reads the file into a list of dictionaries, returning the list as well as the byte where any corrupted records start.

The supported reading functions are:

• read_iqdat,
• read_rawacf,
• read_fitacf,
• read_grid,
• read_map,
• read_snd, and
• read_dmap.

The supported writing functions are:

• write_iqdat,
• write_rawacf,
• write_fitacf,
• write_grid,
• write_map,
• write_snd, and
• write_dmap.

Accessing data fields

To see the names of the variables you've loaded in and now have access to, try using the keys() method:

print(data[0].keys())

which will tell you all the variables in the first (zeroth) record.

Let's say you loaded in a MAP file, and wanted to grab the cross polar-cap potentials for each record:

import dmap
file = "20150302.n.map"
map_data, _ = dmap.read_map(file)

cpcps=[rec['pot.drop'] for rec in map_data]

I/O on a bz2 compressed file

dmap will handle compressing and decompressing .bz2 files seamlessly, detecting the compression automatically. E.g.

import dmap
fitacf_file = "path/to/file.bz2"
data, _ = dmap.read_fitacf(fitacf_file)
dmap.write_fitacf(data, "temp.fitacf.bz2")

will read in the compressed file, then also write out a new compressed file. You can also pass the argument bz2=True to compress with bzip2 regardless of file extension, or even to return compressed byte objects.

Generic I/O

dmap supports generic DMAP I/O, without verifying the field names and types. The file must still be properly formatted as a DMAP file, but otherwise no checks are conducted.

NOTE: When using the generic writing function write_dmap, scalar fields will possibly be resized; e.g., the stid field may be stored as an 8-bit integer, as opposed to a 16-bit integer as usual. As such, reading with a specific method (e.g. read_fitacf) on a file written using write_dmap will likely not pass the FITACF format checks.

import dmap
generic_file = "path/to/file"  # can be iqdat, rawacf, fitacf, grid, map, snd, and optionally .bz2 compressed
data, _ = dmap.read_dmap(generic_file)
dmap.write_dmap(data, "temp.generic.fitacf")  # fitacf as an example
data2, bad_byte = dmap.read_rawacf("temp.generic.fitacf")  # This will fail due to different types for scalar fields
assert bad_byte == 0  # The first record should be corrupted, i.e. not be a valid FITACF record
assert len(data2) == 0  # No valid records encountered

Handling corrupted data files

The self-describing data format of DMAP files makes it susceptible to corruption. The metadata fields which describe how to interpret the following bytes are very important, and so any corruption will lead to the remainder of the file being effectively useless. dmap is able to handle corruption in two ways. The keyword argument mode of the read_rawacf, etc. functions allows you to choose how to handle corrupt records.

In "lax" mode (the default), no error is raised if a corrupt file is read, and the byte where the corrupted records start is returned along with the non-corrupted records. In "strict" mode, the I/O functions will raise an error if a corrupted record is encountered.

import dmap

corrupted_file = "path/to/file"
data, bad_byte = dmap.read_dmap(corrupted_file, mode="lax")
assert bad_byte > 0

good_file = "path/to/file"
data, bad_byte = dmap.read_dmap(good_file, mode="lax")
assert bad_byte is None

In both uses of the above example, data will be a list of all records extracted from the file, but may be considerably smaller than the file.

import dmap

corrupted_file = "path/to/file"
try:
    data = dmap.read_dmap(corrupted_file, mode="strict")
    had_error = False
except:
    had_error = True
assert had_error

good_file = "path/to/file"
try:
    data = dmap.read_dmap(good_file, mode="strict")
    had_error = False
except:
    had_error = True
assert had_error is False

Stream I/O

dmap also can conduct read/write operations from/to Python bytes objects directly. These bytes must be formatted in accordance with the DMAP format. Simply pass in a bytes object to any of the read_[type] functions instead of a path and the input will be parsed.

While not the recommended way to read data from a DMAP file, the following example shows the use of these byte I/O functions:

import dmap
file = "path/to/file.fitacf"
with open(file, 'rb') as f:  # 'rb' specifies to open the binary (b) file as read-only (r)
    raw_bytes = f.read()  # reads the file in its entirety
data, _ = dmap.read_dmap(raw_bytes)
binary_data = dmap.write_fitacf(data)
assert binary_data == raw_bytes

As a note, this binary data can be compressed ~2x typically using zlib, or with another compression utility. This is quite useful if sending data over a network where speed and bandwidth must be considered. Note that the binary writing functions can compress with bzip2 by passing bz2=True as an argument.

File "sniffing"

If you only want to inspect a file, without actually needing access to all the data, you can use the read_[type] functions in with the optional indices argument. This will only read in the specified records from a file, and works in all modes.

import dmap
path = "path/to/file"
recs = dmap.read_dmap(path, mode="...", indices=[0, 1, -2, -1])

Reading only metadata fields

Each DMAP format consists of metadata and data fields. You can read only the metadata fields by passing mode="metadata" to any of the writing functions. Note that the generic read function read_dmap will return all fields, as it by nature has no knowledge of the underlying fields. Note also that the read functions operating on a file still read the entire file into memory first, so reading metadata only may not significantly decrease read times. Also, bzipped files are fully decompressed before reading begins, so again read times may not be significantly quicker in this mode under certain circumstances.