idelib 3.3.0

Python API for accessing IDE data recordings

idelib README

idelib is a Python API for accessing enDAQ's IDE recordings. The IDE format is an EBML encoded file using a custom schema. This library utilizes our ebmlite to parse the files, and provides classes that make reading data simple.

IDE File Basics

What's an IDE file?

An IDE file is a read-only hierarchical data format that stores recording information generated by an enDAQ sensor device. It contains both time-indexed data from several different kinds of sensors (like acceleration, pressure, temperature, etc.), as well as metadata about the recording device (like device serial number, model number, device name, etc.) and recording settings.

Accessing an IDE file

The top-level interface for an IDE file is the Dataset object, through which one can access all of the above-listed information. When you open a file for reading, for example, this is the type of object that is returned.

Opening an IDE File

You can open an IDE file like so:

filename = "your_file.IDE"
with idelib.importFile(filename) as ds:
    print(type(ds))

Note: a Dataset object perfoms lazy-loading, meaning that it only loads information as is needed. As a result, it internally retains a handle to the source file which after use needs to be closed. This can be accomplished by either using Dataset as a context manager (as seen above; this is the recommended method), or by using Dataset as a normal value and calling the close() method manually:

filename = "your_file.IDE"
ds = idelib.importFile(filename)
# use `ds` here
ds.close()  # remember to close the file after use!

Getting recording data

Channels and Subchannels

IDE files organize recording data into channels and subchannels. A channel encapsulates data recorded by a particular individual sensor on the device (e.g., XYZ acceleration from the ADXL375 DC Accelerometer); a subchannel, if present, specifies a particular data stream within a channel (e.g., the X-coordinate acceleration from the ADXL375 DC Accelerometer).

At the top-level, a Dataset object has a channels member, which is a dict of all channels recorded in the file. The dict is keyed by channel id numbers, with Channel objects in the values.

Each Channel object has a subchannels member, which is a list of Subchannel objects. If the channel has no subchannels, this member will be None.

The below table lists current conventions for channels across all enDAQ sensors:

(Abbreviated) Product No.	Description	Example Product Nos.
S-D	enDAQ S-series devices with a single digital accelerometer	S3-D16, S4-D40
S-DD	enDAQ S-series devices with dual digital accelerometers	S1-D100D40, S2-D25D16
S-ED	enDAQ S-series devices with an analog piezoelectric and digital accelerometer	S5-E25D40, S4-E100D40
S-RD	enDAQ S-series devices with an analog piezoresistive and digital accelerometer	S4-R500D40, S5-R2000D40
W-D	enDAQ W-series devices with a single digital accelerometer	W5-D40
W-ED	enDAQ W-series devices with an analog piezoelectric and digital accelerometer	W8-E100D40, W8-E2000D40
W-RD	enDAQ W-series devices with an analog piezoresistive and digital accelerometer	W8-R500D40, W8-R2000D40
SSX	Midé Slam Stick X data recorders	SSX
SSC	Midé Slam Stick C data recorders	SSC
SSS	Midé Slam Stick S data recorders	SSS

The below table lists channel ID numbers used in a recording file based on the recording device’s product number (device series numbers and accelerometer sensitivity ranges are omitted when applicable to all such devices):

Sensor	Channel	Valid Devices	Suchannels
Main Accelerometer	8	S-RD, S-ED, SSS, SSX	X-, Y-, Z-axis Acceleration
16/200g Accelerometer	32	S-DD, SSX, SSS, SSC, S-D16, S-D200	X-, Y-, Z-axis Acceleration
8/40g Accelerometer	80	S-RD, S-DD, S-ED, S-D40, S-D8	X-, Y-, Z-axis Acceleration
IMU Gyroscope	47	All1	X-, Y-, Z-axis Rotation
Absolute Orientation	65	All1	X-, Y-, Z-, W-axis Quaternion; Acc
Relative Orientation	70	All1	X-, Y-, Z-, W-axis Quaternion
MPL3115	36	S-D16, All1 before Mid-2023	Pressure, Temperature 2
MS8607 Internal	20	All1 after Mid-2023	Pressure, Temperature, Humidity
MS8607 Control Pad	59	All1	Pressure, Temperature, Humidity
SI1133	76	All1	Lux, UV
BMI270/BMG250 Gyroscope	84	All1 after Mid-2023	X-, Y-, Z-axis Rotation
CAM-M8Q GPS	88	W-D, W-ED, W-RD	Latitude, Longitude, Time, Speed

¹ excluding early SSC/SSS/SSX models

² 1 Hz Internal Measurements

³ 10 Hz Control Pad Measurements

To simply use all recording data, we can iterate through each subchannel in a dataset like so:

for ch in ds.channels.values():
    for sch in ch.subchannels:
        print(sch)

EventArrays and raw data

Each Channel and Subchannel object has a getSession() method, which returns an EventArray object. EventArray is a wrapper around a channel's underlying recording data that loads data on demand from the source file. You can index an EventArray (e.g., eventarray[i] for some index i) to get a numpy ndarray of data. Data is organized in an n-dimensional array.

For subchannels, this will always be a 2-by-n array, where n is the number of samples recorded; eventarray[1] indexes the samples, eventarray[0] indexes the respective timestamps.

For channels, this will be a (c+1)-by-n array, where n is the number of samples recorded and c is the number of subchannels; eventarray[1:] indexes the samples, eventarray[0] indexes the respective timestamps.

Getting metadata

Dataset makes available some basic metadata. Some useful pieces of information are stored directly as members:

>>> ds.filename
'C:\\Users\\Public\\SSX09546_019.IDE'

Other data is stored in the dict member recorderInfo:

>>> ds.recorderInfo['RecorderSerial']
9546
>>> ds.recorderInfo['PartNumber']
'S3-E500D40'

EventArray also stores some sample-specific metadata, like the data's units:

>>> eventarray.units
('Acceleration', u'g')

Command-line Utilities

idelib includes two command-line utilities for exporting and inspecting IDE data. These are installed as executable scripts as well as Python submodules. The latter (the executable versions) may require modifying your computer's 'path' environment variables(particularly in Microsoft Windows).

ideexport

This utility converts one or more IDE files into formats directly usable by other software (.MAT, .CSV, .TXT, etc.). More information can be attained running the utility using the -h or --help arguments.

Usage

Obtaining a list of command-line arguments:

ideexport --help

or directly from the Python submodule:

python -m idelib.tools.ideexport --help

Note that the first example (executing ideexport) may not work if you system paths are not set up to include the installed ideexport executable.

ideinfo

This utility displays basic information about IDE files: the device that recorded it, the date of recording, a list of data channels recorded, etc. More information can be attained running the utility using the -h or --help arguments.

Usage

Obtaining a list of command-line arguments:

ideinfo --help

or directly from the Python submodule:

python -m idelib.tools.ideinfo --help