asammdf 4.0.0

ASAM MDF measurement data file parser

.. image:: https://raw.githubusercontent.com/danielhrisca/asammdf/development/asammdf.png
:height: 200px
:width: 200px
:align: center

----

*asammdf* is a fast parser/editor for ASAM (Associtation for Standardisation of Automation and Measuring Systems) MDF (Measurement Data Format) files.

*asammdf* supports MDF versions 2 (.dat), 3 (.mdf) and 4 (.mf4).

*asammdf* works on Python 2.7, and Python >= 3.4 (Travis CI tests done with Python 2.7 and Python >= 3.5)


----

.. image:: https://raw.githubusercontent.com/danielhrisca/asammdf/development/gui.png

Status
======

+-------------+----------------+-------------------+-----------------+---------------+
| | Travis CI | Coverage | Codacy | ReadTheDocs |
+=============+================+===================+=================+===============+
| master | |Build Master| | |Coverage Master| | |Codacy Master| | |Docs Master| |
+-------------+----------------+-------------------+-----------------+---------------+
| development | |Build Status| | |Coverage Badge| | |Codacy Badge| | |Docs Status| |
+-------------+----------------+-------------------+-----------------+---------------+

+----------------+-----------------------+
| PyPI | conda-forge |
+================+=======================+
| |PyPI version| | |conda-forge version| |
+----------------+-----------------------+


Project goals
=============
The main goals for this library are:

* to be faster than the other Python based mdf libraries
* to have clean and easy to understand code base
* to have minimal 3-rd party dependencies

Features
========

* create new mdf files from scratch
* append new channels
* read unsorted MDF v3 and v4 files
* read CAN bus logging files
* filter a subset of channels from original mdf file
* cut measurement to specified time interval
* convert to different mdf version
* export to pandas, Excel, HDF5, Matlab (v4, v5 and v7.3), CSV and parquet
* merge (concatenate) multiple files sharing the same internal structure
* read and save mdf version 4.10 files containing zipped data blocks
* space optimizations for saved files (no duplicated blocks)
* split large data blocks (configurable size) for mdf version 4
* full support (read, append, save) for the following map types (multidimensional array channels):

* mdf version 3 channels with CDBLOCK
* mdf version 4 structure channel composition
* mdf version 4 channel arrays with CNTemplate storage and one of the array types:

* 0 - array
* 1 - scaling axis
* 2 - look-up

* add and extract attachments for mdf version 4
* handle large files (for example merging two files, each with 14000 channels and 5GB size, on a RaspberryPi) using *memory* = *minimum* argument
* extract channel data, master channel and extra channel information as *Signal* objects for unified operations with v3 and v4 files
* time domain operation using the *Signal* class

* Pandas data frames are good if all the channels have the same time based
* a measurement will usually have channels from different sources at different rates
* the *Signal* class facilitates operations with such channels

* graphical interface to visualize channels and perform operations with the files


Major features not implemented (yet)
====================================

* for version 3

* functionality related to sample reduction block: the samples reduction blocks are simply ignored

* for version 4

* functionality related to sample reduction block: the samples reduction blocks are simply ignored
* handling of channel hierarchy: channel hierarchy is ignored
* full handling of bus logging measurements: currently only CAN bus logging is implemented with the
ability to *get* signals defined in the attached CAN database (.arxml or .dbc)
* handling of unfinished measurements (mdf 4): warnings are logged based on the unfinished status flags
but no further steps are taken to sanitize the measurement
* full support for remaining mdf 4 channel arrays types
* xml schema for MDBLOCK: most metadata stored in the comment blocks will not be available
* full handling of event blocks: events are transfered to the new files (in case of calling methods
that return new *MDF* objects) but no new events can be created
* channels with default X axis: the defaukt X axis is ignored and the channel group's master channel
is used

Usage
=====

.. code-block:: python

from asammdf import MDF

mdf = MDF('sample.mdf')
speed = mdf.get('WheelSpeed')
speed.plot()

important_signals = ['WheelSpeed', 'VehicleSpeed', 'VehicleAcceleration']
# get short measurement with a subset of channels from 10s to 12s
short = mdf.filter(important_signals).cut(start=10, stop=12)

# convert to version 4.10 and save to disk
short.convert('4.10').save('important signals.mf4')

# plot some channels from a huge file
efficient = MDF('huge.mf4', memory='minimum')
for signal in efficient.select(['Sensor1', 'Voltage3']):
signal.plot()



Check the *examples* folder for extended usage demo, or the documentation
http://asammdf.readthedocs.io/en/master/examples.html

Documentation
=============
http://asammdf.readthedocs.io/en/master

Contributing
============
Please have a look over the [contributing guidelines](https://github.com/danielhrisca/asammdf/blob/master/CONTRIBUTING.md)

Contributors
------------
Thanks to all who contributed with commits to *asammdf*

* Julien Grave `JulienGrv <https://github.com/JulienGrv>`_.
* Jed Frey `jed-frey <https://github.com/jed-frey>`_.
* Mihai `yahym <https://github.com/yahym>`_.
* Jack Weinstein `jacklev <https://github.com/jacklev>`_.
* Isuru Fernando `isuruf <https://github.com/isuruf>`_.
* Felix Kohlgrüber `fkohlgrueber <https://github.com/fkohlgrueber>`_.
* Stanislav Frolov `stanifrolov <https://github.com/stanifrolov>`_.
* Thomas Kastl `kasuteru <https://github.com/kasuteru`_.
* venden `venden <https://github.com/venden`_.

Installation
============
*asammdf* is available on

* github: https://github.com/danielhrisca/asammdf/
* PyPI: https://pypi.org/project/asammdf/
* conda-forge: https://anaconda.org/conda-forge/asammdf

.. code-block: python

pip install asammdf
# or for anaconda
conda install -c conda-forge asammdf


Dependencies
============
asammdf uses the following libraries

* numpy : the heart that makes all tick
* numexpr : for algebraic and rational channel conversions
* matplotlib : for Signal plotting
* wheel : for installation in virtual environments
* pandas : for DataFrame export
* canmatrix : to handle CAN bus logging measurements

optional dependencies needed for exports

* h5py : for HDF5 export
* xlsxwriter : for Excel export
* scipy : for Matlab v4 and v5 .mat export
* hdf5storage : for Matlab v7.3 .mat export
* fastparquet : for parquet export

other optional dependencies

* chardet : to detect non-standard unicode encodings
* PyQt4 or PyQt5 : for GUI tool
* pyqtgraph : for GUI tool


Benchmarks
==========

Graphical results can be seen here at http://asammdf.readthedocs.io/en/master/benchmarks.html


Python 3 x64
------------
Benchmark environment

* 3.6.2 (v3.6.2:5fd33b5, Jul 8 2017, 04:57:36) [MSC v.1900 64 bit (AMD64)]
* Windows-10-10.0.17134-SP0
* Intel64 Family 6 Model 69 Stepping 1, GenuineIntel
* 16GB installed RAM

Notations used in the results

* full = asammdf MDF object created with memory=full (everything loaded into RAM)
* low = asammdf MDF object created with memory=low (raw channel data not loaded into RAM, but metadata loaded to RAM)
* minimum = asammdf MDF object created with memory=full (lowest possible RAM usage)
* compress = mdfreader mdf object created with compression=blosc
* noDataLoading = mdfreader mdf object read with noDataLoading=True

Files used for benchmark:

* mdf version 3.10
* 167 MB file size
* 183 groups
* 36424 channels
* mdf version 4.00
* 183 MB file size
* 183 groups
* 36424 channels



================================================== ========= ========
Open file Time [ms] RAM [MB]
================================================== ========= ========
asammdf 4.0.0.dev full mdfv3 1466 337
asammdf 4.0.0.dev low mdfv3 1372 184
asammdf 4.0.0.dev minimum mdfv3 420 70
mdfreader 2.7.8 mdfv3 2794 430
mdfreader 2.7.8 compress mdfv3 4323 129
mdfreader 2.7.8 noDataLoading mdfv3 1187 176
asammdf 4.0.0.dev full mdfv4 1786 312
asammdf 4.0.0.dev low mdfv4 1637 147
asammdf 4.0.0.dev minimum mdfv4 1099 71
mdfreader 2.7.8 mdfv4 6706 441
mdfreader 2.7.8 compress mdfv4 8542 141
mdfreader 2.7.8 noDataLoading mdfv4 4539 182
================================================== ========= ========


================================================== ========= ========
Save file Time [ms] RAM [MB]
================================================== ========= ========
asammdf 4.0.0.dev full mdfv3 894 343
asammdf 4.0.0.dev low mdfv3 866 190
asammdf 4.0.0.dev minimum mdfv3 3135 78
mdfreader 2.7.8 mdfv3 7733 459
mdfreader 2.7.8 noDataLoading mdfv3 9353 520
mdfreader 2.7.8 compress mdfv3 7827 428
asammdf 4.0.0.dev full mdfv4 982 316
asammdf 4.0.0.dev low mdfv4 974 157
asammdf 4.0.0.dev minimum mdfv4 3600 80
mdfreader 2.7.8 mdfv4 4669 459
mdfreader 2.7.8 noDataLoading mdfv4 6612 478
mdfreader 2.7.8 compress mdfv4 4525 418
================================================== ========= ========


================================================== ========= ========
Get all channels (36424 calls) Time [ms] RAM [MB]
================================================== ========= ========
asammdf 4.0.0.dev full mdfv3 1605 346
asammdf 4.0.0.dev low mdfv3 7224 199
asammdf 4.0.0.dev minimum mdfv3 9965 87
mdfreader 2.7.8 mdfv3 102 430
mdfreader 2.7.8 nodata mdfv3 16696 211
mdfreader 2.7.8 compress mdfv3 622 129
asammdf 4.0.0.dev full mdfv4 1685 316
asammdf 4.0.0.dev low mdfv4 12592 157
asammdf 4.0.0.dev minimum mdfv4 16428 84
mdfreader 2.7.8 mdfv4 93 441
mdfreader 2.7.8 compress mdfv4 624 141
mdfreader 2.7.8 nodata mdfv4 27146 206
================================================== ========= ========


================================================== ========= ========
Convert file Time [ms] RAM [MB]
================================================== ========= ========
asammdf 4.0.0.dev full v3 to v4 5677 680
asammdf 4.0.0.dev low v3 to v4 5737 352
asammdf 4.0.0.dev minimum v3 to v4 9341 118
asammdf 4.0.0.dev full v4 to v3 5095 610
asammdf 4.0.0.dev low v4 to v3 5328 263
asammdf 4.0.0.dev minimum v4 to v3 9983 115
================================================== ========= ========


================================================== ========= ========
Merge 3 files Time [ms] RAM [MB]
================================================== ========= ========
asammdf 4.0.0.dev full v3 17059 1641
asammdf 4.0.0.dev low v3 16730 622
asammdf 4.0.0.dev minimum v3 25156 166
mdfreader 2.7.8 v3 24608 1335
mdfreader 2.7.8 compress v3 30669 1347
mdfreader 2.7.8 nodata v3 24093 1456
asammdf 4.0.0.dev full v4 17949 1513
asammdf 4.0.0.dev low v4 17592 461
asammdf 4.0.0.dev minimum v4 36417 166
mdfreader 2.7.8 v4 36287 1326
mdfreader 2.7.8 nodata v4 35904 1361
mdfreader 2.7.8 compress v4 42410 1336
================================================== ========= ========


.. |Build Master| image:: https://travis-ci.org/danielhrisca/asammdf.svg?branch=master
:target: https://travis-ci.org/danielhrisca/asammdf
.. |Coverage Master| image:: https://api.codacy.com/project/badge/Coverage/a3da21da90ca43a5b72fc24b56880c99?branch=master
:target: https://www.codacy.com/app/danielhrisca/asammdf?utm_source=github.com&utm_medium=referral&utm_content=danielhrisca/asammdf&utm_campaign=Badge_Coverage
.. |Codacy Master| image:: https://api.codacy.com/project/badge/Grade/a3da21da90ca43a5b72fc24b56880c99?branch=master
:target: https://www.codacy.com/app/danielhrisca/asammdf?utm_source=github.com&utm_medium=referral&utm_content=danielhrisca/asammdf&utm_campaign=badger
.. |Docs Master| image:: http://readthedocs.org/projects/asammdf/badge/?version=master
:target: http://asammdf.readthedocs.io/en/master/?badge=stable
.. |Build Status| image:: https://travis-ci.org/danielhrisca/asammdf.svg?branch=development
:target: https://travis-ci.org/danielhrisca/asammdf
.. |Coverage Badge| image:: https://api.codacy.com/project/badge/Coverage/a3da21da90ca43a5b72fc24b56880c99?branch=development
:target: https://www.codacy.com/app/danielhrisca/asammdf?utm_source=github.com&utm_medium=referral&utm_content=danielhrisca/asammdf&utm_campaign=Badge_Coverage
.. |Codacy Badge| image:: https://api.codacy.com/project/badge/Grade/a3da21da90ca43a5b72fc24b56880c99?branch=development
:target: https://www.codacy.com/app/danielhrisca/asammdf?utm_source=github.com&utm_medium=referral&utm_content=danielhrisca/asammdf&utm_campaign=badger
.. |Docs Status| image:: http://readthedocs.org/projects/asammdf/badge/?version=development
:target: http://asammdf.readthedocs.io/en/master/?badge=stable
.. |PyPI version| image:: https://badge.fury.io/py/asammdf.svg
:target: https://badge.fury.io/py/asammdf
.. |conda-forge version| image:: https://anaconda.org/conda-forge/asammdf/badges/version.svg
:target: https://anaconda.org/conda-forge/asammdf
.. |anaconda-cloud version| image:: https://anaconda.org/daniel.hrisca/asammdf/badges/version.svg
:target: https://anaconda.org/daniel.hrisca/asammdf