spark-mf4 (v2) — asammdf-backed Spark reader

v2 changes the approach. Instead of a hand-written MDF parser (v1, the Scala data source), v2 delegates parsing to the mature asammdf library and distributes the work across Spark. This reads the real-world files (CANedge, Vector, ETAS, vendor variants) that a hand-written parser can silently fail on — e.g. returning 0 rows — because asammdf handles the full spec.

How it works

Spark's built-in binaryFile source lists and reads each file as (path, content) on any storage it supports (DBFS, Unity Catalog Volumes, S3, ADLS, local). The bytes are handed to asammdf through an in-memory buffer inside mapInPandas, so there's no dependence on a local filesystem path on the executors, and files are read in parallel (one task per file).

binaryFile (path, content)  ──►  mapInPandas  ──►  asammdf.MDF(BytesIO(content))
                                                     └► pandas ─► Spark rows

Install (Databricks)

pyspark and pyarrow are already on the cluster. Install the library (cluster Libraries → PyPI, or in a notebook):

%pip install spark-mf4        # once published; or: %pip install /Volumes/.../spark_mf4-2.0.0-py3-none-any.whl

Use

from spark_mf4 import read_mf4

# A single file, a directory, or a glob — all read in parallel.
df = read_mf4(spark, "/Volumes/main/default/logs/")
df.display()

# Options
read_mf4(spark, "s3://bucket/log.mf4", raster=0.1)        # resample to 0.1 s
read_mf4(spark, ".../log.mf4", channel_group=0)            # one channel group only
read_mf4(spark, ".../log.mf4", channels=["EngineSpeed"])   # subset of channels

Output columns: timestamp, one column per channel (names sanitised for Spark), and source_file (the originating path; disable with add_source_column=False).

Options

Notes

• Heterogeneous files in one read_mf4 call share the schema inferred from the first file; read files of the same type together.
• CAN/LIN raw payloads come through as-is; DBC signal decoding (via asammdf.extract_bus_logging) is a planned follow-up.
• v1 (the pure-Scala format("mf4") data source) remains on main; v2 lives here under python/.