EMFC

emfc reads and writes complex electromagnetic fields directly as CUDA PyTorch tensors. .emfc (Electromagnetic Field Container) files support bit-exact or encoder-verified relative-L2 field compression plus bit-exact, self-contained simulation scenes.

The package has no CPU field codec and never compiles code at runtime. Install a precompiled wheel matching the supported Windows/Linux, CUDA, and GPU architecture matrix.

Quick start

import emfc

# One call returns the primary field on the requested CUDA device.
field = emfc.load("field.emfc", device="cuda:0")

# Omit tolerance for bit-exact primary-field storage.
emfc.save("field-lossless.emfc", field)

# Or request a verified global relative-L2 bound for the primary field.
emfc.save("field-1pct.emfc", field, tolerance=0.01)

# The same 1% error bound expressed in decibels: 20 * log10(0.01) = -40 dB.
emfc.save("field-minus-40db.emfc", field, tolerance_db=-40)

# Also verify 0.1% complex error independently in each 20 dB band from -20 to -100 dB.
emfc.save(
    "field-weak-preserving.emfc", field,
    tolerance=0.001,
    preserve_weak_fields=True,
)

Paths and binary file objects are accepted. The .emfc suffix is conventional; the EMFC magic and exact 1.0 version stored in the file are authoritative. Bare private codec streams are not accepted by this public file API.

Containers without files

encode and decode are the same codec with the file step removed, for containers that live in memory — an object store, a socket, a database column, or a dataset pipeline:

# Every save() argument works here; the result is exactly the bytes save() would write.
blob = emfc.encode(field, tolerance_db=-40, scene=scene)

# Every load() argument except the file; bytes, bytearray, and memoryview are accepted.
field, scene = emfc.decode(blob, device="cuda:0", return_scene=True)

# Metadata without decoding any field or scene array.
info = emfc.inspect(blob)

Because save and load are thin wrappers over these two, containers cross freely between the paths: decode reads a saved file's bytes and load reads encoded bytes. The tensor stays on its CUDA device throughout; only the finished container is host memory.

Complete simulation scenes

Use SceneData to store the complete compiled material fields and declarative simulation input. The definition can be a WiTwin Maxwell Scene, a dataclass tree, or mappings/lists containing analytic geometries, triangle meshes, sources, monitors, ports, boundaries, and grids.

scene_data = emfc.SceneData.from_scene(
    scene,
    eps_r=eps_r,
    mu_r=mu_r,
    sigma_e=sigma_e,
    simulation=simulation,  # solver method and configuration, if separate from scene
    metadata={"run": "reference"},
)

emfc.save("result.emfc", electric, tolerance=1e-3, scene=scene_data)

electric, restored_scene = emfc.load(
    "result.emfc", device="cuda", return_scene=True
)
assert restored_scene is not None
eps_r = restored_scene.eps_r

load() reads the file once. It returns a Tensor by default and returns (tensor, scene) when return_scene=True; the scene value is None for a container without one. tolerance and tolerance_db apply only to the primary E/H field. Permittivity, permeability, conductivity, mesh vertices/faces, custom grid coordinates, source datasets, and observables always retain their original dtype and bytes. Scene arrays may use lossless Deflate compression and carry SHA-256 checksums.

Analytic geometry and other Python objects load as safe ObjectRecord trees containing their qualified type names and state. The reader does not use pickle, dynamically import named modules, or instantiate classes from an untrusted file.

An EMSample-shaped object with relative_permittivity, relative_permeability, grid, boundary, sources, and material table attributes can be passed directly as scene=.

Shard datasets for training

emfc.shard packs complete containers into fixed-volume .emfcs shards (4 GiB by default), so a training job can plan an epoch from one small index and fetch a sample with one read. Records are stored and returned byte-identically — the shard layer never re-encodes — and samples that share a scene store its arrays once per shard, when those samples land in the same shard and the shared suffix is at least dedup_min_bytes (64 KiB).

from emfc.shard import ShardWriter, index, loader, verify

# Write: encode straight into shards, or append already-encoded containers with write_bytes().
with ShardWriter("dataset/", target_bytes=1 << 30) as writer:
    for sample_id, field in enumerate(fields):
        writer.write(field, sample_id=sample_id, tolerance_db=-40, scene=scene_data)

for path in writer.sealed_shards:
    verify(path)          # crc32 of every reconstructed container, checksums, sidecars

# Index once, after every writer has finished: merges the per-shard sidecars into
# dataset.json/dataset.idx without opening a single shard.
dataset = index("dataset/")
print(len(dataset), "records in", len(dataset.shards), "shards")

# Train: workers read bytes, this process decodes on a private CUDA stream.
for batch in loader("dataset/", batch_size=8, num_workers=2, device="cuda", epoch=0):
    fields = batch.fields          # (B, C, X, Y, Z) complex64 CUDA tensor when shapes agree
    ...

ShardReader(path).read(i) returns exactly the bytes emfc.encode produced, so emfc.shard.extract(path, i, "sample.emfc") writes an ordinary .emfc file. emfc.shard.inspect summarises a shard without decoding, and pack/repack import .emfc files or re-shard at a new volume, both with zero re-encode. There is no CPU codec, so a DataLoader worker must never decode; loader enforces that split. See docs/SHARD.md for the byte-level format.

A dataset can be grown incrementally and safely. A later session resumes the sample-id numbering instead of restarting it, several ranks fan out with a stride, a crashed writer's .emfcs.tmp is salvaged with recover, and the daily small shards are consolidated with compact — none of it re-encodes a container:

from emfc.shard import ShardWriter, index, recover, compact

# A later session continues past the sample ids already in the dataset.
with ShardWriter("dataset/", resume=True) as writer:
    for field in todays_fields:
        writer.write(field, tolerance_db=-40, scene=scene_data)

# Several writers fan out with distinct ranks; rank r emits ids r, r + world, r + 2*world, …
with ShardWriter("dataset/", rank=r, sample_id_stride=world) as writer:
    ...

dataset = index("dataset/")        # refuses a dataset that reissued a sample id; ignores .emfcs.tmp
print(dataset.sample_id_max, dataset.shard_count, dataset.pending)

for tmp in index("dataset/").pending:
    recover(tmp)                    # rebuild a sealed shard from a crash's unsealed .emfcs.tmp

compact("dataset/")                 # merge under-filled shards in place, then re-run index()

index is the dataset-level safety net: every record's sample_id is its identity, and a session that forgot resume=True and reissued ids is refused loudly rather than trained on.

Durability is a deliberate trade-off. A writer buffers appends in buffer_bytes (64 MiB by default) and only sealed shards are on disk, so a hard crash — SIGKILL, power loss — loses the records still in that buffer; recover salvages only what already reached the .emfcs.tmp. Lower buffer_bytes to flush more often (down to one write per record) when the tail of an interrupted session must survive, and pay the extra I/O; keep it large for throughput when losing the current batch on a crash is acceptable. A crash never corrupts a sealed shard either way. See docs/SHARD.md section 10 for the incremental workflow, concurrency, and crash semantics.

Binary file objects

Paths are the shortest form, while already-open binary files work as expected:

with open("result.emfc", "wb") as file:
    emfc.save(file, electric, tolerance_db=-60, scene=scene_data)

with open("result.emfc", "rb") as file:
    electric, restored_scene = emfc.load(
        file, device="cuda:0", return_scene=True
    )

Install

Install a CUDA-compatible PyTorch build first, followed by the tested wheel:

python -m pip install "torch>=2.10" --index-url https://download.pytorch.org/whl/cu128
python -m pip install emfc-0.1.0rc1-cp310-abi3-win_amd64.whl

Release wheels target Windows and manylinux x86-64, CPython 3.10+, Torch 2.10+, CUDA 12.8, and the GPU architectures embedded by the release workflow. A missing or incompatible _C extension raises an installation error; it never falls back to Python or invokes a compiler.

Stable Python API

emfc.load(file, *, device="cuda", return_scene=False) -> torch.Tensor
emfc.load(file, *, device="cuda", return_scene=True) -> tuple[torch.Tensor, SceneData | None]
emfc.save(
    file, tensor, *, tolerance=None, tolerance_db=None, scene=None,
    preserve_weak_fields=False, weak_tolerance=None, weak_floor_db=-100, weak_ceiling_db=-20,
) -> None
emfc.inspect(file) -> ContainerInfo

• tensor is complex CUDA (C, X, Y) or (C, X, Y, Z) data.
• tolerance=None preserves complex64 or complex128 IEEE words exactly.
• A finite tolerance in [0, 1) selects the verified lossy frontier for complex64 input.
• A finite negative tolerance_db is converted with 10 ** (tolerance_db / 20); for example, -40 dB is 0.01. Pass only one tolerance form.
• preserve_weak_fields=True additionally verifies complex relative-L2 in each 20 dB GT-relative amplitude band from weak_ceiling_db down to weak_floor_db; weak_tolerance defaults to the global tolerance. Bands already within tolerance cost nothing; failed bands use complete-byte selection over structured Tucker and exact-support point repairs. The opt-in stream retains weak-field amplitude, phase, and support.
• SceneData uses the conventional material names eps_r, mu_r, sigma_e, and sigma_m.
• inspect() reads metadata without decoding field or scene tensors.
• emfc.__version__ reports the encoder package version (0.1.0rc1).

Low-level codec and stream classes remain public for stream accounting, certificates, and lossless square/cubic power-of-two patch decode.

Format and execution guarantees

• EMFC stores format version 1.0 in both its fixed binary header and manifest, and records the encoder package version (0.1.0rc1) in the manifest.
• Its lossless/lossy primary-field stream is a private payload, alongside named bit-exact scene arrays.
• Lossless field streams use independent 16^3 blocks (16x16x1 for direct 2D), checksums, and complete-container MDL selection over raw, rANS, byte shuffle, Lorenzo, PCIX, and DESL.
• Lossy field streams select the smallest verified native SVDQ/Tucker stream with raw fallback.
• The public file API accepts complete EMFC 1.0 files only.
• Weak-preserving lossy streams independently repair only failed log-amplitude bands, selecting complete bytes over masked Tucker and group-quantized point repairs with indexed or bit-packed support, then verify the real decode in every declared band.
• Field-sized encode/decode work stays in registered CUDA, cuBLAS, and cuSOLVER operators.
• No field or encoded payload tensor crosses to CPU during codec execution; explicit file and scene serialization are the host I/O boundary.

Build and test

Building requires MSVC x64 or a compatible Linux C++ toolchain, CUDA Toolkit/NVCC, Ninja, PyTorch 2.10, and an explicit architecture list. Runtime JIT builds are intentionally unsupported.

$env:TORCH_CUDA_ARCH_LIST = "12.0"
python -m build --wheel --no-isolation
python -m pytest -q -p no:cacheprovider --basetemp .tmp/pytest-wheel
python -m ruff check src tests setup.py
python -m twine check dist/*.whl

The release workflow builds Windows and manylinux cp310-abi3 wheels with CUDA 12.8.1 and PyTorch 2.10, audits their native imports, and loads the identical binaries across Python 3.10–3.14 and PyTorch 2.10–2.12. It uploads GitHub Actions artifacts only; it does not publish GitHub Releases or PyPI packages.

See docs/FORMAT.md, docs/SHARD.md, CHANGELOG.md, and LICENSE.