ipmkit

Lossless Mastercard IPM tooling. A drop-in replacement for cardutil with its bugs fixed.

IPM → CSV → IPM gives a file identical to the original, byte for byte. Nothing is converted, trimmed or masked.

Install

pip install ipmkit

The PIN block and key modules (ipmkit.pinblock, ipmkit.key) also need pip install "ipmkit[crypto]".

Convert files

ipmkit ipm2csv T112.ipm -o T112.csv

ipmkit csv2ipm T112.csv -o T112_rebuilt.ipm

ipmkit compare T112.ipm T112_rebuilt.ipm

The encoding is always EBCDIC cp037 unless you pass --encoding (for example --encoding latin_1 for an ASCII file). ipm2csv warns if the first MTI looks like ASCII. 1014 blocking is detected automatically; use --blocked/--unblocked to override it, or --config-file for a custom field config.

ipm2csv then verifies that the CSV rebuilds the identical file. If it doesn't, it fails instead of handing you a lossy CSV.

Speed on large files

Files with 20,000 or more records are split into chunks and converted on several CPU cores, one process per core up to 8. The chunks are joined back in their original order, so records keep their exact order and the CSV is byte-identical to a single-process run. Use --workers N to choose the number of processes (--workers 1 for a single process).

100,002 records (72 MB, cp037, 1014-blocked), on a 10-core Mac:

What's in the CSV

A text value that can't sit safely in a CSV (control characters, or text that itself starts with hex:) is written as hex: followed by its bytes in hex.

Opening the CSV in Excel and saving it drops leading zeros and reformats values. Edit it with a text editor, or import every column as Text.

Editing the CSV

You can change values. Lengths for LLVAR/LLLVAR fields are recalculated and the 1014 blocking is redone. A fixed-length field with the wrong length is rejected, never silently padded or cut. To remove a field, clear its cell and its bit in BITMAP (or clear the BITMAP cell to have it recalculated).

Header and trailer generation

When the CSV has no file header (MTI 1644, DE24 697) or no trailer (MTI 1644, DE24 695), csv2ipm generates them:

• header: DE48 = PDS 0105 File ID + PDS 0122 processing mode (--processing-mode: P production, the default, or T test); DE71 = 00000001
• trailer: DE48 = PDS 0105 + PDS 0301 (sum of all DE4 amounts, 16 digits) + PDS 0306 (message count including header and trailer, 8 digits); DE71 = message count

The File ID is copied from the existing header or trailer if there is one. Otherwise it is built from --file-type, --file-date (YYMMDD, default today), --processor-id and --file-seq. A CSV made from an IPM file that had no header or trailer remembers that, so csv2ipm doesn't add one. --no-header-trailer turns generation off.

A hand-made CSV only needs MTI and DEn columns. It is written as cp037, not blocked, unless you pass --encoding / --blocked. A CSV made by ipm2csv remembers its encoding in the META row, and --encoding overrides that too.

Python API

from ipmkit import ipmcsv

ipmcsv.convert_ipm_to_csv('in.ipm', 'out.csv')          # verified
ipmcsv.convert_csv_to_ipm('out.csv', 'new.ipm')

ipm = ipmcsv.read_ipm(open('in.ipm', 'rb').read())      # IpmFile: records with raw field bytes
ipm.records[1].fields[42]                               # b'MERCHANT0000001'
data = ipmcsv.write_ipm(ipm)

cardutil compatibility

ipmkit.iso8583, ipmkit.mciipm, ipmkit.card, ipmkit.pinblock, ipmkit.key and ipmkit.config have the same API as cardutil 0.7.3. Change cardutil to ipmkit in your imports. The library functions (loads, dumps, IpmReader, IpmWriter) keep cardutil's latin_1 default so existing code behaves the same. The commands that read or write IPM files (mci_ipm_to_csv, mci_csv_to_ipm, mci_ipm_param_to_csv) default to cp037. The cardutil test suite passes against ipmkit, with one test changed because a value too long for its field now raises an error instead of being cut off.

ipmkit installs the same command names as cardutil (mci_ipm_to_csv, mideu and others). If both packages are installed in one environment, whichever was installed last provides those commands, so use separate environments if you need both.

Fixes (details in ANALYSIS.md):

• messages with only a primary bitmap can be read; DE128 is processed
• values too long for a field raise an error instead of being cut off or corrupting the length prefix
• dumps() no longer changes the dict you pass in; too much PDS data raises a clear error
• DE55 is parsed as real BER-TLV (multi-byte tags such as DF01, long-form lengths)
• 10–12 digit PINs work in ISO-0/ISO-4 PIN blocks
• Luhn validation works under python -O; short PANs are masked safely
• loads is about 2.7× faster (it no longer builds a hex dump of every message when debug logging is off)

License

MIT. Based on cardutil, Copyright (c) 2019-2025 Anthony Delosa.