REAP

Recursive Extraction And Parsing — a general-purpose CLI tool for identifying and recursively extracting firmware and partition images. Works with raw eMMC/flash dumps, individual partition images, full-disk images (GPT or Rockchip PARM), and forensic disk images from a wide range of embedded Linux and Android devices.

Pure Python. No root, no FUSE, no mounting, no Linux kernel modules. Runs on macOS, Linux, and Windows.

What it does

Point it at a directory of partition .bin files, a single image, or a set of 7z archives and it will:

1. Identify each image's format via magic bytes (nearly 50 format signatures)
2. Annotate what partition it is (boot, recovery, system, userdata, etc.) by reading ext4 superblock metadata and analyzing ramdisk contents
3. Extract contents recursively -- e.g. a boot image yields a kernel + ramdisk; the ramdisk decompresses to a cpio archive; the cpio extracts to a filesystem tree
4. Analyze kernels (version, config, build paths, kallsyms symbol table), bootloaders (U-Boot environment, embedded DTBs), and unknown partitions (forensic hex dump, strings, SHA256)
5. Report everything found in text, JSON, CSV, and deterministic SHA-256 forensic manifests, including FBE encryption detection

Supported formats

Partition tables and disk layouts

Format	Detection	Extraction
GPT partition table	EFI PART at offset 0x200 or 0x1000 (UFS 4K sectors)	Individual partition images
MBR / DOS partition table	0x55AA at offset 510 + a valid primary entry (not a FAT VBR or GPT protective MBR)	Primary and bounded EBR logical partitions carved at their LBA-start offsets, with cycle/overlap checks, then recursed.
Rockchip PARM partition table	PARM at offset 0	Individual partition images (RK29xx/RK3xxx flash dumps)
Android super.img (LP metadata)	0x67446C70 at offset 0x1000	Logical partition images (system, vendor, product, etc.)

Android boot and kernel

Format	Detection	Extraction
Android Boot Image (v0--v4)	ANDROID! magic	Kernel, ramdisk, second-stage, recovery DTBO, DTB
ARM zImage	0x016F2818 at offset 0x24	Decompressed vmlinux, kernel config, version string, source paths, kallsyms, all strings
ARM64 Image	ARM\x64 at offset 0x38	Kernel config, version string, source paths, kallsyms, all strings
Raw ARM kernel binary	MSR CPSR instruction + Linux version string	Kernel config, version string, source paths, kallsyms, all strings
Device Tree Blob (DTB)	0xD00DFEED	Extracted DTB, optional dtc decompile to DTS
DTBO container	0xD7B7AB1E	Individual DT overlay entries

Bootloaders and firmware

Format	Detection	Extraction
U-Boot uImage	0x27051956	Unwrapped payload (kernel, ramdisk, firmware, device tree, etc.)
U-Boot binary	U-Boot <version> string, 64 KB--4 MB	Default environment, embedded DTBs, strings
U-Boot environment	CRC32 + key=value pairs, power-of-2 size	Parsed environment variables
Samsung Exynos boot partition	BL1 header pointer + Exynos BL label	bl1.bin, u-boot.bin, tzsw.bin
Rockchip KRNL wrapper	KRNL at offset 0	Unwrapped payload (re-identified as gzip, zImage, etc.)
ELF binary	\x7fELF magic	Metadata dump (class, machine, entry point), strings
AVB vbmeta	AVB0 / AVBf	Metadata dump (version, algorithm, rollback index, flags)

Encrypted firmware containers

Format	Detection	Extraction
IM*H firmware container	IM*H at offset 0 or 0x400	Header parse (version, module name/type, chunk table, key family). Encrypted chunks (RTOS, kernel, TZOS, DTB, etc.) extracted as raw .bin files. Decryption is out of scope for this tool.
Ambarella environment (UNR0)	UNR0 + 0x5AA5 flags	Boot config, A/B slot status, firmware versions, bootloader logs
BitLocker (BDE) volume	-FVE-FS- OEM ID at offset 3, or BitLocker volume GUID at offset 0x10 of an NTFS / FAT-style boot sector	Boot-sector + FVE metadata parse: variant (Vista vs. Win 7+ / To-Go), volume GUID, encryption method (AES-128/256, CBC ± Elephant diffuser, XTS), dataset GUID, FVE block locations, full inventory of key protectors (TPM, TPM+PIN, password, recovery password, startup key, smart card, clear key, etc.). Surfaces clear-key suspended volumes prominently. Decryption is out of scope.
LUKS1 / LUKS2 volume	LUKS\xBA\xBE magic at offset 0; version word distinguishes LUKS1 from LUKS2	LUKS1: cipher / mode / hash / key size / UUID + per-keyslot PBKDF2 iterations, salt, active flag. LUKS2: binary header + JSON metadata — segments (cipher), keyslots (PBKDF2 vs. Argon2i / Argon2id with memory + parallelism), tokens (systemd-cryptenroll TPM2 / FIDO2 / PKCS11 bindings), digests. Decryption is out of scope.

Filesystems

Format	Detection	Extraction
ext4	0xEF53 at offset 0x438	Full filesystem tree with FBE encryption detection
FAT12/16/32	0xEB/0xE9 + 0x55AA at 510	Full filesystem tree (LFN support)
exFAT	EXFAT OEM ID at offset 3	Full filesystem tree (long names, fragmented + NoFatChain runs)
NTFS	NTFS OEM ID at offset 3 + 0x55AA at 510	Full filesystem tree (resident + non-resident $DATA, LZNT1 compression, $ATTRIBUTE_LIST, $INDEX_ROOT + $INDEX_ALLOCATION, ADS, USA fixup; EFS-encrypted files marked, reparse points recorded as text)
cramfs	0x28cd3d45 at offset 0 + Compressed ROMFS at 16	Full filesystem tree (zlib blocks, little- and big-endian)
romfs	-rom1fs- at offset 0	Full filesystem tree (files, dirs, symlinks, hard links)
squashfs	hsqs (little-endian) / sqsh (big-endian) at offset 0	Full filesystem tree for v4.0 little-endian images — inode/directory metadata + data/fragment blocks. Compressors: gzip, xz/lzma, lz4, zstd (if zstandard installed); lzo and big-endian images are flagged, not decoded
JFFS2	node magic 0x1985 (85 19 LE / 19 85 BE) at offset 0	Full filesystem tree (log-structured node scan, newest-version resolution; none/zero/copy/zlib/rtime compressors; lzo/rubin flagged, not decoded)
UBI	UBI# erase-counter header at offset 0 (per PEB)	Reassembles each logical volume from its PEBs (infers PEB size, orders LEBs, parses volume names) and carves each volume as a child image for recursion
UBIFS	node magic 0x06101831 at offset 0	CRC-validated linear node-scan recovery (tolerates mid-commit/partial dumps where the committed index is inconsistent): rebuilds the tree from surviving newest-sqnum inode/dentry/data nodes, decompresses none/zlib (lzo/zstd if the optional lib is present), zero-fills + flags missing blocks, and puts unreferenced inodes under __ORPHANS__/
EROFS	0xE0F5E1E2 at offset 0x400	Flat plain/inline files plus conservative LZ4 full-index and 4-byte compact-index, one-pcluster mappings; advanced/chunk/fragment/2-byte layouts fail closed with typed warnings.
F2FS	0xF2F52010 at offset 0x400	CRC-validated checkpoint-pack recovery, checkpoint-authoritative NAT selection, direct/indirect/double-indirect nodes, sparse files, inline data and inline directories. Encrypted files, compressed clusters, and special inodes fail closed.

Compression and archives

Format	Detection	Extraction
gzip	1F 8B	Decompressed content
LZ4 frame	04 22 4D 18	Decompressed content
LZ4 legacy	02 21 4C 18	Decompressed content (Android ramdisk format)
LZMA	5D 00 00	Decompressed content
bzip2	BZh	Decompressed content
XZ	FD 37 7A 58 5A 00	Decompressed content
cpio newc	070701 / 070702	Files, directories, symlinks (as text files with -> target)
7z archive	37 7A BC AF 27 1C	Full decompression (supports split .7z.001 parts)
ZIP / TAR / Odin .tar.md5	Native signatures and validated TAR trailer	Safely extracts regular members; rejects traversal, links/devices, collisions, corruption, excessive members and expansion; Odin MD5 is verified.
Zstandard	28 B5 2F FD	Bounded streamed decompression
Android sparse image	0xED26FF3A	Converted to raw image, then re-identified and extracted

Device-specific partitions

Format	Detection	Extraction
Android devinfo	ANDROID-BOOT! magic	Lock status, tamper flags
ArduPilot / Autel DataFlash flight log	0xA3 0x95 head + FMT type 0x80	Self-describing FMT schema, per-message CSVs, decoded firmware versions, and a GPS track (GPX + KML). Auto-detects the Autel 8-byte FMT name field vs. stock ArduPilot's 4-byte field.
Qualcomm modemst (EFS)	IMGEFS marker in first 64 bytes	Forensic scan (SHA256, strings, hex dump)
BMP image	BM + valid DIB header	Trimmed BMP (strips partition padding)
Boot logo container	ASCII count/sizes header + BMP at 0x200	Individual BMP images
Empty / zeroed	All-zero content	Verified-empty marker with likely purpose annotation

Installation

Requires Python 3.10+ (supported through Python 3.13).

From PyPI:

pip install reap-cli

The PyPI distribution is reap-cli because the bare reap name on PyPI is held by an unrelated, long-abandoned 2012 package. We are pursuing a PEP 541 transfer. The installed CLI command is reap regardless.

From source (for development):

git clone https://gitlab.com/blackbox-research/reap
cd reap
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

Dependencies (installed automatically):

• ext4 -- pure-Python ext4 filesystem reader (no FUSE/mounting)
• lz4 -- LZ4 decompression for Android ramdisks
• zstandard -- Zstandard archives and filesystem blocks

Third-party plugins that add format handlers or detectors are discovered automatically via the reap.plugins entry point group. See the architecture section below.

Inspect installed plugin metadata without importing plugin code using reap plugins; add --load to explicitly validate registrations. Extraction supports repeatable --enable-plugin NAME and --disable-plugin NAME options. Selections and distribution versions are bound into resume snapshots, and incompatible, conflicting, or unknown selections fail closed with exit 7. Plugin API v1 runs trusted Python code in-process; its isolation limitation and metadata contract are documented in docs/API.md. Inventory is metadata-only, but --load and extraction import and execute the selected plugin with the same filesystem, network, and process permissions as REAP. Only enable plugins whose package and publisher you trust.

Usage

reap <input_path> [options]

input_path can be a single image file, a directory containing partition images, or a set of 7z archives.

Options

Flag	Description
-o DIR	Exact output directory for a single file; output root for a directory (defaults: <file>_unpacked/ or <directory>/unpacked/)
--identify-only	Print format identification only, no extraction
--skip-ext4	Skip ext4 filesystem extraction (useful for huge partitions)
--skip-archives	Skip 7z archive extraction
--force-archives	Force archive extraction even when physicalImage/ already exists
--no-recursive	Don't recurse into extracted children
--salvage, --carve-scan	For images with an unparseable partition table (identified as Unknown), scan for embedded ext4/FAT/exFAT/NTFS filesystems and a UBI area (UBI# after a boot region) and carve them out for extraction
--max-depth N	Maximum recursion depth (default: 10)
--max-items N	Total extraction-item budget (default: 100000; 0 = unlimited)
--max-output-bytes N	Per-artifact extraction cap (default: max(1 GiB, 200× input); 0 = unlimited)
--max-total-output-bytes N	Thread-safe whole-run output cap (default: 20 GiB; 0 = unlimited)
--timeout SECONDS	Cooperative deadline; timeout exits 124 and cleans partial files
--progress none|human|json	Progress on stderr; JSON mode is JSON Lines
--resume	Resume only from a matching hash-validated journal; stale/missing state exits 3
-y, --assume-yes	Use the default archive-skip answer without prompting
-j, --jobs N	Parallel extraction workers (0=auto, 1=sequential; default: auto)
-v	Verbose output (INFO level)
-vv	Debug output
--report text|json|csv|both|all	Report format (all emits all three; default: both)
--manifest none|json|csv|both	Portable SHA-256 evidence manifest (default: both)

Detached authenticity is opt-in and never creates or discovers keys:

pip install 'reap-cli[signing]'
chmod 600 investigator-ed25519-private.pem
reap sign-manifest output/manifest.json --private-key investigator-ed25519-private.pem --signature output/manifest.sig.json
reap verify-signature output/manifest.json --signature output/manifest.sig.json --public-key trusted-investigator-ed25519.pem

Private keys must be unencrypted Ed25519 PEM files with owner-only permissions on POSIX systems. Keep them outside evidence/output trees. Public trust is always an explicit --public-key path; key generation, persistence, lookup, and logging are deliberately out of scope. Signature mismatch exits 6; malformed inputs, unsafe key permissions, missing signing support, and I/O errors exit 5. Existing reap verify content verification and exit codes are unchanged and can be run before or after authenticity verification.

Examples

Identify all partitions in a dump:

reap ./physicalImage --identify-only

Full extraction (skip large ext4 partitions):

reap ./physicalImage --skip-ext4 -v

Extract a single boot image:

reap boot.img -o ./boot_extracted -v

Extract a directory of 7z archives (split parts supported):

reap ./archives/ -v

Parallel extraction with 4 workers:

reap ./physicalImage -j 4 -v

Output structure

For a boot image, the recursive extraction produces:

boot_unpacked/
    kernel_info.txt          # Kernel analysis summary
    kernel_config.txt        # Build-time .config (if IKCONFIG enabled)
    kernel_source_paths.txt  # Build-time source paths
    kernel_strings.txt       # All embedded ASCII strings
    kallsyms.txt             # Kernel symbol table (if present)
    vmlinux                  # Decompressed kernel binary
    ramdisk_unpacked/
        init                 # cpio filesystem tree
        init.rc
        fstab.*
        sbin/
        ...

For a directory of partitions, you get a subdirectory per partition plus reports:

physicalImage/unpacked/
    report.txt               # Human-readable report
    report.json              # Machine-readable report
    report.csv               # Flat machine-readable report (with --report csv/all)
    manifest.json            # Versioned portable evidence manifest
    manifest.csv             # Flat evidence manifest
    mmcblk0p1_unpacked/      # boot image contents
    mmcblk0p2_unpacked/      # DTB contents
    mmcblk0p3_unpacked/      # recovery image contents
    mmcblk0p4_unpacked/      # system filesystem tree
    ...

Partition annotation

The tool automatically identifies partition roles by:

• Reading the ext4 superblock s_last_mounted field (e.g. /system, /data, /cache)
• Analyzing boot image ramdisks for /sbin/recovery to distinguish boot vs recovery
• Parsing U-Boot uImage type fields (kernel, ramdisk, firmware, device tree)
• Parsing IM*H firmware module names and types (bootloader, kernel, RTOS)
• Recognizing format-specific roles (DTB, vbmeta, DTBO, sparse, super, modemst)
• Inferring empty partition purpose from size (<=4 MB zeroed = likely misc or metadata)

Annotations appear in reports and verbose output as labels like (recovery), (system), (userdata), etc.

FBE encryption detection

When extracting ext4 filesystems with File-Based Encryption (FBE), the tool:

• Detects the encryption superblock flag and per-inode encryption flags
• Hex-encodes encrypted filenames for safe extraction
• Writes encrypted_paths.txt listing all encrypted files and directories
• Reports encryption algorithms (AES-256-XTS, AES-256-GCM, etc.) in JSON output

Architecture

reap/
    cli.py              # Argument parsing, entry point
    identify.py         # Ordered detection registry (52 core formats)
    annotate.py         # Partition role inference
    pipeline.py         # Recursive extraction orchestrator (parallel workers)
    report.py           # Atomic text + JSON + CSV reports (FBE-aware)
    manifest.py         # Deterministic portable JSON + CSV evidence manifests
    limits.py           # Output budgets, bounded writers, cancellation
    resume.py           # Atomic hash-validated resume journals
    handlers/
        __init__.py     # BaseHandler ABC, handler registry
        ambarella_env.py # Ambarella UNR0 boot environment
        avb.py          # AVB vbmeta metadata
        bitlocker.py    # BitLocker (BDE) detection + FVE metadata + protectors
        bmp.py          # BMP image (partition padding trim)
        boot_img.py     # Android boot image (v0--v4)
        bootlogo.py     # Boot logo container (multiple BMPs)
        compression.py  # gzip, LZ4, LZMA, bzip2, XZ
        cpio_handler.py # cpio newc archives
        dataflash.py    # ArduPilot/Autel DataFlash flight log (schema, CSVs, GPS track)
        devinfo.py      # Android devinfo (lock status)
        dji_imah.py     # IM*H encrypted firmware container (header parse, encrypted chunks)
        dtb.py          # Device Tree Blob
        dtbo.py         # DTBO container
        elf.py          # ELF binary metadata + strings
        ext4_handler.py # ext4 filesystem (FBE detection, dir_index fallback)
        exynos_boot.py  # Samsung Exynos eMMC boot partition
        mbr.py          # MBR/DOS partition table (splits primary partitions)
        fat.py          # FAT12/16/32 filesystem
        gpt.py          # GPT partition table (512-byte + 4K UFS sectors)
        luks.py         # LUKS1 + LUKS2 detection + keyslot / token metadata
        modemst.py      # Qualcomm modem EFS partition
        ntfs.py         # NTFS filesystem (MFT + runlists + LZNT1 + indexes)
        raw.py          # Empty + unknown fallback (forensic scan)
        salvage.py      # --salvage carve-by-scan for unparseable partition tables
        raw_kernel.py   # Raw ARM kernel binary
        rk_krnl.py      # Rockchip KRNL wrapper
        rkparm.py       # Rockchip PARM partition table
        seven_zip.py    # 7z archive (split-part support)
        sparse_img.py   # Android sparse -> raw conversion
        super_img.py    # super.img LP metadata
        uboot_bin.py    # U-Boot binary (environment, embedded DTBs)
        uboot_env.py    # U-Boot environment block
        uimage.py       # U-Boot uImage wrapper
        zimage.py       # ARM zImage / ARM64 Image kernel extraction
        _kernel_utils.py # Shared kernel analysis (version, config, kallsyms)

Each handler implements BaseHandler.extract() and returns an ExtractionResult with optional children for recursive processing. Versioned public interfaces, plugin keys and conflict rules are documented in docs/API.md.

The pipeline orchestrator (pipeline.py) drives identify → annotate → dispatch → recurse. Children can run concurrently while sharing one reservation-based output ledger and cancellation token. See architecture, schemas, and limitations.

Symlink handling

Symlinks found inside ext4 filesystems and cpio archives are not created as OS symlinks (which can cause issues on some platforms and create security risks with path traversal). Instead, they're written as small text files containing -> target and recorded in the extraction metadata / JSON report.

Running tests

pip install -e ".[dev]"
pytest tests/ -v

The comprehensive synthetic-data suite covers format detection, handler extraction, hostile-input bounds, kernel analysis (kallsyms), pipeline orchestration, and forensic scanning; no real image files are required.

Known limitations

• EROFS and F2FS: Useful conservative subsets are extracted. Advanced layouts intentionally fail closed; see the exact support matrix.
• Encrypted partitions: FBE-encrypted ext4 partitions are detected and documented, but file contents remain encrypted. The tool does not perform Android FDE/FBE decryption. EFS-encrypted files on NTFS volumes are similarly skipped, with a per-file marker recording the encryption. BitLocker and LUKS volumes are detected and their full encryption-scheme metadata is surfaced, but the encrypted payload itself is not decrypted — pipe the source image through dislocker / libbde (BitLocker) or cryptsetup (LUKS) for that.
• IM*H decryption: Out of scope. The core tool parses IM*H headers and extracts encrypted chunks as raw .bin files. Producing plaintext requires AES keys that are not distributed with this tool.
• Large partitions: Extracting a 54 GB ext4 partition takes time and disk space. Use --skip-ext4 to skip these, or extract individual partitions as needed.
• 7z extraction: Requires system 7z binary for split archives; falls back to py7zr for single files.
• Symlinks: Recorded as text files, not created as actual OS symlinks.
• Text files: Plain-text metadata files (.txt, .sha256, .xml, README) in the input directory are detected and skipped rather than subjected to forensic extraction.