twinflame 0.2.0b1

DEX-level Android APK class-diffing engine (SimHash + LSH + abstract-opcode comparison)

twinflame

DEX/APK bytecode-level diffing engine for Android reverse engineers. Matches classes and methods across two builds by structure — surviving R8/ProGuard renaming — and emits a ranked, method-localized change report (what was added / removed / modified) plus a ProGuard mapping.txt to carry recovered names into your decompiler. Implements the four-stage architecture from Quarkslab's "Android Application Diffing: Engine Overview" (Czayka & Thomas, 2019).

Use cases:

• Vulnerability patch identification — find the class a vendor changed to fix a flaw.
• Repackaging / mod analysis — find injected code that replaces vendor logic with no-ops or hostile callbacks.
• Malware triage / kinship — find shared code across samples and flag when both implement the same permission-gated component (AccessibilityService, NotificationListener, DeviceAdmin). twinflame score (experimental) condenses this to a single containment number — how much of a known family's code is present in a candidate — cheap enough to confirm a prefilter hit without diffing the whole corpus.
• Deobfuscation correlation — pair classes across Proguard-renamed builds.

Inputs can be APKs, single .dex files, or directories of dumped .dex (memory dumps included). For pipelines that compare a sample against many, twinflame prepare fingerprints a sample once into a packed binary record (.tfr, ~20× smaller than JSON) that any diff accepts in place of the APK — record-vs-record compares run about 2× faster end-to-end. twinflame migrate upgrades a prepared corpus in place across releases where the stale layer is recomputable, so a large record DB doesn't have to be rebuilt from the original samples.

Documentation

Full documentation lives in the project wiki:

• Using Twinflame — task-oriented guide: the three use cases, reading the change report, feeding the mapping to jadx/retrace.
• Change Report Schema — the versioned JSON output format (for downstream tooling).
• How It Works — how the pipeline works internally, with diagrams.
• Evaluation Corpus / Accuracy Benchmarks — the corpora the accuracy work is graded against and the P/R/F1 results. Wall-clock perf numbers stay in-repo: BENCHMARKS.md.

Pipeline

1. Multi-DEX merge — every classes*.dex is unioned into one logical view (first-wins on descriptor, matching ART). The loader also records each method's call targets + incoming-xref count and each class's string constants, which feed anchoring.
2. Stage 1 — clustering. Classes group into pools keyed by package name; matching pools across the two APKs compare in parallel. Obfuscated packages (Shannon entropy + length heuristic) fall into a single fallback pool.
3. Stage B — anchoring. Within each pool, R8-invariant seeds are locked in before structural scoring: classes sharing a rare string constant (IDF-weighted) or a distinctive set of android/*/androidx/*/java/*/kotlin/* framework calls are paired with high confidence even when SimHash can't tell them apart. Disable with --no-anchors.
4. Stage 2 — bulk SimHash + bucketed LSH. Each remaining class gets a 128-bit signature built from four 32-bit partial hashes — cls, fld, mth, code — over structure-only features (no names, no strings). Nearest neighbors are found via Dullien-style bit-permutation LSH instead of all-pairs.
5. Stage 3 — accurate, method-level comparison. Top-k Stage-2 neighbors are re-scored at method granularity: methods are assigned 1-to-1 within a class pair (abstract-opcode Levenshtein over 13 semantic categories + prototype + xref), yielding per-method matched / modified / added / deleted verdicts. The class score is the instruction-weighted roll-up of those verdicts blended with structural features. Greedy 1-to-1 assignment resolves class-level ties.

Optional Redex pre-pass strips junk-instruction obfuscation (LocalDcePass + RegAllocPass) before loading.

Installation

pip (recommended)

twinflame is on PyPI. Requires Python ≥ 3.11; runtime deps (androguard, numpy, rapidfuzz) install automatically.

pip install --pre twinflame    # --pre needed while the latest release is a beta
twinflame --help                # console entry point is installed

(pipx install --pip-args=--pre twinflame works too if you prefer an isolated CLI install.)

That's everything for the core tool. Two features need external programs that aren't Python packages: --normalize needs Redex on PATH (optional; only for junk-instruction normalization), and applying a recovered mapping.txt needs your own decompiler (e.g. JADX).

Development checkout

git clone https://github.com/ankorio/twinflame && cd twinflame
python -m venv .venv && . .venv/bin/activate
pip install -e '.[test]'
pytest tests/

Nix flake (reproducible environment)

For a fully-pinned environment (Python + all deps + Redex built from source + JADX), use the Nix flake — everything is locked in flake.lock:

# Install Nix with flakes (Determinate Systems installer is easiest):
curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install
git clone https://github.com/ankorio/twinflame && cd twinflame
nix develop            # drops you in a shell with twinflame + Redex + JADX

Running

One-shot CLI via nix run

nix run builds the tool on demand and invokes it. The first run compiles Redex from source (~5 min); after that everything is cached.

# diff two APKs, scope to a vendor package, write a JSON report
nix run . -- old.apk new.apk \
    --auto-package \
    --threshold 0.8 \
    -o report.json

# real-world example: vuln/patched pair, Redex-normalized to defeat
# junk-instruction obfuscation, with the obfuscated-package fallback
# so Proguard-renamed packages cluster together for comparison
nix run . -- vuln.apk patched.apk --normalize --find-obfuscated

Inputs: APK, .dex, or dumped DEX (no APK)

Each side accepts an APK, a single .dex, or a directory of .dex files — so you can diff dumped/extracted DEX (e.g. pulled from memory or an unpacked payload) with no surrounding APK, with no flag: the input kind is detected. Partial dumps are tolerated — an unparseable or corrupt .dex in a directory is skipped (with a warning) rather than aborting the load. A DEX-only input has no manifest, so --auto-package is unavailable — pass --app-package <prefix> for app-vs-library ranking instead.

# directory of dumped classes*.dex on each side
twinflame dump_old/ dump_new/ --no-cluster

# write the machine report as CSV instead of the default JSON, only the
# classes that actually differ
twinflame a/ b/ --no-cluster --app-package com.target.app -s changed -f csv -o changes.csv

By default, stdout shows a ranked change summary (modified / added / removed / cosmetic / unchanged, app code first) and any sensitive components both builds share, while the full machine report is written to a file (-f json, default; csv/xml also available). Pass -m/--matches for the raw per-class match list instead:

[+] com.acme.foo: Login - Login.java | com.acme.foo: Login - SourceFile -> 0.8523
      ~ validateToken (0.6100)                              # method modified
      - legacyHash                                          # method removed
      + verifyNonce                                         # method added
[-] com.acme.foo: DeletedClass - DeletedClass.java          # in lhs only
[*] com.acme.foo: AddedClass - SourceFile                   # in rhs only

Each [+] paired line is followed by the methods that actually changed, so you see which method moved the class score. The machine report carries the full per-method verdict list, plus each paired class's superclass/interfaces and any sensitive component it implements.

Timings go to stderr; the report to stdout (so you can | less or redirect freely).

Worked example

Diffing a vuln/patched APK pair (a real Android wallet app, ~8 k classes per side, the new build is Proguard'd):

$ nix run . -- old.apk new.apk --normalize --find-obfuscated

Reading the output:

• The load: and diff: lines on stderr report timings and class counts so you can see what was actually compared after --normalize (Redex strips junk instructions on both sides) and post-filtering.
• [+] lines are paired matches with their similarity score (distance < 1.0). The two androidx.activity examples above pair clear-text class names on the left (lhs preserved source files) against the same logical classes on the right (rhs lost source files to SourceFile — Proguard's default). Source-file name was dropped, but the structure + bytecode similarity is high (~0.85–0.95).
• [-] is a class only present in lhs; [*] only in rhs. Counts at the top tell you the global picture (e.g. 11 042 matches: 602 paired below 1.0, 5 027 deleted, 5 664 added — the bulk being unmatched is the price of heavy obfuscation; tweak --find-obfuscated and --threshold to trade quality for coverage).
• Drop the --normalize and re-run to see what uncleaned bytecode looks like — if the average paired similarity drops from ~0.9 to ~0.95+ everywhere, that's the fingerprint of junk-instruction obfuscation that Redex defeats.

Build a standalone binary

nix build              # produces ./result/bin/twinflame
./result/bin/twinflame --help

Dev shell

For interactive use (REPL, hacking on the code, running tests):

nix develop            # enter shell with python + redex + jadx on PATH
pytest tests/          # 265 tests, ~5s
python -m twinflame.cli --help

Python API

from twinflame import load, filter, diff

lhs_app = load("app-1.6.1.apk")
rhs_app = load("app-1.6.3.apk")

lhs = filter(lhs_app.classes, {"package_filtering": "com.vendor.app"})
rhs = filter(rhs_app.classes, {"package_filtering": "com.vendor.app"})

matches = diff(lhs, rhs, 0.8, {
    "synthetic_skipping": True,
    "min_inst_size_threshold": 5,
    "top_match_threshold": 3,
})
for m in matches:
    if m.is_paired and m.distance < 1.0:
        print(f"[+] {m.lhs.info} | {m.rhs.info} -> {m.distance:.4f}")

Cross-version deobfuscation map

--deobfuscation-map OUT writes a ProGuard-format mapping.txt for apk2 (the obfuscated/stripped build) by propagating class names recovered from the matched classes in apk1 (the donor build that still carries the DEX SourceFile attribute). Load it into JADX/Ghidra and the rename propagates to every reference automatically — twinflame never rewrites the DEX.

# apk1 = older build that kept SourceFile; apk2 = the one you want to read
twinflame app-old.apk app-new.apk \
    --find-obfuscated --deobfuscation-map app-new.map --map-min-confidence 0.8

Load into JADX with -Prename-mappings.invert=yes:

jadx --mappings-path app-new.map -Prename-mappings.format=PROGUARD_FILE -Prename-mappings.invert=yes \
    -d out_deobf app-new.apk

invert=yes is required: our file follows the standard ProGuard convention (<original> -> <obfuscated>:, what retrace consumes), but jadx's PROGUARD_FILE reader treats the left side as the name currently in the binary — the opposite. Without the flag it silently renames nothing (no error). Same applies in jadx-gui: check "Invert" alongside the mapping path.

• Recovers the class simple name (ContextCompat.java → x6.q becomes x6.ContextCompat); the obfuscated package and inner-class structure are kept (source files carry no package).
• Anchored / exact (distance == 1.0) matches are trusted; lower-confidence ones are still emitted but flagged with a # low-confidence comment. Tune the floor with --map-min-confidence.
• This is the cross-version superpower over single-APK source-file deobfuscation: it names classes in a build that stripped SourceFile, using the build that didn't. Method/field name propagation is a planned next step.

Workflow recipes

• Patch identification. Scope with --package PREFIX or --auto-package (drastically shrinks the comparison set), use --threshold 0.8 since bug fixes are minor changes. Skip distance == 1.0 matches.
• Modded / repackaged app. Drop --package (mods often live in bundled SDKs, not the vendor package). If you see a sea of ~0.95 "everything changed" matches, run with --normalize — that's the signature of junk-instruction obfuscation.
• Heavily Proguard'd inputs. Add --find-obfuscated so single-letter package names (a.b.c) collapse into a fallback pool rather than failing to pair with their clear-text counterparts.
• Deobfuscating a stripped build. Diff it against an older build that kept SourceFile, add --find-obfuscated --deobfuscation-map out.map, then load out.map in JADX.

Evaluation harness (eval/, plan M3.2)

Grades twinflame's own class matches against real ground truth instead of eyeballing them. Build an OSS Android app twice — R8 off, then R8 on at full optimization — the mapping.txt R8 writes for the R8-on build is a free, perfect oracle for the renaming-only case (directly grades M1.1 + M1.2; inlining/outlining is a future difficulty-graded slice per M3.1).

python -m eval.cli app-r8-off.apk app-r8-on.apk app-r8-on/mapping.txt --package com.vendor.app

Prints true/false positive & negative class-match counts plus precision/recall/F1. eval/mapping.py parses the ProGuard-format oracle (same format deobf.py emits); eval/score.py exposes score_class_matches(matches, ground_truth) -> ScoreResult for use as a library, independent of the CLI. Kept out of the src/twinflame package on purpose — it's tooling to grade the engine, not part of it.

Layout

Module	Role
src/twinflame/model.py	Dataclasses: App, Class, Method, Field, Signature, Match, MethodMatch
src/twinflame/loader.py	androguard wrapping + multi-DEX merge; captures calls/strings/xrefs
src/twinflame/manifest.py	AndroidManifest.xml → ManifestInfo; dev-package suggestion
src/twinflame/normalize.py	Redex subprocess wrapper
src/twinflame/cluster.py	Package-based pools + entropy/length obfuscation fallback
src/twinflame/anchor.py	Stage B: string-IDF + framework-call seed matches
src/twinflame/signature.py	128-bit SimHash + LSH bucket index
src/twinflame/accurate.py	Abstract opcodes, method-level + class scoring, greedy 1-to-1 assignment
src/twinflame/opcodes.py	Dalvik opcode → 13-category lookup table
src/twinflame/_hot.py	Hot loops (popcount, Hamming, rapidfuzz-backed Levenshtein)
src/twinflame/parallel.py	ProcessPool helpers for parallel pool comparison / side loading
src/twinflame/prepare.py	Packed .tfr record codec: prepare / migrate, per-layer versioning
src/twinflame/propagate.py	Type-graph match-propagation cascade over confirmed pairs
src/twinflame/features.py	Semantic feature layer feeding change classification
src/twinflame/changes.py	Change classifier: typed, ranked change report from raw matches
src/twinflame/provenance.py	App vs. bundled-library class discrimination
src/twinflame/boilerplate.py	Generated-boilerplate (structural-twin) detection
src/twinflame/components.py	Sensitive-component detection (AccessibilityService & co.)
src/twinflame/score.py	Tier-1 containment scoring, twinflame score (experimental)
src/twinflame/report.py	Machine-report serialization (JSON/CSV/XML)
src/twinflame/deobf.py	Cross-version deobfuscation: recovered names → ProGuard mapping.txt
src/twinflame/api.py	Public load / filter / diff entry points
src/twinflame/cli.py	twinflame console script

Tests

nix develop --command pytest tests/

Currently: 265 tests (2 need Redex on PATH and auto-skip without it — the dev shell provides it).

Test fixtures are pure-synthetic: tests/fixtures/synthetic.py builds Class objects directly via Python, no binaries committed to git. The integration tests exercise the full diff pipeline (cluster → anchor → signature → accurate → api) on these objects.

The on-disk DEX/APK round-trip (loader.py end-to-end) currently has a smoke-only test; a real binary DEX emitter under tests/fixtures/build_dex.py is stubbed for future work.

Known limitations

• Identical-structure classes collide. Pools where many classes share signatures (trivial getter/setter classes, generated stubs) may produce false pairings via greedy assignment. --min-instr 5 filters trivial classes; raising --neighbors widens the candidate pool.
• No inheritance context. Matching is per-class (now with method-level detail inside a paired class), but ignores first-level parent/child signals (cf. LibPecker); these can be added later without disturbing the core.
• No cross-boundary / optimization resilience yet. Matching assumes a 1:1 class and method correspondence. R8 optimizations that break that assumption — inlining, outlining, class merging — are a planned future track (see twinflame-next-dev-plan.md, M3.1), not yet implemented.
• JNI / native-code changes are invisible. The diff is purely Dalvik-level — native library mutations require a complementary native-code diff.
• LSH is approximate by design. --buckets N tunes the accuracy/speed knob.

License

MIT.