fly-ftl-extract

Drop-in replacement for ftl-extract 0.12.1 — the same ftl extract <code> <locales>, with the same options and byte-for-byte the same output — in which the decision "is this a Fluent key" is made not by a parser but by a simulated fruit-fly mushroom body: LIF neurons on the real FlyWire FAFB v783 connectome (124 projection neurons → 2597 Kenyon cells ⇄ 1 APL → 48 MBONs), a readout on the MBONs trained with "dopamine" (delta rule). The tokenizer only proposes candidates (string literals and attribute chains before () and encodes their context into an odour; the fly decides which of them is a key and which kwarg is placeable. This is a joke — but a joke that is literally true. Switch the fly off and no keys are found: the hot path has neither ast nor rules about i18n.get; the test tests/test_no_ast_in_hot_path.py checks that a plain ftl extract loads neither the AST reference nor the audit.

How it works

Python file ──tokenize──▶ candidate + window (6 tokens before · focus · 3 after)
                                   │
                                   ▼  feature hashing → 14 puffs × 124 PN (rate coding, tanh)
        ┌─────────────────────────────────────────────────────────────────────┐
        │   PN 124  ──ACh──▶  KC 2597  ──ACh──▶  MBON 48       mushroom body  │
        │  Poisson,          LIF, τm 20 ms,        LIF          of FlyWire    │
        │  ≤ 200 Hz         V_th −45 mV               ▲         FAFB v783,    │
        │                    │       ▲                │   right hemisphere    │
        │                    ▼       │ GABA ×0.5      │         syn_count ≥ 5 │
        │                   APL 1 ───┘                │                       │
        │               (feedback inhibition,         │                       │
        │             8–10 % of KCs active per puff)  │                       │
        │                                             │                       │
        │   DAN 165 — teaching signal only ───────────┘  (sim current = 0)    │
        └─────────────────────────────────────────────────────────────────────┘
                                   │  KC spike counts per puff (14 × 2597)
                                   ▼
        readout  w · [KC spiked, log1p(spikes)] + b  →  margin
        |margin| < θ  →  "sniff again" (up to 5 extra trials with other seeds), vote = Σ margin
        margin > 0  →  key  /  kwarg placeable

• Temporal coding. The window = 14 puffs of 40 ms with no silence between them + 10 ms of silence (570 ms, 5700 steps of 0.1 ms): 6 puffs of context before the candidate, 5 puffs of focus ([root][attr1][attr2][attr3][summary] for a chain, [<STR>][silence×3][summary] for a string), 3 puffs after. The membrane is not reset between puffs — that is the memory of the preceding tokens. Every puff is a 124-PN vector: hashed slot features (normalised token, token + distance, type, bracket depth, bigram), 2 buckets per feature, ≈ 10 active PNs per puff.
• Normalisation, not rules. --i18n-keys, -p, --ignore-attributes, --ignore-kwargs affect only the tokens (i18n → <I18N>, self → <PREFIX>, set_locale → <IGNORE>, when → <IGNORE_KW>), so -k LF smells the same as i18n. The decision is the sign of the margin.
• Resniff. θ is chosen on val so that ≤ 10 % of windows get extra sniffs; trial seed = sha256(file bytes, candidate index, encoder version) — two runs give the same output.
• Details: CLAUDE.md (the contract), docs/CONNECTOME.md (the subgraph), docs/BENCH.md (calibration, speed), docs/ODOR.md (the encoder), docs/METRICS.md (training), docs/FORMAT.md (what exactly the original does and what is reproduced).

Installation

Python 3.14 is required (requires-python = ">=3.14,<3.15"). From PyPI:

pip install fly-ftl-extract                              # into a 3.14 venv, or
uv tool install --python 3.14 fly-ftl-extract            # as an isolated tool

The same wheel is attached to every GitHub release; the repository itself installs too (uv build → dist/):

uv tool install --python 3.14 ./fly_ftl_extract-0.1.0-py3-none-any.whl
uv tool install --python 3.14 git+https://github.com/andrew000/FLY-FTL-Extract

The wheel contains the connectome (fly_ftl_extract/data/mb_fafb783.npz, 106 KB), its metadata (meta.json) and the trained weights (mbon_weights.npz, 598 KB); runtime dependencies are numpy, scipy, rich, click and fluent.syntax.

Entry-point conflict. The package installs the scripts ftl and fly-ftl. The original ftl-extract also installs ftl; in one environment the last one installed wins, so do not put both into one venv — keep them in separate uv tool environments or call ours as fly-ftl.

Usage

Everything works as in the original 0.12.1 (the FTL-Extract README): ftl extract CODE_PATH LOCALES_PATH, -l/--language, -k/--i18n-keys, -K, -p/--i18n-keys-prefix, -e/-E, -i/-I, --ignore-kwargs, --default-ftl-file, --comment-keys-mode {comment,warn}, --line-endings, --dry-run, --cache/--cache-path/--clear-cache, --allow-parse-errors, -v, the global --config, the [tool.ftl-extract.extract] section in pyproject.toml (CLI > pyproject > defaults), ftl config sample. Output, exit codes and the .ftl tree match the Rust binary; after its ✅ Done line the block [INFO fly] Fly statistics: is appended (neurons, files, candidates, trials, resniffs, trials/s, brain time).

ftl extract app/bot app/bot/locales -l en -l uk          # drop-in
ftl extract app locales --fly-audit                       # run the AST reference alongside, differences → stderr, exit 1
ftl extract app locales -v --fly-no-tui                   # the margin of every window after ✅ Done

Our options (all prefixed --fly-):

TUI

In a terminal ftl extract draws a live panel (rich, ≤ 15 fps): a spike raster of the Kenyon cells (one dot = one cell that spiked in the trial's summary puff, one row = one trial), the PN/KC/APL/MBON activity of the last trial, an event log (ODOR → MBON margin → KEY / NOT A KEY / placeable, RESNIFF) and counters. All numbers are real — from the connectome, the spikes and the readout; the TUI module never imports random.

Accuracy

Dataset grammar-5: 20 000 synthetic snippets labelled by the AST reference (460 672 candidates, 147 876 kwargs; split 80/10/10 by snippet). The golden fixtures from tests/ and the real bot's code are not part of the dataset — they are the holdout.

Limitations

• One hemisphere (right: 2597 KCs vs 2580 in the left); the left one is not used.
• 15 GABAergic iPNs per side are excluded from the PNs (three have ≥ 5 synapses onto KCs and would break the rule "PN→KC only positive").
• syn_scale = 5.0 and apl_scale = 0.5 are calibrated on our odours (8–10 % KCs per puff, APL inhibits ≥ 2×), not taken from Shiu et al.; the APL weight is scaled separately.
• A 40 ms puff, not 20: at 20 ms the per-puff KC code was not reproducible (same-candidate Jaccard 0.34).
• Synapse threshold syn_count ≥ 5 per neuron pair; DANs carry no current in the forward simulation.
• Speed ≈ 50 trials/s per process (one trial = 5700 LIF steps), 290–410 trials/s on 16 processes depending on machine load; a bot with ~10 000 candidates takes 24–34 s versus 0.02 s for the Rust original.
• ftl stub and ftl check are not implemented (a message and exit 2).
• The --cache cache is written to the same file as in the original, but the format is ours and not compatible with the original.
• -v does not reproduce the original's debug lines (globset, Saved …); our margin lines are printed instead.
• i18n.get("dotted.key.name") and other invalid Fluent identifiers are written as is — the original does the same (the next run of either will fail reading the .ftl).
• The fly makes mistakes. On the grammar-5 test split (with resniff) 15 false keys among 34 369 negatives and 0 misses among 11 554 positives; all 10 worst false keys are self.get("…") / cls.get("…") with -p self -p cls (margin up to +10.5; for the reference a prefix without an i18n name right after it is not a key). The fly learns only what is in the grammar: the first version (grammar-4) missed on the real bot every L("…", _path=…) in dict values and in constructor fields (description=L(…) on the line after a recognised name=L(…), margin −1…−31), LF("…") as the first argument of the decorator @router.message( (−8…−32), and flagged a string inside a tuple, ("captcha_timeout_task", …), with no call at all, as a key (+7.00). This was fixed not by adding rules to the code but by extending the corpus (grammar-5) with these positions and their negative twins: on the same bot it became 0 differences, and the same 13 windows give +5.7…+40.7 (REAL_PROJECT.md). The next unseen construct will likewise be a miss until it gets into the grammar.

Citations and licences

• Connectome: FlyWire FAFB v783, Zenodo 10676866, CC-BY 4.0. Dorkenwald S. et al. Neuronal wiring diagram of an adult brain. Nature 634, 124–138 (2024). Schlegel P. et al. Whole-brain annotation and multi-connectome cell typing of Drosophila. Nature 634, 139–152 (2024) — neuron annotations (flyconnectome/flywire_annotations).
• Neuron model and parameters: Shiu P. K. et al. A Drosophila computational brain model reveals sensorimotor processing. Nature 634, 210–219 (2024) (philshiu/Drosophila_brain_model).
• Inspiration (the mushroom body as a hash function): Dasgupta S., Stevens C. F., Navlakha S. A neural algorithm for a fundamental computing problem. Science 358, 793–796 (2017).
• Fluent: python-fluent (fluent.syntax, Apache 2.0) — parser and serializer, byte-for-byte compatible with fluent-rs on every golden file.
• The original: FTL-Extract © andrew000, MIT. The output format, key order (FxHashMap), file walk and message texts are reproduced from the behaviour of the 0.12.1 binary.
• This package is MIT. The data fly_ftl_extract/data/mb_fafb783.npz is a derivative work of FlyWire (CC-BY 4.0).

How to reproduce

Everything is deterministic (the seeds are fixed); the times are from the author's machine (Ryzen 9 9950X3D, 32 threads) (train.py: 30 processes for the brain; the ftl extract pool: 16).