Bytewise MIME detector

Intelligent file identification, one byte at a time.

Bytewise is a standalone neural MIME detector trained on raw file bytes. It does not require Java, a Tika server, a filename, or a file extension. The repository preserves its complete research lineage: BFA/BFC baselines, neural experiments, strict-host validation, deduplication audits, and D3 reports.

The Python API and CLI intended for the 1.0 stability boundary are recorded in docs/api-stability-v1.json. These entry points will not be removed or changed incompatibly before the next major release; additive fields and model aliases remain permitted.

The Bytewise 1.1.0 accuracy-first default remains the hash-pinned, byte-only Bytewise-250 model introduced in 1.0.0. Version 1.1.0 promotes the separately frozen bytewise-fast-v6 model for latency-sensitive use and adds transparent server-side request coalescing without changing the stable per-file API. It preserves the complete 225-class 0.13.1 detector and adds 24 calibrated general Transformer routes plus a separately calibrated binary SLA/STL neural head, exposing 252 supported outputs including octet-stream fallback and M4V refinement. The 1.0.0/1.1.0 accuracy-first path keeps those frozen weights and thresholds while reducing TensorFlow adapter overhead and skipping the SLA head outside its audited base-MIME domain. The prior 225-, 200-, 175-, 151-, 150-, 125-, 100-, 58-, and original 57-class models remain bundled as explicit rollback and reproducibility options.

Usage

Homebrew Python is an externally managed environment and must not be modified with pip --break-system-packages. For the command-line application, install Bytewise into an isolated Python 3.12 environment with uv:

uv tool install --python 3.12 "bytewise[metal]"
uv tool update-shell

Open a new terminal (or add $HOME/.local/bin to PATH) and verify it:

bytewise --version
bytewise model-info
bytewise doctor

Choose the runtime extra for the target platform:

# Apple Silicon GPU
python -m pip install "bytewise[metal]"

# Linux with an NVIDIA GPU and a current NVIDIA driver
python -m pip install "bytewise[cuda]"

# CPU inference on Linux or Windows
python -m pip install "bytewise[inference]"

The metal extra pins the validated TensorFlow 2.18 and tensorflow-metal 1.2 runtime. The cuda extra installs TensorFlow's pip-managed CUDA and cuDNN libraries; the host still needs a compatible NVIDIA driver. Confirm CUDA is visible with:

python -c 'import tensorflow as tf; print(tf.config.list_physical_devices("GPU"))'

For development or library use, keep the dependency in a project environment:

cd "$HOME/git/bytewise"
uv sync --python 3.12 --extra metal --group tests
uv run bytewise model-info
uv run python

Inside that uv run python session:

from bytewise import Detector

detector = Detector.load_default()
result = detector.detect_file("document.bin")

print(result.mime_type)
print(result.confidence)
print(result.alternatives)

The same model is available from the command line:

bytewise detect document.bin image.dat
bytewise detect document.bin --top-k 5 --threshold 0.80 --json
cat unknown.bin | bytewise detect -
bytewise supported
bytewise model-info

For latency-sensitive, one-file-at-a-time workflows, the optional fast engine uses the frozen bytewise-fast-v6 single-pass 250-class student. Fast-v6 is prediction-identical across the 987-file Metal/CUDA parity suite and agrees with fast-v5 on 99.72% of the frozen 5,667-file DRAT workload. Concurrent request coalescing is transparent to per-file clients. The accuracy-first production cascade remains the default:

from bytewise import FastDetector

detector = FastDetector.load_default()
result = detector.detect_file("document.bin")

bytewise detect --engine fast document.bin

Java applications can keep that model warm in one local process rather than starting Python or loading TensorFlow for every file:

bytewise serve --engine fast                 # binds to 127.0.0.1:8765
curl http://127.0.0.1:8765/health

The dependency-free Java 11 client is in java/bytewise-client. Retain one BytewiseClient inside DRAT and call detect(Path) with its existing per-file workflow.

Select an earlier immutable model when reproducibility requires it:

legacy = Detector.load_default(model="legacy")
previous = Detector.load_default(model="previous")
production58 = Detector.load_default(model="production58")
production125_v2 = Detector.load_default(model="production125_v2")
java_router = Detector.load_default(model="java_router")
production175 = Detector.load_default(model="production175")
production200 = Detector.load_default(model="production200")
production225_v1 = Detector.load_default(model="production225")
production225_v2 = Detector.load_default(model="production225_hardened")

bytewise detect --model-version legacy document.bin
bytewise detect --model-version previous document.bin
bytewise detect --model-version production58 document.bin
bytewise detect --model-version production125_v2 document.bin
bytewise detect --model-version java_router Example.java
bytewise detect --model-version production175 document.bin
bytewise detect --model-version production200 document.bin
bytewise detect --model-version production225 document.bin
bytewise detect --model-version production225_hardened document.bin
bytewise model-info --model-version legacy

Routine TensorFlow startup diagnostics are suppressed so CLI output remains script-friendly. To diagnose device selection or CUDA loading, enable them for one invocation:

bytewise doctor --require-gpu
bytewise detect --tensorflow-logs document.bin

Allow uncertain inputs to abstain so another detector can handle them:

detector = Detector.load_default(confidence_threshold=0.80)
result = detector.detect_bytes(payload)
if result.abstained:
    # Fall back to tika-python, libmagic, or another detector.
    pass

Production 250-class model

The Bytewise 1.1.0 accuracy-first default is bytewise-250-v1. Its exact 42,289-file full-cascade replay passed every frozen gate:

• Overall accuracy: 82.81%; expansion macro F1: 68.44%
• Frozen 225-class base accuracy: 89.56%
• Internal SLA precision / recall / F1: 99.05% / 96.76% / 97.89%
• Fresh sealed SLA precision / recall / F1: 100% / 86.67% / 92.86%
• M3 Metal / RTX CUDA routing mismatches: zero across 75 sealed files
• Learned labels: 250; supported outputs: 252
• Bundle SHA-256: 052b6f88e8b8e99bbabeef05e8529f0a7edb8fcbcd7f029d96365f786a0a240f

The 1.0.0 execution pass preserves exact MIME routes on the deterministic performance benchmark and all existing SLA routes in the 42,289-file replay. On the M3 benchmark it reduced warm single-file latency by 58.6%, increased 1,000-file throughput by 29.3%, and reduced peak RSS by 11.8%.

Select it explicitly with --model-version production250.

Preserved 225-class hardening model

The Bytewise 0.13.1 model is bytewise-225-v2-hardening-candidate. It keeps the frozen 0.13.0 neural components and adds 70 base-conditioned safety gates for ambiguous specialist overrides:

• Full 33,121-file accuracy: 91.30%; macro F1: 83.81%
• New 25-class benchmark accuracy: 95.59%
• Frozen source-disjoint new-class accuracy: 94.66%
• Exact 29,310-file parent-intersection accuracy: 90.74%
• Harmful specialist overrides in the regression cohort: zero
• Learned labels: 225; supported outputs: 227
• Bundle SHA-256: 5716b2ced6574b75439018e9273cf30c75062358f586d6438792a84251ef0c71

Select 0.13.1 with --model-version production225_hardened; the Bytewise 0.13.0 model remains selectable with --model-version production225.

Preserved 200-class composite model

The Bytewise 0.12.0 model is bytewise-200-v4-candidate, the exact frozen composite candidate promoted after local and RTX parity gates:

• Frozen 175-class fallback plus 25 confirmed frontier MIME types
• Fresh enriched holdout: 31/31 correct (88.78% exact 95% lower bound)
• Fresh hard-negative suite: 2,150 files with zero harmful routes
• M3 and RTX parity gates: passed
• Learned labels: 200; supported outputs: 202
• Bundle SHA-256: 8ea9e88551e89becef9ab689d88b6f583735d292e450adde565372fd1fce0f7d

The 0.11.0 rehearsal-refined model remains selectable with --model-version production175_rehearsal; the 0.10.0 repaired model remains selectable with --model-version production175. Select 0.12.0 with --model-version production200.

Preserved 151-class Java router

The Bytewise 0.9.0 model is bytewise-151-java-router-v1. It preserves the 0.8.0 result unless Java neural confidence and structural Java syntax both pass the locked routing contract. Independent evaluation achieved 98.0% Java recall and 98.99% Java F1 with zero changed predictions on the established Wikimedia and complementary regression suites.

The equivalent explicit alias is --model-version java_router. The Bytewise 0.8.0 model remains selectable with --model-version production150.

Preserved 150-class fusion model

The Bytewise 0.8.0 default is bytewise-150-fusion-v1. It combines the frozen 150-class v3 model with the Bytewise 0.7.1 125-class parent using the externally validated 0.4/0.6 probability weights and a 4x expansion-class bias:

• Internal accuracy: 92.35%; macro F1: 0.826
• Internal expansion accuracy: 86.75%; macro F1: 0.892
• Fresh mixed holdout: +1.14 accuracy points overall and +1.01 expansion points
• Expanded legacy confirmation: 85.41% accuracy, +0.54 points over the parent
• Learned labels: 150; supported outputs: 152
• Fusion bundle SHA-256: e2f85dcaf8416b318fc6e01e311e8872d8be7d8b65fcb8b0cfd3a7084fd05e99

The exact contract, model components, and external validation are documented in the Bytewise-150 report.

The Bytewise 0.7.1 125-class model remains selectable with --model-version production125_v2.

The Bytewise 0.6.0 125-class model remains selectable with --model-version production125_v1.

The Bytewise 0.5.0 100-class model remains selectable with --model-version previous. The Bytewise 0.3.0 58-class model remains selectable with --model-version production58.

Legacy v1 model

The preserved legacy model is polar-byte-transformer-seed550-v1:

• Validation accuracy: 96.77%
• Independent-reference accuracy: 94.59%
• Input: first 4,096 raw bytes
• Labels: 57 MIME types
• Model SHA-256: 72a2f5f2dd0fbb4ffaf88488618bc8e034c03876c7cea94d11da23439ed5b849

See the model card and Full-v3 report for the complete evidence.

Repository organization

• src/bytewise/: production API and migrated byte-frequency research code
• artifacts/: immutable model release bundle
• configs/, scripts/: reproducible experiments and evaluation
• reports/: curated D3 reports from early pilots through Full-v3
• docs/: research and repository-extraction documentation
• tests/: standalone production and research regression tests

Dataset bytes, feature caches, databases, raw predictions, and transient logs are intentionally kept outside Git.