Bytewise MIME detector

Bytewise is a standalone neural MIME detector trained on raw file bytes. It does not require Java, a Tika server, a filename, or a file extension. The repository preserves its complete research lineage: BFA/BFC baselines, neural experiments, strict-host validation, deduplication audits, and D3 reports.

The default model is a compact Transformer trained on the first 4,096 bytes of a leakage-safe POLAR, GovDocs, and curated public-format corpus. It learns 125 MIME classes and exposes 127 supported outputs, including octet-stream fallback and M4V refinement. The prior 100-, 58-, and original 57-class models remain bundled as previous, production58, and legacy options.

Usage

Homebrew Python is an externally managed environment and must not be modified with pip --break-system-packages. For the command-line application, install Bytewise into an isolated Python 3.12 environment with uv:

uv tool install --python 3.12 "bytewise[metal]"
uv tool update-shell

Open a new terminal (or add $HOME/.local/bin to PATH) and verify it:

bytewise --version
bytewise model-info

Choose the runtime extra for the target platform:

# Apple Silicon GPU
python -m pip install "bytewise[metal]"

# Linux with an NVIDIA GPU and a current NVIDIA driver
python -m pip install "bytewise[cuda]"

# CPU inference on Linux or Windows
python -m pip install "bytewise[inference]"

The metal extra pins the validated TensorFlow 2.18 and tensorflow-metal 1.2 runtime. The cuda extra installs TensorFlow's pip-managed CUDA and cuDNN libraries; the host still needs a compatible NVIDIA driver. Confirm CUDA is visible with:

python -c 'import tensorflow as tf; print(tf.config.list_physical_devices("GPU"))'

For development or library use, keep the dependency in a project environment:

cd "$HOME/git/bytewise"
uv sync --python 3.12 --extra metal --group tests
uv run bytewise model-info
uv run python

Inside that uv run python session:

from bytewise import Detector

detector = Detector.load_default()
result = detector.detect_file("document.bin")

print(result.mime_type)
print(result.confidence)
print(result.alternatives)

The same model is available from the command line:

bytewise detect document.bin image.dat
bytewise detect document.bin --top-k 5 --threshold 0.80 --json
cat unknown.bin | bytewise detect -
bytewise supported
bytewise model-info

Select an earlier immutable model when reproducibility requires it:

legacy = Detector.load_default(model="legacy")
previous = Detector.load_default(model="previous")
production58 = Detector.load_default(model="production58")

bytewise detect --model-version legacy document.bin
bytewise detect --model-version previous document.bin
bytewise detect --model-version production58 document.bin
bytewise model-info --model-version legacy

Routine TensorFlow startup diagnostics are suppressed so CLI output remains script-friendly. To diagnose device selection or CUDA loading, enable them for one invocation:

bytewise detect --tensorflow-logs document.bin

Allow uncertain inputs to abstain so another detector can handle them:

detector = Detector.load_default(confidence_threshold=0.80)
result = detector.detect_bytes(payload)
if result.abstained:
    # Fall back to tika-python, libmagic, or another detector.
    pass

Production 125-class model

The Bytewise 0.6.0 default is bytewise-125-v1-candidate:

• Held-out accuracy: 94.26%
• Held-out macro F1: 0.825
• Held-out weighted F1: 0.942
• Support-aware diversity: 99 / 103 test-supported classes correctly predicted
• Learned labels: 125; supported outputs: 127
• Model SHA-256: 348276b894e540898d96dd3146d046b68016dc22c622ae1311aa74edab6478f2

See its model card.

The Bytewise 0.5.0 100-class model remains selectable with --model-version previous. The Bytewise 0.3.0 58-class model remains selectable with --model-version production58.

Legacy v1 model

The preserved legacy model is polar-byte-transformer-seed550-v1:

• Validation accuracy: 96.77%
• Independent-reference accuracy: 94.59%
• Input: first 4,096 raw bytes
• Labels: 57 MIME types
• Model SHA-256: 72a2f5f2dd0fbb4ffaf88488618bc8e034c03876c7cea94d11da23439ed5b849

See the model card and Full-v3 report for the complete evidence.

Repository organization

• src/bytewise/: production API and migrated byte-frequency research code
• artifacts/: immutable model release bundle
• configs/, scripts/: reproducible experiments and evaluation
• reports/: curated D3 reports from early pilots through Full-v3
• docs/: research and repository-extraction documentation
• tests/: standalone production and research regression tests

Dataset bytes, feature caches, databases, raw predictions, and transient logs are intentionally kept outside Git.