bytesense 1.2.0

Detect the encoding. Validate every byte. Keep your data intact.

bytesense is a Python encoding detector for file imports, multilingual text pipelines and legacy data. It combines fast Unicode checks, compact character-pair statistics and optional Rust acceleration—with zero runtime dependencies.

Final detection results strictly validate the selected encoding against the complete supplied input. Linguistic scoring uses a bounded sample; stream inputs spill to a temporary file after a configurable memory limit. Results tell you what was examined, what was validated, and whether the input ended.

from bytesense import from_bytes

data = "İstanbul'da çalışan mühendisler için doğru metin çözümleme. ".encode("cp1254")
result = from_bytes(data)
if result.encoding:
    text = data.decode(result.encoding)  # strict decoding; no replacement characters
    print(result.encoding, result.bytes_validated, result.complete)

Install

pip install bytesense

Python 3.9+. Platform wheels include Rust acceleration; the portable wheel works without a compiler. Source installations automatically use Rust when a toolchain is available. To explicitly choose a source build:

BYTESENSE_BUILD_RUST=0 pip install --no-binary=bytesense bytesense  # pure Python
BYTESENSE_BUILD_RUST=1 pip install --no-binary=bytesense bytesense  # require Rust

The old [fast] extra is a compatibility alias. It does not install an accelerator separately. BYTESENSE_PURE_PYTHON=1 selects the Python implementation at runtime.

Built for dependable imports

• Complete validation: a valid prefix cannot hide an undecodable suffix. Unknown and binary inputs have explicit outcomes.
• Bounded streams: feed small chunks or large ones; finalization accounts for every accepted byte. Preview stability never silently ends a stream.
• Useful controls: codec aliases, inclusion/exclusion filters, encoding hints, optional language estimates and a minimum evidence score.
• Transparent results: why, alternatives, sample/validation counts and completion status. Confidence is a heuristic score; no fabricated statistical intervals.
• Portable acceleration: native character-pair scoring and byte histograms, with matching Python behavior and typed public APIs.
• Data repair tools: opt-in mojibake repair and byte-preserving mixed-document segmentation, with strict decoding.

Decodability alone cannot prove the original encoding. Several legacy codecs can decode identical bytes into different, plausible text. Use a known encoding or an explicit hint when you have one, and evaluate representative data before switching detectors.

Files, streams and existing integrations

from bytesense import detect, detect_stream, from_path

result = from_path("export.csv", include_language=True)
print(result.to_dict())

with open("archive.txt", "rb") as source:
    result = detect_stream(iter(lambda: source.read(64 * 1024), b""))
assert result.complete

# Familiar dictionary interface for code that uses chardet.detect().
metadata = detect(b"hello world")

detect_stream() consumes to EOF by default. max_bytes=... or early_stop=True is an explicit sampling choice and returns complete=False when it stops early. Streaming uses temporary disk space beyond memory_limit (1 MiB by default); close an unfinished StreamDetector or use it as a context manager.

bytesense report.csv
bytesense --language --verbose report.csv
cat report.csv | bytesense --minimal -

The CLI exits 0 when every input has an encoding, 1 for read failures or unknown/binary inputs, and 2 for invalid arguments. JSON goes to stdout; errors go to stderr.

Used by

ingest-sentry uses bytesense for encoding detection and full-snapshot validation before CSV/TSV imports. It adds import contracts, safe UTF-8 normalization, string-preserving pandas/Polars adapters, and a GitHub Action. Built by the same maintainer; an independent project, not affiliated with Sentry.

Measured accuracy and performance

The v1 engine replaces the previous collection of special-case ranking rules with a small, reproducibly generated statistical model. The evaluation separates development documents from held-out documents, grouping identical decoded text across encodings. Accuracy means exact Unicode equality, not just a compatible codec name.

See benchmarks and methodology for measured comparisons with chardet, charset-normalizer and chardetng, including the cases where another detector wins. Timing reports distinguish default behavior from an additional full-decode validation step.

New in 1.2: from_bytes() validates UTF-8 using native SIMD without allocating decoded Unicode; its pure-Python path uses 64 KiB blocks above 64 KiB. On the measured 1 MiB UTF-8 workload, native from_bytes() median latency decreased from 1.564 ms to 0.294 ms (5.3×) versus 1.1, with every byte still validated. Peak temporary Python allocations decreased from about 6 MiB to 6 KiB; this is not total process memory.

The existing 2,078-file and 432-case transfer results are unchanged. This release improves UTF-8 validation cost, not general encoding accuracy; chardet remains more accurate on both corpora. See the benchmark methodology for paired measurements, competitors, backend scope and reproduction commands.

Upgrading from 0.x

• Language reporting is opt-in: include_language=True.
• confidence_interval remains available and is None; scores are not calibrated probabilities.
• An empty cp_isolation=[] allows no codecs. Filters apply to every detection path.
• BOM-bearing UTF-16/32 selects utf_16/utf_32, which consume the signature when decoding.
• No unvalidated UTF-8 fallback; stream finalization and repair decoding reject silent data loss.
• StreamDetector.feed() after finalization raises. reset() starts a new stream.
• Python 3.8 is no longer supported. steps and chunk_size remain accepted compatibility arguments; use sample_size to control statistical work.

Quick start · API · Changes · Contributing

Created by Oğuzhan Kır. MIT-licensed code. The aggregate model's data provenance and reproduction steps are documented with the benchmarks; upstream test documents remain copyrighted by their respective publishers.