CodecCat

Fast character-encoding detection for Python, backed by native inference and strict decoding.

import codeccat

data = "こんにちは".encode("shift_jis")

result = codeccat.detect(data)

print(result.encoding)
print(result.text)

CodecCat detects the character encoding of byte input and decodes it to Unicode.

It is designed for applications that need predictable decoding behavior, low detection latency, and explicit handling of ambiguous input.

Why CodecCat?

• Strict decoding — invalid text is not silently replaced.
• Native inference — the default detector does not require NumPy or scikit-learn.
• Explicit ambiguity — callers can inspect result.ambiguous instead of assuming every detection is certain.
• Portable wheels — published for Linux, macOS, and Windows on supported x86-64 and ARM64 targets.
• Stable public API — the 0.1.x line has a documented compatibility contract.

Encoding detection is inherently probabilistic: some byte sequences can validly represent more than one encoding. CodecCat exposes that uncertainty rather than treating every result as uniquely identifiable.

Install

Install the latest Stable release from PyPI:

python -m pip install codeccat

To pin the currently published first Stable release (before v0.1.1 publication):

python -m pip install codeccat==0.1.0

CodecCat ships binary Wheels for Python 3.10–3.14 on supported Linux (glibc 2.28+, x86-64/ARM64), macOS (Intel/Apple Silicon) and Windows (x86-64/ARM64). The first PyPI release has no sdist, so unsupported platforms cannot fall back to building it from source. Future Stable releases now have a source-distribution round-trip CI (source archive → native Wheel → clean install, Python 3.10–3.14) and a reviewed manual GitHub Release / PyPI OIDC publication pipeline. This future release pipeline has passed its dry-run; it does not retroactively add an sdist to v0.1.0.

For offline installs and artifact SHA256 checks, use GitHub Stable v0.1.0.

Usage

Detect and decode

import codeccat

data = "こんにちは".encode("shift_jis")
result = codeccat.detect(data)

print(result.encoding)
print(result.text)
print(result.ambiguous)

Decode directly

If you only need the decoded text:

text = codeccat.decode(data)

Command line

Inspect a file:

codeccat file.txt

Decode it:

codeccat --decode file.txt > readable.txt

Performance

CodecCat v0.1.0 was compared with chardet and charset-normalizer using the same 843 text samples across five fixed test sets on Ubuntu 24.04 / Python 3.12.

CodecCat is not uniformly better on every metric.

In this benchmark, chardet correctly decoded 3 more samples, while CodecCat identified the exact encoding more often, used less total measured CPU time, and showed substantially lower p95 latency. CodecCat used more peak memory.

These measurements were collected on a shared CI runner. Small timing differences should not be interpreted as statistical guarantees.

Independent-source confirmation (published Stable 0.1.0)

An additional pinned external-source benchmark tested the published Stable Wheel, not a development build. It screened every eligible byte input against the historical training corpus before scoring (no raw-byte/prefix/sampled-block overlap indicators triggered). Ground truth was strict Unicode text, and all three detectors received the same input bytes.

Nautilus is a new external source for this evaluation. CPython is a previously examined, preregistered confirmation source, not a newly blind sample. The Nautilus group is UTF-8-only, and the CPython set is small and CJK-focused. Neither source represents the real-world mix of encodings; do not pool the groups into a claimed production accuracy rate. A negative raw-byte overlap screen is not proof of semantic independence.

Historical development benchmark limitation

The 843-sample benchmark above includes 633 samples from the model's training source, plus previously examined development texts. Its 97.98% figure is therefore an engineering regression benchmark, not an independent estimate of real-world accuracy. The external-source findings also show that CodecCat can underperform chardet on unseen source families.

A source-level paired GNOME Software test showed 18 additional correct decodes with the same frozen model without observed regressions in that test set.

See full benchmark methodology and limitations and the frozen external-source protocol.

Optional ambiguity diagnostic (included in v0.1.1 release candidate)

The repository contains an opt-in, non-mutating C1-control ambiguity diagnostic after PR #47. It is included in the v0.1.1 release candidate, but not in the published codeccat==0.1.0 Wheel. Do not assume PyPI has v0.1.1 until its release has been verified. This diagnostic is not a model-accuracy upgrade.

On a v0.1.1 Wheel containing this feature:

from codeccat import detect
from codeccat.diagnostics import inspect_ambiguity

raw = b"Price: \x80; total."
result = detect(raw)
diagnostic = inspect_ambiguity(raw, result)

if diagnostic.warning:
    print(diagnostic.reason, diagnostic.alternative_encodings)
    # Check a trusted source encoding before choosing another interpretation.

The function inspects only already returned model candidates when selected text contains C1 controls (U+0080–U+009F) and an alternative strictly decodes those bytes without C1 controls. It does not change result.encoding, result.text, result.candidates, result.ambiguous, BOM handling, or L3 behavior. A warning does not prove the current decoding is wrong. Some legal ISO-8859-1 text intentionally contains these controls, and lack of a warning does not prove correctness.

A fixed historical HTML study screened 2,675 files and evaluated 1,886 eligible files after deduplication, strict label decoding, and historical training-source overlap checks. The warning flagged 17 of 572 label-disagreeing results and none of 1,314 label-agreeing results. Archive labels are not independently verified ground truth, so these numbers are not a real-world sensitivity or false-positive rate. See the validation record and protocol.

Detection semantics

Character encoding is not always uniquely identifiable from bytes alone.

For applications where the exact encoding matters:

result = codeccat.detect(data)

if result.ambiguous:
    # Apply application-specific handling.
    ...

CodecCat uses strict decoding behavior. Invalid byte sequences are not silently converted using replacement characters.

Encoding inference remains probabilistic and cannot guarantee a uniquely correct answer for arbitrary input.

Stability

CodecCat v0.1.0 is the first stable release.

The 0.1.x line preserves the documented public API, including the primary detection and decoding calls and documented result fields.

See the public API compatibility contract.

Distribution and provenance

Published wheels are built for supported Linux, macOS, and Windows targets. The frozen default model is distributed with the package and verified as part of the release process.

Model and source provenance, including third-party rights considerations, are documented separately.

See Rights and Provenance.

Train your own model (independent PyPI package)

codeccat-train 0.1.0 is independently published on PyPI (Trainer release CI). It is not included in the default pip install codeccat and is not required for production inference. The runtime stays dependency-light; training uses NumPy and scikit-learn in its own environment.

From a repository checkout, with your own labeled byte files and a grouped JSONL manifest:

python -m venv .trainer-env
.trainer-env/bin/python -m pip install codeccat-train==0.1.0
.trainer-env/bin/python -m codeccat_train fit \\
  --manifest my-data/manifest.jsonl \\
  --output custom.ccm \\
  --report training-report.json

Then load the model using an independent environment containing the published Runtime:

from codeccat import CodecCat
custom_detector = CodecCat.from_files("custom.ccm")
result = custom_detector.detect("café".encode("utf-8"))
print(result.encoding, result.text)

The Trainer also supports --precision f32 (compact CCM2), evaluate (candidate-vs-unchanged Stable with paired improvements/regressions on labeled bytes), and optional fit-l3 (separately trained language artifact, disabled by default). The common training engine can be used with your private dataset or a separately approved official dataset; supplying your data does not grant the project rights to it.

Verified: separate training/Runtime environments, deterministic training, safe manifest checks, native custom-model loading, Linux x64/ARM64 and macOS/Windows checks, and exact historical official L2/L3 artifact reconstruction under pinned sources and dependencies. Not established: universal legal clearance for historical datasets, or improved real-world accuracy of a new model. Custom models do not automatically replace the shipped default model.

See quick start, Trainer README, model contribution rules, rights/provenance and roadmap.

Contributing and maintenance

CodecCat is maintained in public. Contributions are welcome, including concise bug reports, documentation, tests, native packaging fixes, source-aware accuracy studies and reproducible performance work.

• Start contributing: CONTRIBUTING.md and Issue templates.
• Roadmap: ROADMAP.md — shipped work, near-term priorities and longer-term research; no fixed delivery promises.
• Review and release decisions: MAINTAINING.md, API stability contract, and Release notes.
• Help and conduct: SUPPORT.md, CODE_OF_CONDUCT.md. Please report security problems privately.

You can submit small documentation fixes directly as PRs. Discuss larger API, model, dependency or release changes in an Issue before implementation. CI covers supported platforms; contributors are not expected to reproduce the entire six-platform matrix locally.

Links

PyPI · GitHub Releases · Release notes · Issues · Roadmap · Benchmarks · Stability · License