cChardet

cChardet is high speed universal character encoding detector. - binding to uchardet.

This fork (faust-cchardet) vs PyYoshi/cChardet

This is the faust-streaming maintained fork of upstream PyYoshi/cChardet. It exists mainly to keep the project building on current Python and Windows toolchains and to publish up-to-date prebuilt wheels across platforms. The public Python API is unchanged from upstream.

The distribution is renamed on PyPI; the import name is not:

pip install faust-cchardet

import cchardet   # same import name as upstream

Key differences

• Prebuilt wheels built with cibuildwheel, so most users install without a C++ compiler:
  • Linux: x86_64, i686, aarch64 (CPython and PyPy)
  • macOS: x86_64, arm64
  • Windows: x86_64 / AMD64
• Windows / MSVC support — pinned to an MSVC-compatible uchardet (bdb8a0…) so the C++ extension compiles under Microsoft Visual C++. Upstream's newer uchardet header does not build under MSVC.
• Modern build — uses the meson-python build backend (distutils is removed in Python 3.12+); Meson builds the Cython output as C++ and links the matching C++ runtime automatically.
• Versioning — the version lives in src/cchardet/version.py and is exposed as cchardet.__version__ (upstream uses setuptools_scm).
• Python support — requires Python >= 3.10 (3.6–3.9 are dropped).

Detection differences (a consequence of the pinned uchardet)

• A UTF-8 byte-order mark is reported as UTF-8-SIG (upstream reports UTF-8).
• On 32-bit (i686) builds a few near-equivalent labels differ — e.g. Thai TIS-620 is detected as ISO-8859-11.

Building against a system uchardet

By default cChardet builds a bundled copy of uchardet (vendored as a git submodule), which is what the published wheels ship. Distributions and source builds can instead link the system uchardet through the system-uchardet Meson feature option:

Pass the option through your build front-end. With pip / build the meson-python backend reads it from setup-args:

pip install . --config-settings=setup-args=-Dsystem-uchardet=enabled

Requirement: the system uchardet must be recent enough to expose uchardet_get_n_candidates. That symbol currently ships only in the git version (https://gitlab.freedesktop.org/uchardet/uchardet.git) — no tagged release provides it yet — so enabled/auto probe for it and reject an older library.

[!WARNING] A system build currently detects non-UTF-8 input less accurately than the bundled build. The bundled copy compiles a cChardet-specific replacement for uchardet's multi-byte group prober, which both restores detection throughput and rejects non-UTF-8 byte sequences that upstream reports as UTF-8. A system build links upstream's prober, so neither applies.

Measured over 1650 non-UTF-8 documents spanning 7 encodings (benchmarks/make_nonutf8_corpus.py, median of 3 runs):

Build	Throughput	Non-UTF-8 reported as UTF-8
bundled (wheel default)	3.10 MB/s	0.0%
system uchardet	1.44 MB/s	16.2%

Reporting non-UTF-8 bytes as UTF-8 is the defect that caused v3.0.0 to be yanked from PyPI, so this is not a cosmetic difference. It is an upstream issue rather than a packaging mistake: nsUTF8Prober does not reject invalid sequences on its own, and its confidence never falls low enough for the candidate to be discarded. Patches have been sent upstream; once they land, the build can require a uchardet version that includes them and this caveat goes away.

If you are packaging cChardet for a distribution that forbids bundled libraries, consider carrying the overlay (src/cchardet/uchardet-overlay/nsMBCSGroupProber.cpp) as a patch against your system uchardet until then. The corpus generator and benchmark above are in-tree, so you can verify the result yourself.

Supported Languages/Encodings

• International (Unicode)
  • UTF-8
  • UTF-16BE / UTF-16LE
  • UTF-32BE / UTF-32LE / X-ISO-10646-UCS-4-34121 / X-ISO-10646-UCS-4-21431
• Arabic
  • ISO-8859-6
  • WINDOWS-1256
• Bulgarian
  • ISO-8859-5
  • WINDOWS-1251
• Chinese
  • ISO-2022-CN
  • BIG5
  • EUC-TW
  • GB18030
  • HZ-GB-2312
• Croatian:
  • ISO-8859-2
  • ISO-8859-13
  • ISO-8859-16
  • Windows-1250
  • IBM852
  • MAC-CENTRALEUROPE
• Czech
  • Windows-1250
  • ISO-8859-2
  • IBM852
  • MAC-CENTRALEUROPE
• Danish
  • ISO-8859-1
  • ISO-8859-15
  • WINDOWS-1252
• English
  • ASCII
• Esperanto
  • ISO-8859-3
• Estonian
  • ISO-8859-4
  • ISO-8859-13
  • ISO-8859-13
  • Windows-1252
  • Windows-1257
• Finnish
  • ISO-8859-1
  • ISO-8859-4
  • ISO-8859-9
  • ISO-8859-13
  • ISO-8859-15
  • WINDOWS-1252
• French
  • ISO-8859-1
  • ISO-8859-15
  • WINDOWS-1252
• German
  • ISO-8859-1
  • WINDOWS-1252
• Greek
  • ISO-8859-7
  • WINDOWS-1253
• Hebrew
  • ISO-8859-8
  • WINDOWS-1255
• Hungarian:
  • ISO-8859-2
  • WINDOWS-1250
• Irish Gaelic
  • ISO-8859-1
  • ISO-8859-9
  • ISO-8859-15
  • WINDOWS-1252
• Italian
  • ISO-8859-1
  • ISO-8859-3
  • ISO-8859-9
  • ISO-8859-15
  • WINDOWS-1252
• Japanese
  • ISO-2022-JP
  • SHIFT_JIS
  • EUC-JP
• Korean
  • ISO-2022-KR
  • EUC-KR / UHC
• Lithuanian
  • ISO-8859-4
  • ISO-8859-10
  • ISO-8859-13
• Latvian
  • ISO-8859-4
  • ISO-8859-10
  • ISO-8859-13
• Maltese
  • ISO-8859-3
• Polish:
  • ISO-8859-2
  • ISO-8859-13
  • ISO-8859-16
  • Windows-1250
  • IBM852
  • MAC-CENTRALEUROPE
• Portuguese
  • ISO-8859-1
  • ISO-8859-9
  • ISO-8859-15
  • WINDOWS-1252
• Romanian:
  • ISO-8859-2
  • ISO-8859-16
  • Windows-1250
  • IBM852
• Russian
  • ISO-8859-5
  • KOI8-R
  • WINDOWS-1251
  • MAC-CYRILLIC
  • IBM866
  • IBM855
• Slovak
  • Windows-1250
  • ISO-8859-2
  • IBM852
  • MAC-CENTRALEUROPE
• Slovene
  • ISO-8859-2
  • ISO-8859-16
  • Windows-1250
  • IBM852
  • MAC-CENTRALEUROPE

Example

One-shot detection

import cchardet as chardet

with open(r"src/tests/samples/wikipediaJa_One_Thousand_and_One_Nights_SJIS.txt", "rb") as f:
    msg = f.read()
    result = chardet.detect(msg)
    print(result)

Streaming detection

import cchardet

detector = cchardet.UniversalDetector()
with open("bigfile.txt", "rb") as f:
    for line in f:
        detector.feed(line)
        if detector.done:
            break
detector.close()
print(detector.result)

Thread safety

cchardet.detect() is safe to call concurrently from any number of threads. Each call creates its own uchardet detector, uses it, and frees it before returning, so no state is shared between calls.

A UniversalDetector instance is not safe to share across threads. An instance holds the state of a single stream -- the native uchardet handle, the completion flags, and the leading bytes kept for BOM detection -- and feed(), close(), reset() and result all mutate it. Give each thread its own detector, or guard a shared one with your own lock:

import cchardet
from concurrent.futures import ThreadPoolExecutor

def sniff(path):
    # one detector per call, so nothing crosses a thread boundary
    with cchardet.UniversalDetector() as detector:
        with open(path, "rb") as f:
            for line in f:
                detector.feed(line)
                if detector.done:
                    break
        return detector.result

with ThreadPoolExecutor(max_workers=8) as pool:
    results = list(pool.map(sniff, paths))

On the free-threaded builds of CPython (3.13t / 3.14t) the _cchardet extension declares itself free-threading compatible, so importing cchardet does not re-enable the GIL for the whole process. The rules above do not change on those builds. Sharing one detector across threads still gives meaningless results, but it cannot corrupt the interpreter: the instance methods take a per-instance critical section, which costs nothing on ordinary GIL builds.

Note that this buys correctness, not extra parallelism -- detection itself still runs while holding the GIL on non-free-threaded builds.

Command line

A cchardetect console script is installed with the package:

$ cchardetect src/tests/samples/wikipediaJa_One_Thousand_and_One_Nights_SJIS.txt
src/tests/samples/wikipediaJa_One_Thousand_and_One_Nights_SJIS.txt: SHIFT_JIS with confidence 0.99

$ cat somefile.txt | cchardetect        # also reads from stdin

Benchmark

$ pip install -e .
$ python src/tests/bench.py

Results

CPU: AMD Ryzen 9 7950X3D

RAM: DDR5-5600MT/s 96GB

Platform: Ubuntu 24.04 amd64

Python 3.12.3

LICENSE

See COPYING file.

Contact

• Issues

Support Platforms

Prebuilt wheels are published for:

• Windows x86_64 (AMD64)
• Linux x86_64, i686, aarch64
• macOS x86_64, arm64