charded

charded is a type-safe, generic string/bytes container with automatic charset detection. It wraps anything you throw at it — str, bytes, even int — and lets you work with it as either text or raw bytes, without guessing encodings.

Installation

Via uv:

uv add charded

Features

• Generic type safety — Str[str] or Str[bytes] with compile-time checking
• Automatic charset detection — combines charset-normalizer, python-magic, and internal heuristics
• MIME type sniffing — detect content type via libmagic
• Zero-copy lazy evaluation — cached properties, decode/encode only when needed
• Full str proxy — use len(), [], split(), startswith(), tokenize(), etc. directly on the container
• Binary detection — adaptive hybrid sampling strategy for large files
• Utility functions — to_ascii(), to_bytes(), to_text() for quick conversions

Quick Start

Wrap anything

from charded import Str

# From bytes with unknown encoding
x = Str(b"\xc3\xa9l\xc3\xa8ve")  # UTF-8 French text
print(x.string)   # "élève"
print(x.charset)  # "utf-8"

# From str
y = Str("hello")
print(y.bytes)    # b"hello"

# Or use built-in constructors directly
print(bytes(Str("a")))   # b'a'
print(str(Str(b"asd")))  # "asd"

Automatic charset detection

raw = b"\xef\xf0\xe8\xea"  # some Russian text in unknown encoding
s = Str(raw)

print(s.charset)           # detected encoding, e.g. "cp1251"
print(str(s))              # decoded text
print(bytes(s))            # original bytes

Detection order:

1. Explicit charset= argument if provided
2. charset-normalizer result
3. Internal heuristic scan (adaptive binary-offset sampling)
4. Fallback to ASCII

MIME type detection

data = Str(b"%PDF-1.4\n1 0 obj")
print(data.mime)
# {"type": "application/pdf", "description": "PDF document, version 1.4"}

Use as a string

Str proxies almost all str methods, so you can use it directly:

s = Str(b"hello world")

len(s)           # 11
s[0:5]           # "hello"
s.split()        # ["hello", "world"]
s.startswith("h")  # True
s.upper()        # "HELLO WORLD"

Tokenize

s = Str("foo-bar_baz, qux")
print(s.tokenize(r"[\w]+"))
# ("foo", "bar", "baz", "qux")

Utility functions

from charded import to_ascii, to_bytes, to_text

# Force ASCII transliteration
text = to_ascii("Café résumé naïve")
# "Cafe resume naive"

# Convert anything to bytes
raw = to_bytes("hello", charset="utf-8")

# Convert anything to text
text = to_text(b"hello", charset="utf-8")

Type Safety

Str is a true generic. Use Str[str] or Str[bytes] for explicit typing:

from charded import Str

def process(data: Str[bytes]) -> str:
    return data.string.upper()

process(Str(b"hello"))   # OK
process(Str("hello"))     # type error (mypy/pyright will catch it)

Type aliases are also provided:

from charded.string import StrBytes, StrText

def load(raw: StrBytes) -> StrText:
    return Str(raw.string)

How charset detection works

For small inputs (< 64 bytes), charded does a full scan. For medium and large inputs it uses an adaptive hybrid binary-offset strategy:

• Header-heavy sampling — first 1 KB scanned with moderate density
• Exponential sparse middle — O(log n) samples for large files
• Footer sampling — last 1 KB checked for trailing signatures

This guarantees detection of BOMs, null-byte injection, and encoding signatures without reading the entire file into memory.

API Reference

Str[T]

All standard str methods (split, startswith, upper, etc.) are proxied.

License

BSD 3-Clause. See LICENSE for details.