title: charded description: Type-safe, generic string/bytes container with charset detection

[ref: #charded]

charded

charded is a small Python library for working with str and bytes as a single, type-safe container. It detects charsets automatically, converts lazily between text and bytes, and proxies the familiar str API so the wrapper feels invisible.

It gives you four public entry points:

• varchar — a generic str | bytes container.
• to_text — convert str or bytes to text.
• to_bytes — convert str or bytes to bytes.
• to_ascii — decode bytes to text with a default ASCII charset.

[ref: #installation]

Installation

Install with uv:

uv add charded

Or with any PEP 517-compatible tool:

pip install charded

[ref: #str]

Wrapping text and bytes with varchar

varchar[T] is parameterized by T, which is either str or bytes. Pass any str or bytes object and read it back through str(), bytes(), .text, or .bytes.

from charded import varchar

s = varchar(b"hello")
assert str(s) == "hello"
assert bytes(s) == b"hello"
assert s.text == "hello"
assert s.bytes == b"hello"

t = varchar("héllo")
assert str(t) == "héllo"
assert bytes(t) == b"h\xc3\xa9llo"

[ref: #charset-detection]

Charset detection

varchar detects the content charset using charset-normalizer, internal heuristics, and an adaptive binary-offset scan. Access .charset to see the result.

from charded import varchar

assert varchar(b"hello").charset == "ascii"
assert varchar(b"hello\x00world").charset == "bin"

The detection order is:

1. Explicit charset= argument if provided.
2. Result from charset-normalizer.
3. Internal heuristic scan.
4. Fallback to ASCII for bytes or a compatible charset for text.

[ref: #mime-detection]

MIME type detection

varchar.mime uses python-magic to sniff the content type and description. It returns None when detection fails.

from charded import varchar

mime = varchar(b'{"a": 1}').mime
assert mime is not None
assert mime["type"] == "application/json"

[ref: #string-api]

Using varchar as a string

varchar proxies common str operations, so you can slice, iterate, compare, and call most str methods directly.

from charded import varchar

s = varchar(b"hello world")

assert len(s) == 11
assert s[0:5] == "hello"
assert s.startswith("hello")
assert s.split() == ["hello", "world"]
assert s.upper() == "HELLO WORLD"
assert "world" in s

[ref: #tokenize]

Tokenizing with varchar.tokenize

varchar.tokenize splits the string using a regular expression. Capturing groups are not allowed; use non-capturing groups (?:...) instead.

from charded import varchar

s = varchar("foo-bar_baz, qux")
assert s.tokenize(r"[a-z]+") == ("foo", "bar", "baz", "qux")

[ref: #utility-functions]

Utility functions

to_text, to_bytes, and to_ascii perform quick conversions without creating a persistent varchar instance.

from charded import to_ascii, to_bytes, to_text

assert to_text(b"hello") == "hello"
assert to_bytes("hello") == b"hello"
assert to_ascii(b"hello") == "hello"

[ref: #type-safety]

Type safety and aliases

varchar is a true generic. Use varchar[bytes] or varchar[str] to constrain input and let type checkers catch mismatches at compile time.

from charded import varchar


def process(data: varchar[bytes]) -> str:
    return data.text.upper()


assert process(varchar(b"hello")) == "HELLO"

Convenience aliases are available in charded.varchar:

from charded.varchar import StrBytes, StrText

raw: StrBytes = varchar(b"hello")
text: StrText = varchar(raw.text)

assert text.text == "hello"

[ref: #full-example]

Full example

from charded import varchar, to_bytes, to_text

raw = b"h\xc3\xa9llo"

s = varchar(raw)
assert str(s) == "héllo"
assert bytes(s) == raw
assert s.charset is not None
assert s.startswith("hél")

# Quick conversions without wrapping
assert to_text(raw) == "héllo"
assert to_bytes("héllo") == raw