cnbe32 1.0.2

CNBE-32: Chinese Native Binary Encoding — Structured 32-bit CJK encoding for AI and hardware

CNBE-32
Chinese Native Binary Encoding

English · 简体中文 · English mirror

A 32-bit structural fingerprint for CJK characters — built for people who wonder what Chinese text would look like if it were designed closer to the metal.

CNBE-32 is a research prototype. The packaged Python SDK currently targets 20,902 Basic CJK entries. The broader 97,686 CJK figure is an intended / experimental extended scope, not current packaged SDK coverage.

---

Why this is interesting

Unicode tells computers which character this is.

CNBE-32 asks a different question:

Can part of a CJK character's visual and structural logic be carried directly in a compact binary form?

That makes CNBE-32 interesting for experiments in CJK-aware embeddings, low-level lookup tables, hardware-friendly text features, and language-specific model inputs.

---

The idea in one picture

31              24 23        19 18     15 14                 4 3        0
┌────────────────┬────────────┬─────────┬─────────────────────┬──────────┐
│ Radical/Radix  │  Stroke    │ Struct  │     Glyph Index     │   Ext    │
│     8 bits     │  5 bits    │ 4 bits  │       11 bits       │  4 bits  │
└────────────────┴────────────┴─────────┴─────────────────────┴──────────┘

Think of it as a compact structural fingerprint, not a replacement for Unicode.

---

Quick start

python -m pip install .

from cnbe32 import encode_cnbe, decode_cnbe, bit_hamming_distance

a = encode_cnbe(radix=72, stroke=8, struct=1, index=123, ext=0)
b = encode_cnbe(radix=72, stroke=9, struct=1, index=124, ext=0)

print(decode_cnbe(a))
print(bit_hamming_distance(a, b))

---

What is stable today

• CNBE-32 field encoding and decoding
• strict validation of all bitfield ranges
• true bit-level Hamming distance and legacy field-weighted distance
• optional SQLite database lookup
• explicit SkillTable construction for experiments
• wheel build, pip install, pytest, ruff, GitHub Actions CI

---

What is experimental

• LLM prompting and feature experiments
• JEPA-style representation learning
• RISC-V and hardware instruction prototypes
• OS and kernel-level experiments
• finance, biology, physics, and social-science-style experiments

These should be interpreted as preliminary research prototypes unless the corresponding directory includes fixed datasets, reproducible scripts, baseline comparisons, random seeds, and clear train/test separation.

---

Coverage terminology

Term	Meaning
Packaged Python SDK database	20,902 Basic CJK entries (shipped in the wheel)
Experimental extended scope	97,686 CJK characters as a design / research target
Experiment-specific coverage	depends on the dataset and reproduction script for each experiment

Claims about collision rate, full coverage, or extended CJK breadth should be interpreted only within the scope of the specific dataset and script used for that experiment.

---

Evidence level

This repository contains research prototypes and early experiments. Results should be interpreted as preliminary unless the corresponding experiment includes:

• fixed dataset versions,
• reproducible scripts,
• baseline comparisons,
• random seeds or deterministic settings,
• raw outputs or result artifacts,
• and clear train/test separation where applicable.

---

Bitfield layout

Field	Bits	Description
Radical / Radix	8	Radical or structural root field
Stroke	5	Stroke-count field
Structure	4	Character structure field
Glyph Index	11	Basic CJK glyph index field
Extension	4	Experimental extension field

---

Python SDK example

from cnbe32 import (
    encode_cnbe, decode_cnbe,
    bit_hamming_distance, field_weighted_distance,
)

a = encode_cnbe(radix=72, stroke=8, struct=1, index=123, ext=0)
b = encode_cnbe(radix=72, stroke=9, struct=1, index=124, ext=0)

print(decode_cnbe(a))
print(bit_hamming_distance(a, b))
print(field_weighted_distance(a, b))

---

For geeks

If you like...	CNBE-32 gives you...
bitfields	a fixed 32-bit CJK structure layout
language internals	radical, stroke, structure, glyph-index fields
ML features	compact CJK-aware feature inputs
hardware experiments	a layout testable near RISC-V / instruction prototypes
weird text encoding ideas	a research sandbox for Chinese-native representation

---

For Chinese language enthusiasts

Chinese characters are not just arbitrary symbols. Many carry visible structure: components, strokes, layout, and historical form.

CNBE-32 does not claim to fully understand characters. It simply asks whether some of that visible structure can be encoded in a way computers can use directly.

---

Roadmap

1. Keep the Python SDK build, install, test, and lint pipeline green.
2. Add reproducible scripts for each experiment.
3. Separate stable SDK claims from experiment-specific claims.
4. Publish dataset provenance and coverage validation scripts.
5. Add golden vectors shared across Python, C, Rust, and hardware prototypes.
6. Add benchmark baselines (Unicode codepoint, one-hot, IDS, learned embeddings).

---

Implementation consistency

CNBE-32 includes machine-readable golden vectors in spec/golden_vectors.json. These vectors define canonical bitfield encode/decode examples for Python, C, Rust, and hardware-oriented implementations. The same vector set is now exercised by Python tests, a minimal C consistency test, and a minimal Rust consistency test.

Project maintenance

• Changelog
• Release process
• v1.0.2 release notes draft
• Contributing guide
• Security policy

License

MulanPSL-2.0