Unicode Tools (uchr)

A powerful command-line tool for searching and exploring Unicode characters, emoji sequences, and character properties.

🚀 Features

• Search by name: Find characters by their Unicode name
• Search by code: Look up characters by code point or range
• Search by character: Reverse lookup from character to details
• Search by block: Explore characters within Unicode blocks
• Emoji support: Full support for emoji sequences and ZWJ sequences
• CJK details: Enhanced descriptions for CJK characters using kDefinition
• Flexible output: Multiple output formats for different use cases
• Text normalization: Unicode normalization and text conversion utilities

📖 Table of Contents

• Installation
• Quick Start
• Usage Examples
• Command Reference
• Database Management
• Text Normalization
• Contributing
• License

🛠 Installation

Install from PyPI (Recommended)

pip install uchr

Install from source

git clone https://github.com/mkyutani/uchr.git
cd uchr
poetry install

Initialize database

The database is created automatically on first use of uchr search, fetching the latest Unicode version. To fetch it explicitly instead:

uchr db update

This downloads the latest Unicode data and creates a local SQLite database at:

• Linux/macOS: ~/.local/share/uchr/unicode.db
• Root users: Automatically chooses between system (/var/lib/uchr/) or personal location

The database file is created with mode 0644: anyone who can reach it can search it, but only its owner can change it (db update/use/delete). For the system location this means root.

Multiple Unicode versions can coexist in the same database — see Database Management below.

⚡ Quick Start

# Search for ghost-related characters
uchr search ghost

# Find characters in a code range
uchr search -c 1F47A-1F480

# Search by character
uchr search -x 👻

# Search within a Unicode block
uchr search -b "Emoticons"

📋 Usage Examples

Search by Name

uchr search goblin

👺 1F47A JAPANESE GOBLIN

Search by Code Range

uchr search -c 1F479-1F47B

👹 1F479 JAPANESE OGRE
👺 1F47A JAPANESE GOBLIN
👻 1F47B GHOST

Search by Character

uchr search -x 👻

👻 1F47B GHOST

Search by Unicode Block

uchr search -b "Misc_Pictographs"

Search with Details (CJK Characters)

uchr search -d "pray for happiness"

祝 795D CJK UNIFIED IDEOGRAPH-#; PRAY FOR HAPPINESS OR BLESSINGS

Output Formatting

# Simple format (characters only)
uchr search ghost -f simple
👻

# UTF-8 format
uchr search ghost -f utf8
👻 F0 9F 91 BB GHOST

# Custom delimiter
uchr search ghost -D "|"
👻|1F47B|GHOST

🔧 Command Reference

uchr

Main command with subcommands for all Unicode operations.

uchr search

Search Unicode characters with various criteria.

uchr normalize (deprecated)

Unicode text normalization and conversion. Deprecated — see Text Normalization.

uchr db

Database management operations. Multiple Unicode versions can be stored in the database at once; current_version (set by update/use) is what search uses by default.

💾 Database Management

The database is created automatically the first time uchr search runs with no local database yet — it fetches the latest Unicode version. There's no separate "create" step to run manually.

Fetch a Version

# Fetch the latest version and make it current
uchr db update

# Fetch a specific version
uchr db update --version 16.0.0

A version that is already stored is not downloaded again; update just makes it current. The exception is a version stored as unreleased draft data, which is fetched again so it picks up changes (or the final release).

Switch Between Locally Stored Versions

uchr db use 16.0.0

List Versions

uchr db list

15.0.0
16.0.0 (local) [157667 chars]
17.0.0 (current) (latest) [162626 chars]
18.0.0 (draft)

Search a Specific Version Without Switching

uchr search ghost --unicode-version 16.0.0

If that version isn't stored locally yet, this errors and tells you to run uchr db update --version 16.0.0 first — it won't silently download on a possibly-mistyped version number.

Remove Data

# Delete one version's data
uchr db delete 16.0.0

# Wipe the entire database file
uchr db delete --all

🔤 Text Normalization

Deprecated: uchr normalize will be removed in a future release. It uses the Unicode data built into Python (unicodedata.unidata_version, e.g. 15.0.0 on Python 3.12), not the uchr database, so it ignores uchr db versions. It prints a warning to stderr when used.

The uchr normalize command provides Unicode text normalization and conversion utilities.

Basic Normalization

# Normalize to NFC (default)
echo "café" | uchr normalize

# Normalize to NFD
echo "café" | uchr normalize --form nfd

# Normalize to NFKC
echo "file" | uchr normalize --form nfkc

# Normalize to NFKD
echo "file" | uchr normalize --form nfkd

Text Conversion

# Convert fullwidth to halfwidth
echo "ＨｅｌｌｏＷｏｒｌｄ" | uchr normalize --halfwidth

# Show detailed binary representation
echo "café" | uchr normalize --detail

# Compare all normalization forms
echo "café" | uchr normalize --compare

Interactive Mode

# Read from stdin (interactive)
uchr normalize
# Type text and press Ctrl+D to process

🌟 Advanced Examples

Finding Emoji Sequences

# National flags
uchr search -b "RGI_Emoji_Flag_Sequence"

# Family emoji with ZWJ sequences
uchr search family

Terminal Display vs. Browser/Application Support

Many terminals don't properly display complex emoji sequences, but the characters work correctly when copied to browsers or applications.

National Flag Example

When searching for flags, you might see separate letters in your terminal:

uchr search -b "RGI_Emoji_Flag_Sequence" | grep -i norway

🇳🇴 1F1F3 1F1F4 flag: Norway

Even though you see two separate letters (🇳🇴) in the terminal, when you copy and paste them into a browser or application like Twitter, they combine to display the Norwegian flag 🇳🇴.

ZWJ Sequence Example

The same applies to Zero Width Joiner (ZWJ) sequences. Complex emoji like family groups or professional emoji might not render correctly in terminals:

uchr search "polar bear"

In a terminal without proper font support:

But when pasted in Twitter or other applications:

💡 Tip: This is expected behavior. The Unicode data is correct, and the characters will work properly in applications that support modern emoji rendering.

Pipe Operations

# Get just the character
uchr search ghost -f simple

# First match only
uchr search snow -1

# Custom format for scripting
uchr search ghost -D "," | cut -d',' -f1

Complex Searches

# CJK characters with specific meanings
uchr search -d "dragon"

# Characters in multiple blocks
uchr search -b "Mathematical" | head -10

🏗 Data Sources

This tool fetches official Unicode data directly from unicode.org — the Unicode Character Database (ucdxml) and Emoji Sequences for whichever version you request with uchr db update. By default it tracks the latest released version; run uchr db list to see everything currently published.

🏗 Development

This project uses Poetry for dependency management and packaging.

Development Setup

# Clone the repository
git clone https://github.com/mkyutani/uchr.git
cd uchr

# Install dependencies
poetry install

# Run tests
poetry run pytest

# Format code
poetry run black src/

# Lint code
poetry run ruff check src/

Building

poetry build

Releasing

Releases are published by GitHub Actions (.github/workflows/release.yml). Bump version in pyproject.toml, merge it to main, then push a matching tag:

git tag v1.0.0
git push origin v1.0.0

The tag must equal v + the pyproject.toml version, or the build fails. The workflow then runs the tests, builds the sdist and wheel, publishes them to TestPyPI, installs the release back from TestPyPI as a smoke test, waits for approval on the pypi environment, publishes to PyPI, and creates a GitHub Release with the built files attached.

Uploads use Trusted Publishing, so no API tokens are stored in the repository. One-time setup:

1. PyPI: in the uchr project's Publishing settings, add a GitHub publisher — owner mkyutani, repository uchr, workflow release.yml, environment pypi.
2. TestPyPI: add the same publisher with environment testpypi (as a pending publisher if the project does not exist there yet).
3. GitHub: create the environments testpypi and pypi under Settings → Environments. Give pypi a required reviewer so the production upload waits for approval, and restrict both to v* tags.

Tests also run on every pull request and push to main (.github/workflows/ci.yml).

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes and add tests
4. Run the test suite: poetry run pytest
5. Format your code: poetry run black src/
6. Lint your code: poetry run ruff check src/
7. Commit your changes (git commit -m 'Add amazing feature')
8. Push to the branch (git push origin feature/amazing-feature)
9. Open a Pull Request

Code Quality

This project maintains high code quality standards:

• Type hints: All functions should include type annotations
• Testing: New features should include appropriate tests
• Documentation: Update README and docstrings for new features
• Code style: Follow Black formatting and Ruff linting rules

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Unicode Consortium for maintaining Unicode standards
• Poetry for modern Python packaging
• Contributors and users of this project

📊 Project Status

• Current Version: 1.0.0
• Python Support: 3.8+
• Unicode Version: tracks the latest released version by default; multiple versions can coexist locally (see uchr db list)
• Package Name: uchr (on PyPI)
• Repository: mkyutani/uchr

For the latest updates and roadmap, see our GitHub Issues.