transphrase 0.1.5

AI-powered web novel translation tool

TransPhrase

Version 0.1.5 (ALPHA) ⚠️ .

TransPhrase is an AI-powered tool for translating web novels and other text content using various language models.

Features

• Supports translation between multiple languages (English, Chinese, Japanese, Korean, Spanish, French, German, etc.)
• Automatic language detection for source files
• Interactive model selection with real-time filtering
• Automatic caching of translations to avoid redundant API calls
• Adaptive rate limiting to prevent API quota exhaustion
• Multi-threaded processing for faster translation
• Database tracking of translation jobs and progress
• Plugin system for custom prompt templates
• Semantic text chunking for improved translation quality
• Dependency injection for better testability and maintainability
• Support for both translation and text polishing modes
• HTML processing with structure preservation
• Batch translation with stable delimiters
• Robust error handling and recovery mechanisms
• Advanced context tracking for more consistent translations

Project Structure

The TransPhrase project is organized into several core modules:

transphrase/
├── api/               # API handlers and client implementations
├── cache/             # Translation caching and context management
├── cli/               # Command-line interface implementation
├── core/              # Core processing logic and configuration
├── database/          # Database models and operations
├── formats/           # File format handlers and processors
├── models/            # Language model implementations
├── plugins/           # Plugin system implementation
├── rate_limiting/     # Rate limiting and API quota management
└── ui/                # User interface components

Each module is designed to be independently testable and maintainable, with clear separation of concerns.

Operation Modes

TransPhrase supports two operation modes:

Translation Mode

Converts text between different languages while maintaining tone, style, and meaning.

• Perfect for web novels, technical documents, or any text content
• Preserves character names, terms, and stylistic elements
• Optimized for context-aware translation across multiple files
• HTML structure preservation for web content

Polish Mode

Improves existing translations for better readability and fluency.

• Enhances grammar, flow, and natural phrasing
• Maintains character authenticity and consistent terminology
• Adds emotional depth while preserving the original plot
• Ideal for refining machine-translated content
• Works in any supported language, not just English

Installation

Using pip

pip install transphrase

From source

git clone https://github.com/shinyPy/TransPhrase.git
cd TransPhrase
pip install -e .

Usage

Once installed, you can run TransPhrase from the command line:

transphrase

Follow the interactive prompts to configure your translation job:

1. Select operation mode (translate or polish)
2. Choose a language model
3. Select source and target languages (or use auto-detection)
4. Enter the source directory containing text files to process
5. Select an output directory
6. Configure advanced options (caching, threading, etc.)
7. Start the translation process

Command-line Arguments

TransPhrase supports the following command-line arguments:

# Run without saving configuration
transphrase --no-save-config

# Run with a fresh configuration (ignoring existing saved config)
transphrase --overwrite-config

# Disable automatic language detection:
transphrase --no-auto-detect

Language Support

TransPhrase supports translation between languages depending on the capabilities of the selected language model (LLM). Commonly supported languages include:

• English
• Chinese
• Japanese
• Korean
• Spanish
• French
• German
• Russian
• Italian
• Portuguese
• Dutch
• Arabic
• Hindi
• Vietnamese
• Thai
• Indonesian

Refer to the documentation of your chosen LLM for the full list of supported languages.

Configuration

TransPhrase supports several configuration methods:

Environment Variables

• MASTER_API_KEY: Your API key
• API_BASE_URL: Base URL for API calls (defaults to https://api.electronhub.top)

Configuration File

TransPhrase can be configured using a JSON file located at ~/.transphrase/config.json. This allows you to save your settings and reuse them without going through the interactive prompts each time.

Example config.json:

{
  "api_key": "your-api-key",
  "base_url": "https://api.electronhub.top",
  "model": "deepseek-llm-67b-chat",
  "source_language": "Japanese",
  "target_language": "English",
  "skip_existing": true,
  "workers": 8,
  "use_cache": true,
  "cache_ttl": 604800,
  "db_path": "~/.transphrase/novels.db",
  "plugins": {
    "prompt_template": "LightNovelTemplate",
    "processors": ["NameConsistencyProcessor", "DialogFormatter"]
  },
  "series_id": "my-light-novel-series",
  "auto_add_characters": false,
  "name_frequency_threshold": 15,
  "auto_detect_language": true,
  "polish_style": "natural"
}

Advanced Features

Automatic Language Detection

TransPhrase can automatically detect the source language of your files:

• Samples text from your files to determine the dominant language
• Supports a wide range of languages including Chinese, Japanese, Korean, and more
• Detects language with high accuracy even from small text samples
• Falls back to user selection if detection fails
• Can be disabled with the --no-auto-detect flag

Rate Limiting

TransPhrase includes an adaptive rate limiter that:

• Dynamically adjusts to API feedback
• Handles 429 (Too Many Requests) errors gracefully
• Implements backoff strategies for reliable operation
• Adjusts concurrent workers based on API limits

Translation Caching

All translations are cached locally to:

• Avoid redundant API calls for identical text
• Reduce costs and speed up batch processing
• Provide resilience during network interruptions
• Enhanced caching with glossary term application

Intelligent Text Chunking

TransPhrase uses semantic text chunking that:

• Respects sentence and paragraph boundaries
• Maintains context across chunks
• Optimizes chunk size for better translation quality
• Preserves document structure

HTML Processing

• Preserves document structure and formatting during translation
• Maintains HTML tags, attributes, and nested elements
• Handles special cases like select/option elements properly
• Processes large HTML files efficiently with streaming techniques
• Ensures valid HTML output through structure validation and recovery

Batch Translation

• Stable, UUID-based delimiters for consistent batch processing
• Robust handling of API responses that don't maintain delimiters
• Graceful fallback to individual translation when needed
• Efficient caching of batch results for improved performance

Plugin System

TransPhrase includes a flexible plugin system that allows you to extend functionality without modifying core code:

Types of Plugins

1. Prompt Templates: Customize the AI system prompt for different translation scenarios

   • Optimize translations for specific content types (novels, technical documentation, etc.)
   • Create genre-specific translation styles
   • Adapt to different writing conventions

2. Processor Modules: Modify text before or after translation

   • Ensure consistency in character names and terminology
   • Preserve formatting elements like code blocks
   • Apply content-specific post-processing

Using Plugins

Plugins are automatically discovered and presented during the configuration process:

Plugin Selection:
Do you want to use any plugins? (y/n) [n]: y

Available Prompt Templates:
1. NovelTranslationTemplate: Specialized template for translating novels
Use a custom prompt template? (y/n) [n]: y
Select prompt template [1]: 1

Available Processor Modules:
1. NameConsistencyProcessor: Maintains consistency of character names
Use text processor modules? (y/n) [n]: y
Select processor (0 to finish) [0]: 1

Creating Custom Plugins

TransPhrase automatically sets up plugin directories with examples:

~/.transphrase/plugins/
├── README.md                       # Plugin documentation
├── prompt_templates/               # Custom prompt templates
│   └── novel_template.py           # Example template
└── processors/                     # Text processor modules
    └── name_consistency.py         # Example processor

See the detailed documentation in plugins.md for instructions on creating custom plugins.

Usage Examples

Basic Translation with Auto-Detection

# Create a sample Chinese file
echo "这是一个测试文件。它包含中文文本。" > ~/test-zh.txt

# Run TransPhrase with auto-detection
transphrase translate process-files

# Follow prompts to select output directory
# The language will be automatically detected as Chinese

Manual Language Selection

# Disable auto-detection if you want to manually specify the language
transphrase translate process-files --no-auto-detect

# Follow prompts to select languages and other options

Translating HTML Content

# Run TransPhrase with HTML files in your source directory
transphrase

# HTML files will be processed with special handling to preserve structure

Development

The project follows test-driven development practices with comprehensive test coverage. The test suite includes:

tests/
├── __init__.py
├── test_api_handler.py       # API handler tests
├── test_basic.py             # Basic functionality tests
├── test_cache.py             # Cache system tests
├── test_config.py            # Configuration tests
├── test_config_validator.py  # Config validation tests
├── test_database.py          # Database operation tests
├── test_file_processor.py    # File processing tests
├── test_glossary.py          # Glossary and terminology tests
├── test_html_processor.py    # HTML processing tests
├── test_integration.py       # Integration tests
├── test_model_selector.py    # Model selection tests
├── test_rate_limiter.py      # Rate limiting tests

Testing Guidelines

• Unit tests should cover all core functionality
• Integration tests verify module interactions
• Tests are run automatically on every commit via CI/CD
• Code coverage is maintained above 90%
• Mocking is used extensively for external dependencies

Please see CONTRIBUTING for detailed development guidelines.

CI/CD

TransPhrase uses GitHub Actions for continuous integration and deployment:

• Automatic testing on multiple Python versions (3.10+)
• Code quality checks (linting, formatting, type checking)
• Coverage reporting
• Automated releases to PyPI

Requirements

System Requirements

• Python 3.10 or higher
• 4GB RAM minimum (8GB recommended)
• 500MB disk space
• Internet connection for API access

Python Dependencies

Core dependencies (automatically installed):

• requests >= 2.31.0
• tqdm >= 4.66.1
• pydantic >= 2.5.0
• sqlalchemy >= 2.0.0
• python-dotenv >= 1.0.0
• beautifulsoup4 >= 4.12.0
• rich >= 13.0.0

Changelog

Version 0.1.5 (ALPHA)

• Added robust HTML processing with structure preservation
• Improved batch translation with stable UUID-based delimiters
• Enhanced error handling and recovery throughout the application
• Added quality assessment improvements for more consistent evaluations
• Fixed various bugs affecting HTML and text processing
• Added comprehensive testing for HTML processing components

Version 0.1.0 (ALPHA)

• Initial release with basic functionality
• Translation and polish modes
• Support for multiple languages and models
• File processing and caching system
• Database integration for tracking translation jobs