ASR Quality Enhancement Layer

A production-grade post-processing pipeline for improving Parakeet Multilingual ASR outputs. This system addresses common ASR challenges including low-confidence word detection, numeric sequence reconstruction, domain vocabulary correction, and LLM-based contextual polishing.

🎯 Overview

The ASR Enhancement Layer sits between the Parakeet ASR engine and downstream applications, providing:

• Error Detection: Identifies low-confidence spans, anomalies, and incomplete sequences
• Secondary ASR: Re-transcribes problematic segments using Whisper/Riva
• Numeric Reconstruction: Recovers missing digits in phone numbers, OTPs, amounts
• Domain Vocabulary: Applies domain-specific terminology corrections
• LLM Polishing: Fixes grammar and coherence with anti-hallucination safeguards
• Hypothesis Fusion: Combines multiple ASR outputs using weighted scoring

📐 Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                       ASR QUALITY ENHANCEMENT LAYER                      │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐               │
│  │   Parakeet   │───▶│    Error     │───▶│   Re-ASR     │               │
│  │   ASR Input  │    │  Detection   │    │  Processing  │               │
│  └──────────────┘    └──────────────┘    └──────────────┘               │
│         │                   │                   │                        │
│         │            ┌──────┴──────┐            │                        │
│         │            │ • Confidence │           │                        │
│         │            │ • Anomalies  │           │                        │
│         │            │ • Numeric    │           │                        │
│         │            └─────────────┘            │                        │
│         ▼                                       ▼                        │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐               │
│  │   Numeric    │───▶│   Domain     │───▶│  Hypothesis  │               │
│  │ Reconstruct  │    │   Vocab      │    │    Fusion    │               │
│  └──────────────┘    └──────────────┘    └──────────────┘               │
│         │                   │                   │                        │
│         │            ┌──────┴──────┐            │                        │
│         │            │ • Lexicons   │           │                        │
│         │            │ • Fuzzy Match│           │                        │
│         │            │ • Phonetic   │           │                        │
│         │            └─────────────┘            │                        │
│         ▼                                       ▼                        │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐               │
│  │     LLM      │───▶│  Validation  │───▶│   Enhanced   │               │
│  │  Polishing   │    │  & Scoring   │    │   Output     │               │
│  └──────────────┘    └──────────────┘    └──────────────┘               │
│         │                   │                                            │
│         │            ┌──────┴──────┐                                     │
│         │            │ • Consistency│                                    │
│         │            │ • Perplexity │                                    │
│         │            │ • Completeness│                                   │
│         │            └─────────────┘                                     │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

📁 Project Structure

asr_enhancer/
├── __init__.py              # Package exports
├── core.py                  # Main EnhancementPipeline orchestrator
├── detectors/               # Error detection modules
│   ├── confidence_detector.py  # Low-confidence span detection
│   ├── anomaly_detector.py     # Segmentation/repetition anomalies
│   └── numeric_gap_detector.py # Incomplete number sequences
├── resynthesis/             # Secondary ASR processing
│   ├── segment_extractor.py    # Audio segment extraction
│   ├── secondary_asr.py        # ASR backend abstraction
│   ├── whisper_backend.py      # Whisper integration
│   └── riva_backend.py         # NVIDIA Riva integration
├── numeric/                 # Numeric reconstruction
│   ├── pattern_analyzer.py     # Number pattern detection
│   ├── sequence_reconstructor.py # Digit recovery
│   └── validators.py           # Phone/OTP/card validation
├── vocab/                   # Domain vocabulary
│   ├── lexicon_loader.py       # Lexicon loading
│   ├── term_matcher.py         # Term matching (fuzzy/phonetic)
│   └── corrector.py            # Vocabulary correction
├── llm/                     # LLM integration
│   ├── context_restorer.py     # Main LLM processor
│   ├── prompt_templates.py     # Anti-hallucination prompts
│   └── providers.py            # OpenAI/Ollama/Anthropic
├── fusion/                  # Hypothesis fusion
│   ├── fusion_engine.py        # N-best combination
│   ├── scorers.py              # Acoustic/LM scoring
│   └── selector.py             # Candidate selection
├── validators/              # Quality validation
│   ├── consistency_checker.py  # Content consistency
│   ├── perplexity_scorer.py    # Fluency scoring
│   └── completeness_validator.py # Gap detection
├── utils/                   # Utilities
│   ├── config.py               # Configuration management
│   ├── logging.py              # Structured logging
│   ├── audio.py                # Audio utilities
│   └── text.py                 # Text utilities
├── api/                     # FastAPI service
│   ├── main.py                 # Application entry
│   ├── routes.py               # API endpoints
│   └── schemas.py              # Pydantic models
└── cli/                     # Command-line interface
    └── __init__.py             # CLI commands

🚀 Quick Start

Installation

# Clone the repository
git clone <repository-url>
cd sound-web

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -e ".[all]"

Basic Usage

from asr_enhancer import EnhancementPipeline
from asr_enhancer.utils import Config

# Initialize pipeline
config = Config(
    confidence_threshold=0.7,
    llm_provider="ollama",
    llm_model="llama3.1",
)
pipeline = EnhancementPipeline(config)

# Enhance transcript
result = await pipeline.enhance(
    transcript="my phone number is nine one two tree four five six seven ate nine",
    word_timestamps=[
        {"word": "my", "start": 0.0, "end": 0.2},
        {"word": "phone", "start": 0.2, "end": 0.5},
        # ... more timestamps
    ],
    word_confidences=[0.95, 0.92, 0.89, 0.98, 0.85, 0.91, 0.88, 0.45, 0.92, 0.87, 0.90, 0.93, 0.38, 0.91],
)

print(f"Enhanced: {result.enhanced_transcript}")
print(f"Confidence improvement: {result.confidence_improvement:.2%}")

API Server

# Start the API server
asr-enhancer serve --host 0.0.0.0 --port 8000

# Or with Docker
docker-compose up -d

CLI Usage

# Enhance a transcript file
asr-enhancer enhance input.json -o output.json --format json

# Analyze without enhancement
asr-enhancer analyze input.json

# Check dependencies
asr-enhancer check

🔌 API Endpoints

POST /api/v1/enhance

Enhance a transcript using the full pipeline.

{
  "transcript": "raw transcript text",
  "word_timestamps": [{"word": "...", "start": 0.0, "end": 0.1}],
  "word_confidences": [0.9, 0.8, ...],
  "audio_path": "/path/to/audio.wav",  // optional
  "domain_lexicon": {"term": ["variant1", "variant2"]}  // optional
}

POST /api/v1/analyze

Analyze transcript without enhancement.

GET /api/v1/diagnostics

Get pipeline diagnostics and configuration.

GET /health

Health check endpoint.

⚙️ Configuration

Configuration can be set via:

1. Configuration file (config.json)
2. Environment variables
3. Code

Key Settings

Environment Variables

export ASR_CONFIDENCE_THRESHOLD=0.7
export ASR_LLM_PROVIDER=ollama
export ASR_LLM_MODEL=llama3.1
export ASR_LLM_API_KEY=your-api-key  # For OpenAI/Anthropic
export ASR_LOG_LEVEL=INFO

📊 Fusion Formula

The hypothesis fusion uses weighted scoring:

$$Score = \alpha \cdot P_{confidence} + \beta \cdot S_{LM} + \gamma \cdot S_{acoustic}$$

Where:

• $\alpha$ = Original ASR confidence weight (default: 0.4)
• $\beta$ = Language model score weight (default: 0.35)
• $\gamma$ = Acoustic similarity weight (default: 0.25)

🛡️ Anti-Hallucination Safeguards

The LLM polishing stage includes multiple safeguards:

1. Number Preservation: All numeric sequences must appear unchanged
2. Overlap Validation: Enhanced text must maintain >50% word overlap
3. Grounding Prompts: Explicit instructions to only fix errors, not add content
4. Retry Logic: Multiple attempts with validation between each

🧪 Testing

# Run all tests
pytest

# Run with coverage
pytest --cov=asr_enhancer --cov-report=html

# Run specific test file
pytest tests/test_detectors.py -v

🐳 Docker Deployment

# Build and run with Docker Compose
docker-compose up -d

# View logs
docker-compose logs -f asr-enhancer

# Pull Ollama model (first time)
docker exec asr-enhancer-ollama ollama pull llama3.1

📈 Next Implementation Steps

Phase 1: Core Implementation (Current)

• Project scaffolding
• Module stubs with interfaces
• FastAPI service structure
• CLI tool skeleton
• Docker configuration

Phase 2: Detection & Analysis

• Implement sliding window confidence detection
• Add acoustic anomaly detection
• Build numeric gap pattern matching
• Unit tests for detectors

Phase 3: Secondary ASR

• Whisper backend integration
• Audio segment extraction
• Batch processing support
• Latency optimization

Phase 4: Numeric Reconstruction

• Pattern analyzer for phone/OTP/amounts
• Acoustic confusion correction
• Sequence completion rules
• Validation with Luhn checks

Phase 5: Domain Vocabulary

• Lexicon file format and loading
• Fuzzy matching implementation
• Phonetic matching (Soundex/Metaphone)
• Case-preserving correction

Phase 6: LLM Integration

• Prompt template refinement
• Multi-provider support testing
• Anti-hallucination validation
• Fallback strategies

Phase 7: Fusion & Validation

• N-best hypothesis fusion
• Language model perplexity scoring
• Consistency validation
• Completeness checks

Phase 8: Production Hardening

• Performance benchmarks
• Memory optimization
• Streaming support
• Monitoring & metrics
• Load testing

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests: pytest
5. Run linting: ruff check . && black --check .
6. Submit a pull request

📝 License

MIT License - see LICENSE file for details.

🔗 Related Projects

• NVIDIA Parakeet - Multilingual ASR
• OpenAI Whisper - General-purpose ASR
• NVIDIA Riva - Streaming ASR platform