ArabSeg

Deterministic, tashkeel-preserving morphological segmentation for fully vocalized Modern Standard Arabic.

ArabSeg is a Python library and command-line application for transparent, resource-driven analysis of fully vocalized Modern Standard Arabic (MSA) words. It preserves the exact Unicode input, applies explicit lexical and morphotactic rules, retains every supported analysis, and produces deterministic results suitable for research, testing, and reproducible NLP workflows.

Development status: ArabSeg 0.19.0 is an alpha release. Public interfaces and linguistic resources may still change before 1.0.0.

Key Features

• Exact preservation of Arabic characters and tashkeel.
• Inventory-backed Unicode and repertoire validation.
• Grapheme parsing and orthographic validation.
• Explicit lexicon and morphotactic YAML resources.
• Proclitic, stem, and enclitic segmentation.
• Feature unification across candidate segments.
• Deterministic ambiguity grouping without hidden ranking.
• Structured Python results and deterministic JSON output.
• Command-line tools for analysis, validation, resource checking, and diagnostics.
• Optional deterministic caching and performance instrumentation.

Scope

The current release targets one fully vocalized MSA word at a time.

ArabSeg does not currently provide:

• automatic diacritization;
• unvocalized or partially vocalized analysis;
• dialectal Arabic processing;
• sentence tokenization or contextual disambiguation;
• probabilistic ranking or neural prediction.

See Limitations for the complete scope statement.

Requirements

• Python 3.10 or later
• PyYAML 6.x

Installation

Install from the repository

git clone <repository-url>
cd ArabSeg
python -m venv .venv

Activate the environment.

Windows PowerShell

.venv\Scripts\Activate.ps1

Linux or macOS

source .venv/bin/activate

Install ArabSeg:

python -m pip install --upgrade pip
python -m pip install -e .

For development tools:

python -m pip install -e ".[dev]"

Confirm the installation:

arabseg version
python -c "import arabseg; print(arabseg.__version__)"

See Installation for platform-specific details.

Quick Validation Example

The public Unicode validation layer can be used without constructing the complete morphological pipeline:

from arabseg import UnicodeValidator

result = UnicodeValidator().validate("كِتَابُ")

print(result.valid)
print(result.errors)

This validates supported Unicode and repertoire constraints. Structural orthography and full-vocalization policy decisions are separate validation stages.

Morphological Analysis

A complete analyzer is constructed explicitly from:

1. a lexicon resource;
2. a morphotactics resource;
3. their expected versions;
4. an approved vocalization policy.

The repository includes a compact test resource set under tests/golden/resources/.

from pathlib import Path

from arabseg.configuration import (
    AnalyzerConfiguration,
    build_default_analyzer,
)
from arabseg.segmenter.engine import (
    VocalizationDecision,
)


class ExampleVocalizationPolicy:
    """Minimal example policy for the documented test resource set."""

    APPROVED = {
        "كِتَابُ",
        "وَكِتَابُ",
        "وَكِتَابُهُ",
    }

    def evaluate(self, source: str) -> VocalizationDecision:
        if source in self.APPROVED:
            return VocalizationDecision.VALID
        return VocalizationDecision.INVALID


configuration = AnalyzerConfiguration(
    lexicon_path=Path("tests/golden/resources/lexicon.yaml"),
    morphotactics_path=Path(
        "tests/golden/resources/morphotactics.yaml"
    ),
    lexicon_version="1.0",
    grammar_version="1.0",
)

analyzer = build_default_analyzer(
    configuration,
    vocalization_policy=ExampleVocalizationPolicy(),
)

bundle = analyzer.analyze("وَكِتَابُهُ")

for analysis in bundle.analyses:
    print([segment.surface for segment in analysis.segments])
    print(dict(analysis.unified_features))
    print(analysis.reconstruct() == bundle.source)

Expected segmentation:

['وَ', 'كِتَابُ', 'هُ']

The example policy is intentionally limited to the test fixtures. Production or research use requires a policy based on an approved linguistic specification.

Command-Line Interface

Display help and version information:

arabseg --help
arabseg version

Validate input structurally:

arabseg validate "كِتَابُ" --profile structural

Analyze a word using explicit resources and an importable policy:

arabseg analyze "وَكِتَابُهُ" \
  --lexicon tests/golden/resources/lexicon.yaml \
  --morphotactics tests/golden/resources/morphotactics.yaml \
  --lexicon-version 1.0 \
  --grammar-version 1.0 \
  --policy my_project.policies:ApprovedVocalizationPolicy \
  --json

Validate resource integrity:

arabseg resources \
  --lexicon tests/golden/resources/lexicon.yaml \
  --morphotactics tests/golden/resources/morphotactics.yaml \
  --lexicon-version 1.0 \
  --grammar-version 1.0 \
  --validate

Check configuration, resources, policy loading, and analyzer construction:

arabseg doctor \
  --lexicon tests/golden/resources/lexicon.yaml \
  --morphotactics tests/golden/resources/morphotactics.yaml \
  --lexicon-version 1.0 \
  --grammar-version 1.0 \
  --policy my_project.policies:ApprovedVocalizationPolicy

See the CLI Guide for command behavior and exit codes.

Processing Pipeline

Input word
    ↓
Unicode and repertoire validation
    ↓
Grapheme parsing
    ↓
Orthographic and vocalization validation
    ↓
Deterministic boundary generation
    ↓
Lexical candidate lookup
    ↓
Morphotactic compatibility
    ↓
Feature unification
    ↓
Ambiguity grouping
    ↓
Immutable analysis bundle

No stage silently rewrites the source. Every accepted analysis must reconstruct the original input exactly.

Documentation

Document	Purpose
Project Overview	Goals, scope, and design principles
Installation	Environment and installation instructions
Quick Start	First-use workflow
Python API	Public classes, models, and functions
CLI Guide	Commands, arguments, JSON, and exit codes
Output Interpretation	Analyses, failures, ambiguity, and reconstruction
Architecture	Components and processing flow
Resource Format	Lexicon and morphotactics YAML schemas
Extending ArabSeg	Policies and replaceable pipeline components
FAQ	Common usage questions
Limitations	Current boundaries and non-goals
Developer Guide	Testing, quality checks, packaging, and releases

The repository also retains milestone-level implementation notes and formal specification documents under docs/.

Repository Structure

ArabSeg/
├── src/arabseg/          # Package implementation
├── tests/                # Unit, integration, regression, and golden tests
├── docs/                 # User, architecture, and specification documents
├── knowledge/            # Earlier knowledge-resource material
├── pyproject.toml        # Packaging and tool configuration
├── CHANGELOG.md          # Release history
├── MANIFEST.in           # Source-distribution inclusions
└── LICENSE               # License file

Development

Install development dependencies:

python -m pip install -e ".[dev]"

Run the quality checks:

ruff check src tests
ruff format --check src tests
mypy src
pytest

Build distributions:

python -m build
python -m twine check dist/*

See the Developer Guide before preparing a release.

Project Status

ArabSeg is under active development. Version 0.19.0 includes the complete deterministic pipeline, explicit resource loading and validation, the command-line interface, caching support, packaging configuration, and end-to-end golden regression tests.

The project remains alpha because the public API, linguistic coverage, and full-vocalization profile are still being refined before 1.0.0.

Citation

A formal citation will be added when an associated publication or archived software release is available.

Until then, cite the repository name, version, author, and access date according to the citation style required by your work.

License

ArabSeg is licensed under the Apache License 2.0. See LICENSE for details.