aicorpusx

Resumable, multi-key concurrent translation for CSV / XLSX corpora. Any service that speaks OpenAI-compatible /chat/completions works: pass apis, model, and base_url.

import aicorpusx

aicorpusx.trans(
    "terms.xlsx",
    source_column="source",
    targets={"ar": "Arabic", "en": "English"},
    apis=["key-1", "key-2"],
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)

The default output is terms_translated.xlsx. CSV input produces CSV; XLSX input produces XLSX.

Set base_url to the API root (for example https://api.deepseek.com or https://api.openai.com/v1). The library appends /chat/completions. If you already pass a full endpoint, it is used as-is.

Install

python -m pip install aicorpusx

From a local checkout:

python -m pip install .

Examples

Languages

There is no fixed language whitelist. The keys in targets are passed to the translation model as target-language identifiers, so you can use ISO 639-1 codes (recommended) or full language names such as "Japanese". Actual language coverage depends on the selected model or provider.

Common ISO 639-1 language codes:

The value paired with each language is the output column name:

targets={
    "en": "English",
    "ja": "Japanese",
    "ar": "Arabic",
}

Use the same target identifiers in glossary column names or glossary_target_columns mappings.

Translation options

Worksheet, source language, and content mode:

aicorpusx.trans(
    "corpus.xlsx",
    sheet_name="Data",
    source_column="source",
    source_language="zh",
    targets={"ar": "Arabic"},
    mode="sentence",  # auto / term / sentence / text
    apis=["key-1", "key-2"],
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)

Pass a glossary dict:

aicorpusx.trans(
    "corpus.csv",
    source_column="source",
    targets={"fr": "French"},
    glossary={
        "artificial intelligence": "intelligence artificielle",
        "machine learning": "apprentissage automatique",
    },
    glossary_mode="strict",  # strict / prefer / off
    apis=["key-1"],
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)

A multilingual glossary file can use columns such as source,ar,en,de:

aicorpusx.trans(
    "corpus.xlsx",
    source_column="source",
    targets={"ar": "Arabic", "en": "English", "de": "German"},
    glossary="glossary.xlsx",
    apis=["key-1", "key-2"],
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)

Map irregular column names explicitly:

aicorpusx.trans(
    "corpus.xlsx",
    source_column="source",
    targets={"ar": "Arabic"},
    glossary="glossary.xlsx",
    glossary_source_column="source term",
    glossary_target_columns={"ar": "approved Arabic term"},
    apis=["key-1"],
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)

Terms are matched in-place, longest first. strict re-requests when a matched target term is missing from the result; prefer only constrains the prompt; off ignores the glossary.

Scheduling and resume

• strategy="dynamic" (default): every live API pulls from a shared queue; faster keys take more work.
• strategy="balanced": work is split evenly at start, and each API shows a fixed total. Unfinished tasks from a dead key are still handed to others.
• The default request gap is sleep=0.2 seconds.
• 429, 5xx, timeouts, and connection errors back off exponentially with jitter. 401/403 disable that key. Ordinary 400 errors are not retried forever.
• checkpoint=True (default) stores successes and failed tasks. Re-runs with overwrite=False skip cells already filled in the output or checkpoint.
• Rich progress labels workers API 1, API 2, and so on. Logs and checkpoints never store API keys.

Main resilience knobs and defaults:

aicorpusx.trans(
    "corpus.csv",
    source_column="source",
    targets={"en": "English"},
    apis=["key-1", "key-2"],
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
    max_retries=5,
    backoff_base=1,
    max_backoff=60,
    api_failure_threshold=5,
    api_cooldown=30,
    max_api_cooldown=300,
)

Non-compatible APIs

Requests use the OpenAI-compatible protocol by default. Extra fields such as temperature go in provider_options.

If the service is not /chat/completions, pass an object with translate(...) as provider.