Overview

OpenEtruscan is an open-source platform for working with the Etruscan epigraphic record. It normalises transcriptions across notation systems (including structured Leiden-convention parsing of restorations, gaps, and unclear readings), classifies inscriptions into epigraphic types (macro F1 in the 0.31–0.37 band on a held-out set, with overlapping CIs across architectures from ~3K to 274K parameters — no model separates from the field; TF-IDF + Naïve Bayes is the production reference), and publishes the full corpus as Linked Open Data.

The corpus holds 6,633 unified inscriptions, drawn mostly from the Larth Dataset (Vico & Spanakis, 2023; ~71%) and the Corpus Inscriptionum Etruscarum (Vol. I extractions; ~29%), with links to Trismegistos, EAGLE, and Pleiades. The cleaned, ML-ready dataset published on Zenodo is a 6,567-row subset (66 rows dropped during cleaning).

Which number is which. Three corpus totals circulate and they describe three different artifacts: 6,633 archival (this repository's unified corpus), 6,567 published (the Zenodo deposit, 66 rows dropped in cleaning), and 5,932 deployed (every row in the live production database).

The deployed corpus is smaller than the published one, and we do not yet know why. The Linked Open Data feed is unfiltered — getPelagiosFeedRows() is a bare SELECT ... FROM inscriptions — so 5,932 is the live table count, not a subset of a larger served corpus. 701 rows in the archival corpus and 635 in the Zenodo deposit are not in production. Candidates are an older or partial seed, loss during the Cloud SQL → Neon migration, or cleaning applied at load; resolving it needs database access and is tracked as an open corpus item. Until it is resolved, do not read 6,633 or 6,567 as the size of what the website serves.

All of this is reconciled in release-manifest.json, the single source of truth for every version, count, licence, DOI, and model status this project asserts publicly; scripts/ops/check_release_truth.py fails CI when a surface drifts from it. If a number here and a number there disagree, the manifest wins and the other is a bug.

Provenance disclosure

OpenEtruscan separates editorial verification of a text (we trust the published reading) from archaeological provenance (we know where the inscribed object actually surfaced). These are two different scholarly claims and each row carries a provenance_status in one of four tiers.

The table below describes the archival corpus (6,633 rows). The live database, being smaller, reports different absolute counts — 2,203 documented / 3,729 undocumented as of 2026-08-08 — at a similar ratio; /stats/provenance returns the live breakdown and is authoritative for anything the website shows.

The /search endpoint accepts ?has_provenance=true to restrict to the first two tiers, and the website's search UI defaults to that filter. The /stats/provenance endpoint returns the live breakdown.

The "184 archaeological sites" referenced in earlier copy is the count of distinct findspot strings across the 34.9% with documented provenance — not across the whole corpus.

The platform follows a decoupled, cloud-native architecture (as of 2026-05-24):

• Data Layer — PostgreSQL (PostGIS + pgvector) on Neon serverless (was Cloud SQL, migrated). 3,072-dimensional text-embedding-004 embeddings for semantic similarity search.
• Public HTTP API — Vercel Functions (TypeScript + Drizzle ORM + Neon serverless driver) co-located in the openetruscan-frontend repo under app/api/*. Single-origin, no cross-cloud hop. See https://www.openetruscan.com/api/{search,inscription/[id],stats/summary,concordance,clan/[gens],radius,search/geo,names/network,anchors/…}.
• Web app — Next.js 16 on Vercel, with the mobile path shipping as RSC + useSyncExternalStore-gated dynamic-import islands (Lighthouse a11y 100, perf 92 mobile / 99 desktop).
• Python openetruscan package (this repo) — CLI + research-pipeline source of truth. pip install openetruscan ships the 14-command CLI (normalize, classify, train-neural, export-corpus, epidoc, etc.) plus the src/openetruscan/api/ FastAPI surface used for parity testing and local development. The live public HTTP API no longer runs from this codebase.
• Research pipelines — Cloud Build orchestrators (cloudbuild/v2-classify-jury.yaml, v2-lacuna-jury.yaml) drove the v2.0.x LLM-jury annotation work. The former Vertex AI billing project is now deleted; re-running these requires pointing them at a live project.

API

A REST endpoint is available for programmatic normalisation:

curl -X POST https://www.openetruscan.com/api/normalize \
  -H "Content-Type: application/json" \
  -d '{"text": "MI AVILES"}'

Response:

{
  "canonical": "mi aviles",
  "phonetic": "/mi.aviles/",
  "old_italic": "\ud800\udf0c\ud800\udf09 \ud800\udf00\ud800\udf05\ud800\udf09\ud800\udf0b\ud800\udf04\ud800\udf14",
  "source_system": "cie",
  "tokens": ["mi", "aviles"]
}

Other core endpoints:

• GET /stats/timeline — Aggregated temporal distributions across the corpus.
• GET /clan/{gens} — Prosopographical network of co-occurring personal names for a single Etruscan family name.
• GET /concordance — Keyword-in-Context (KWIC) search across transcriptions.

Python Package & CLI

pip install openetruscan          # core (CLI + library)
pip install 'openetruscan[server]' # FastAPI server runtime
pip install 'openetruscan[neural]' # neural classifiers (torch + onnxscript)
pip install 'openetruscan[all]'    # full stack incl. transformers + sotac

Library

from openetruscan import normalize

result = normalize("LARTHAL")
print(result.canonical)  # larθal
print(result.phonetic)  # /lar.tʰal/
print(result.old_italic)  # 𐌓𐌀𐌓𐌈𐌀𐌋

CLI

The openetruscan console script wraps the library and the corpus operations. Run openetruscan --help for the full menu; subcommands:

All commands accept --language (default etruscan) and respect the language adapter registry (list-adapters). The classification commands report bootstrap-CI'd metrics; see research/v2/ for the evaluation protocol.

Repository Structure

openEtruscan/
  src/openetruscan/  Python package: core library (normalizer, Leiden parser,
                     EpiDoc, corpus, prosopography), ml/, db/ (Alembic), api/
                     (local FastAPI parity reference)
  research/          Research narrative: v2 annotation protocol + frozen
                     benchmarks, findings, experiments, parked strands
  eval/              Frozen eval outputs + the rosetta/search harnesses
  scripts/           Data pipeline, ML, ops, training scripts (see scripts/README.md;
                     spent one-offs live in scripts/attic/)
  services/          Cloud Run inference services (ByT5 restorer, reranker)
  tests/             Pytest suite (incl. migration-chain and science-harness
                     invariant tests)
  data/              Local data artifacts (git-ignored; fetch via
                     scripts/ops/fetch_data.py — see data/README.md)

The web application (Next.js 16) and the production TypeScript API live in the separate openEtruscan-frontend repository.

Linked Open Data & Pelagios Network

OpenEtruscan exports Linked Open Data in formats interoperable with the wider ancient-world DH graph:

• The Pelagios-compatible JSON-LD endpoint serves the corpus as a Web Annotation collection — 5,932 inscriptions by the feed's own total count (Pelagios Network format spec; not a formal membership claim).
• Findspots aligned to Pleiades for the subset with documented provenance (see §Provenance disclosure above).

Classification & restoration models

This project ships two small models alongside an LLM-jury annotation pipeline. The numbers below are from research/v2/ — frozen test splits, multi-rater consensus eval, bootstrap-CI'd metrics, full pre-registration in research/v2/PRE_REGISTRATION.md.

Classifier (7-class inscription type) — v2.0.2 head-to-head

Four architectures spanning two orders of magnitude in parameter count were evaluated on the v2.0.2 candidate-gold (n=143, 3-rater LLM-jury unanimous: Sonnet 4.6 + Gemini 2.5 Pro + Llama 4 Maverick, Krippendorff α=0.7649). Train pool: 282 silver-labelled rows. "Candidate-gold" is LLM-consensus silver, not gold: the two-philologist ratification step (target human α ≥ 0.80, research/v2/handoff/) has not yet been performed, and these numbers measure agreement with a frontier-model consensus, not with expert epigraphic judgment. Cite them with that caveat.

Two findings:

1. Architecture-invariance among local-feature models. TF-IDF+NB, CharCNN, and MicroTransformer cluster at 0.31–0.37 macro F1 with overlapping bootstrap CIs despite 100× parameter-count range. Adding parameters does not move macro F1 — the bottleneck is data, not architecture.
2. Out-of-distribution dense embeddings fail. EmbeddingMLP using a multilingual MiniLM encoder lands at 0.124, with a CI that does not overlap with TF-IDF+NB's — significant degradation. A frozen modern-multilingual encoder discards the surface-morphological features (mi…al/-as possessives, tular spural boundary formula, suffixal markers) that carry the typological signal. This contradicts the conventional NLP intuition that dense embeddings beat char-ngrams; for under-resourced ancient corpora with strong domain-specific morphology, the reverse holds.

The dominant funerary and ownership classes are well-modelled (per-class F1 0.84 and 0.79 on TF-IDF+NB); rare classes (boundary, legal, votive, commercial) remain data-starved. The 0.31–0.37 macro band reflects this imbalance honestly — earlier copy in this repository claimed "99% macro F1", which referred to in-training-set performance on a self-labeled subset and is retracted.

v2.0.1 (n=159, 2-rater jury without Sonnet) is preserved in GCS as a looser consensus-silver reference; v2.0.2 supersedes it per the closure of Deviation §A in research/v2/PRE_REGISTRATION.md.

Lacuna restoration

⚠️ RETRACTED (v2.0.3, 2026-07-04): the v2.0.2 lacuna table and "Finding C" below were a harness artifact. The v2.0.2 jury scored empty API responses as hallucinations: 114 of 125 Claude Sonnet 4.6 rows were empty completions (max_tokens=1024 exhausted while echoing restored_full), and lacuna_jury.py counted every empty response as hallucinated=True. The 0.949 rate measured a Vertex integration failure, not model behaviour — on the 11 rows Sonnet actually answered it led the field. The set was also inflated by exact duplicates (125 rows → 70 unique tasks). Both bugs are fixed (no_parse handling; max_tokens=4096 + non-empty retry). The corrected re-run is below.

Per-restoration evaluation on the deduplicated 66 clean-gold tasks (Leiden [abc]-style, unknown-continuation markers excluded; width-1-dominated, 43/66). v2.0.3 — 3-rater jury: Claude Opus 4.8 (direct agentic rater¹) + Gemini 3.1 Pro + Gemini 3.5 Flash, 10 000-resample bootstrap, seed=42:

¹ Opus is not enabled on the Vertex projects available here (only Haiku 4.5), so it ran as a direct first-party rater, blind to gold, scored after — a documented deviation. ² Opus's restored_full was assembled mechanically, so its 0.000 hallucination is by construction and not comparable to the free-generating Gemini raters.

Hallucination = the model emits at least one character outside the marked lacuna span. Earlier copy claiming "Phil. Safety: High (Sentinels)" was a vibes-based label without a metric and remains retracted.

Corrected findings (v2.0.3):

• No model wins on accuracy. All span-exact differences are non-significant (paired bootstrap: Opus vs 3.1-Pro Δ+0.049 p=0.24; Opus vs 3.5-Flash Δ+0.031 p=0.37; 3.1-Pro vs 3.5-Flash p=0.66). The task is difficulty/data-bound — the same "data, not architecture" story as the classifier.
• The real differentiator is hallucination: Gemini 3.5 Flash alters context outside the span on 54.5% of rows vs 3.1-Pro's 16.1% — the small/fast model corrupts context.
• Independence caveat: the two Gemini raters agree with each other (0.339) far more than with Opus (0.18–0.24), so a Krippendorff α over this 2×Google panel is inflated by shared lineage.

Methodology

Full annotation codebook, frozen stratified splits (SHA256-pinned, text-bearing — see research/v2/data/), LLM-jury runners, raw jury evidence + computed metrics (research/v2/results/lacuna/), and the bootstrap-CI eval harness live in research/v2/. The dataset cards and pre-registration are the citation-grade artifacts; this README is a summary. Reproduction steps: docs/REPRODUCE.md.

Development

Python package (this repo):

git clone https://github.com/Eddy1919/openEtruscan.git
cd openEtruscan
pip install -e ".[dev]"      # or: uv sync --extra dev (uv.lock is committed)
pytest

Local corpus + API: see docs/REPRODUCE.md (scripts/ops/fetch_data.py fetches the corpus from the Zenodo DOI; docker-compose.dev.yml stands up Postgres+pgvector and the API).

Web app: clone the separate openEtruscan-frontend repo, then npm install && npm run dev.

Deployment

Development & Preview (from the frontend repo) Push local dev environment variables and deploy a Preview build:

# Pull Vercel preview/dev environment variables
npx vercel env pull .env.local

# Deploy to Vercel Preview Environment
npx vercel -y

Production Push directly to the live production site:

npx vercel --prod -y

Citing this work

If you use OpenEtruscan or the OpenEtruscan corpus in your research, please cite both this software repository and the dataset deposit. Machine-readable citation metadata lives in:

• CITATION.cff — GitHub's Cite this repository button reads from here.
• codemeta.json — schema.org-compatible (used by Zenodo, ORCID, OpenAlex).
• .zenodo.json — controls the auto-deposit on each tagged GitHub release.

A minimal BibTeX entry:

@software{openetruscan_2026,
  author    = {Panichi, Edoardo},
  title     = {{OpenEtruscan: open-source digital corpus platform for Etruscan epigraphy}},
  year      = {2026},
  version   = {1.0.0},
  doi       = {10.5281/zenodo.20075836},
  url       = {https://doi.org/10.5281/zenodo.20075836},
  publisher = {Zenodo}
}

Per the Zenodo record metadata, 10.5281/zenodo.20075835 is the concept DOI (resolves to the latest version) and 10.5281/zenodo.20075836 is the version DOI for the v1.0.0 deposit. Cite the concept DOI when referencing the project, the version DOI when referencing a specific snapshot. (An earlier revision of this section had the two swapped.)

The frozen reference benchmark is rosetta-eval-v1; full reproduction instructions live in research/notes/reproduce-rosetta-eval-v1.md.

Licence

• Code: MIT
• Data: CC BY 4.0 (the licence declared by the Zenodo deposit and the live void.ttl; the Larth-derived fields are CC-BY-4.0 upstream, so attribution cannot be waived)
• Models: Apache 2.0

Acknowledgements

• Compilers of the Corpus Inscriptionum Etruscarum
• The Etruscan Texts Project (UMass Amherst)
• The Larth Dataset (Vico and Spanakis, 2023)
• The EpiDoc community
• The Classical Language Toolkit

What's new

v1.1.0 (2026-07-17) — integrity & reproducibility release

The frozen classification split was found corrupt (99 text-less rows vs the pre-registered 400) and has been regenerated and verified against the jury's own adjudication-queue IDs; the lacuna evidence (raw jury outputs + metrics) moved from local-only staging into the tracked tree at research/v2/results/lacuna/ with SHA256 manifests; the normalizer now parses Leiden editorial markup into a structured apparatus (and EpiDoc export emits real <supplied>/<ex>/<gap>/<unclear>); the Alembic chain bootstraps an empty database (it previously could not); CI gained mypy, a coverage floor, and a pgvector-backed test service. Full detail in CHANGELOG.md §1.1.0 and PRE_REGISTRATION.md Deviation §C.

Architecture shift (2026-05-24) — research-first repo

The public HTTP API moved out of this repo. The live www.openetruscan.com/api/* surface is now TypeScript route handlers in the openetruscan-frontend Vercel project, talking to Neon serverless Postgres via Drizzle ORM + @neondatabase/serverless. Cloud SQL stopped, GCE VM terminated.

What stays here (and gets first billing in this README):

• research/v2/ — the 3-rater LLM-jury annotation pipeline, pre-registration, codebooks, and frozen benchmarks. Source of truth for the v2.0.2 evaluation work below.
• The openetruscan CLI on PyPI — pip install openetruscan ships 14 commands for normalisation, classification, EpiDoc export, batch processing, neural training/inference.
• src/openetruscan/api/ — the legacy FastAPI server stays in-tree as a parity reference + local-dev convenience (uvicorn openetruscan.api.server:app). It is no longer the production HTTP surface.
• Cloud Build research pipelines — cloudbuild/v2-classify-jury.yaml, v2-lacuna-jury.yaml, v2-train-neural.yaml were used for the v2 evaluation work. They are not currently running: the former Vertex billing project is deleted, so a re-run needs a live project.

v2.0.2 annotation & evaluation pipeline (shipped 2026-05-24)

research/v2/ is the gold-annotation and frozen-benchmark infrastructure that this project's earlier metric claims lacked. As of v2.0.2 both the classifier and lacuna streams are evaluated under a full 3-rater LLM jury (Claude Sonnet 4.6 + Gemini 2.5 Pro + Llama 4 Maverick on Vertex AI); the philologist α≥0.80 spot-check on the adjudication queue remains the final ratification step before Hugging Face publication.

• Frozen, stratified test split (seed=42, 400 rows, 7 classes with a class-2 floor) — see research/v2/pipelines/classify_split.py.
• 3-rater LLM jury (Claude Sonnet 4.6 + Gemini 2.5 Pro + Llama 4 Maverick on Vertex AI; Sonnet substituted for Opus per Deviation §A) produces independent labels; Krippendorff α and a unanimous-agreement filter promote rows to candidate-gold. Classifier α = 0.7649 on the full pool; n=143 candidate-gold rows.
• Pre-registered eval with bootstrap 95% CIs on every metric and paired-bootstrap p-values on every model-comparison claim — see research/v2/PRE_REGISTRATION.md and research/v2/eval/bootstrap.py.
• Honest retraction of the earlier "99% Macro F1" headline — the real number on a stricter eval is 0.313 ± 0.038 (TF-IDF + NB on n=143).
• Finding C (v2.0.2) — RETRACTED at v2.0.3. The "Sonnet hallucinates 94.9%, frontier model loses" claim was a harness artifact: empty Vertex completions scored as hallucinations (see §Lacuna restoration above). The corrected v2.0.3 re-run (Opus 4.8 + Gemini 3.1 Pro + Gemini 3.5 Flash) finds no significant model difference on accuracy (span-exact 0.29 / 0.26 / 0.26, all p>0.2); the only real gap is hallucination (Gemini 3.5 Flash 0.545 vs Gemini 3.1 Pro 0.161). See docs/INTELLIGENCE_V2.md.

v0.5.0 infrastructure

• Cloud Run neural restoration: ByT5 lacunae restoration is served from a dedicated Cloud Run inference service (services/byt5-restorer/).
• Search-eval harness with gate flags: hybrid-search NDCG@10 computed by eval/harness/run_search_eval.py --gate … against real DB queries (eval/harness/search_eval_queries.jsonl). Run manually against a populated database — it is not wired into CI (an earlier version of this line claimed a "CI/CD eval gate"; no workflow invokes it, and CI has no populated corpus to run it against).
• Admin curatorial UI: provenance promotion workflow in the Inscription viewer (ProvenancePromoteModal.tsx).