Sieve Referee

English | 日本語

"Should we really use probabilistic AI models to evaluate text generated by other probabilistic AI models?" A fully local, zero-dependency, and deterministic (100% reproducible) text structure and plagiarism screening engine. Developed as an extended application of Sieve-Core.

---

💡 Concept & Philosophy

Sieve-Core is the core engine that applies the design philosophy of "Sieve-AI"—a "sieve of set theory" inspired by the Sieve of Eratosthenes—to the more general domain of text and data processing. Sieve Referee, designed by adding a "sieve of combinatorics" to this Sieve-Core, is a zero-dependency Python library that extracts reliable information from noise based on corroboration from multiple independent sources.

When evaluating text that may have been generated by probabilistic models (such as LLMs), relying on another probabilistic model to calculate a similarity score makes it difficult to mathematically trace why that score was produced.

Sieve Referee was built to address this problem: to ensure every judgment is mathematically traceable, 100% reproducible, and fully explainable.

• Deterministic Approach: Evaluation is based on a 16-state Look-Up Table (LUT with 15 classified states). Identical inputs always produce identical outputs no matter how many times it is run.
• Explainable by Design: Every result carries a reason string and a 4-bit hypothesis mask, so the basis of a judgment can always be inspected.
• Careful Classification to Avoid False Accusations: A shared template or boilerplate format is classified separately as COINCIDENTAL_FORMAT_MATCH (1100 / YELLOW), rather than being treated the same as an actual content match.
• Indirect ("Telephone Game") Chain Safety: A pair with no direct similarity evidence (e.g. A <-> C, where A <-> B and B <-> C are both flagged) is never silently upgraded to a positive match solely based on cluster membership. Such pairs are reported separately on the dashboard as an "indirect chain (telephone game) warning," and the direct comparison result itself is left unmodified.

---

🎯 Primary Use Cases

Built with zero external dependencies and in-memory execution, sieve_referee is well suited as an embedded, privacy-focused screening engine.

1. Education & Learning Management Systems (EdTech / LMS)

Screen student submissions without uploading personal data or copyrighted text to a third-party cloud service. Common assignment templates are classified as YELLOW; unattributed copying or paraphrasing is flagged as RED.

2. Digital Publishing & CMS

Detect rewritten or recycled content submitted by freelance writers at draft-save time, without incurring LLM API costs for every check.

3. LLM Training Data Deduplication

Use as a deterministic pre-processing stage to remove near-duplicate or paraphrased records from a fine-tuning dataset, without GPU acceleration.

4. Legal, Contract & RFP Review

Separate boilerplate clauses (COINCIDENTAL_FORMAT_MATCH) from intentionally altered wording across contract revisions or vendor proposals.

5. CI/CD Documentation Auditing

Run in a GitHub Actions workflow or pre-commit hook to catch unauthorized copy-pasting across internal wikis or specification documents.

---

⚡ Key Technical Advantages

1. Near-zero cold start — no PyTorch, no Transformers, no model loading; well suited to AWS Lambda or other serverless environments.
2. Zero per-call cost — everything runs on local CPU.
3. Zero maintenance burden — zero third-party dependencies means no exposure to upstream CVEs or breaking API changes.

---

📦 Installation

pip install git+https://github.com/neguseatama/sieve-referee.git

or, from a local clone:

git clone https://github.com/neguseatama/sieve-referee.git
cd sieve-referee
pip install .

---

🐍 Usage (Python Library)

from sieve_referee import batch_evaluate, EvaluationResult

# 1. Prepare input documents in memory
documents = {
    "doc_a.txt": "Quantum computing relies on superposition and entanglement to achieve fast factorization.",
    "doc_b.txt": "To achieve fast factorization, quantum computing makes use of entanglement and superposition."
}

# 2. Run the screening
results: list[EvaluationResult] = batch_evaluate(documents)

# 3. Read the results
for res in results:
    print(f"Pair:    {res.item_id} <-> {res.matched_peer_id}")
    print(f"Signal:  {res.evaluation_signal.value}")  # -> RED
    print(f"Mask:    {res.mask}")                      # -> 0010
    print(f"Pattern: {res.pattern_name}")               # -> UNFORMATTED_CONTENT_COPY
    print(f"Reason:  {res.reason}")
    print("-" * 40)

EvaluationResult fields

Property	Type	Description
item_id	str	Primary document identifier
matched_peer_id	str	Comparison peer document identifier
evaluation_signal	SignalColor	Result signal (.value returns "GREEN", "YELLOW", or "RED")
mask	str	4-bit hypothesis mask, e.g. "1010"
pattern_name	str	LUT pattern name, e.g. "PARAPHRASE_DETECTED"
reason	str	Human-readable explanation of the judgment

---

💻 Usage (CLI Tool)

# Basic directory scan (writes reports/sieve_dashboard.html)
sieve-referee ./target_documents

# Single self-contained HTML report (no diff_htmls/ subfolder)
sieve-referee ./target_documents --standalone --output-dir ./reports

# JSON output for automated pipelines, with a non-zero exit code on RED
sieve-referee ./target_documents --json --fail-on-risk

# Multi-core evaluation
sieve-referee ./target_documents -j 4

Option	Type	Description
target_dir	positional	Directory containing the .txt files to screen
--output-dir	string	Report output directory (default: reports)
--standalone	flag	Emit a single self-contained HTML report instead of a report + diff_htmls/ folder
--json	flag	Print results as JSON to stdout
--json-file	string	Save results as JSON to the given path
--fail-on-risk	flag	Exit with status code 1 if any RED signal is found
-j, --jobs	int	Number of worker processes for evaluation (default: CPU count)

---

📊 16-State Look-Up Table (LUT)

Four independent hypotheses (H1–H4) are combined into a 4-bit mask, which maps deterministically to a signal, pattern name, and reason.

Mask	Signal	Pattern Name	Description
1000	🟢 GREEN	STANDARD_STRUCTURE_UNIQUE	Independent document, no structural or content overlap detected.
1100	🟡 YELLOW	COINCIDENTAL_FORMAT_MATCH	Shared format/template only; content itself is independent.
1010	🔴 RED	PARAPHRASE_DETECTED	Content overlap despite a different structure (paraphrasing).
1011	🔴 RED	CLUSTER_PARAPHRASE_GROUP	Paraphrased reuse confirmed within a 3+ document cluster.
1111	🔴 RED	CLUSTER_EXACT_COPY	Exact/templated copy confirmed within a 3+ document cluster.
0111	🔴 RED	CLUSTER_FORMATTED_PARAPHRASE	Paraphrased reuse in shared structure recurring within a 3+ document cluster.
0101	🔴 YELLOW	CLUSTER_FORMAT_ONLY_MATCH	Shared template structure within a 3+ document cluster; content independent.
0011	🔴 RED	CLUSTER_UNFORMATTED_CONTENT_COPY	Unformatted content copy recurring within a 3+ document cluster.

(The LUT classifies 15 of the 16 possible mask states. Exactly two states, 0001 and 1001, where cluster membership would stand without any direct similarity evidence, are structurally unreachable; 1001 is defined in the table but is not producible by the current pipeline, and 0001 is the single state without a table entry. Three further states, 0011, 0101, and 0111, arise via cluster promotion and are classified by their dedicated LUT entries.)

Note on H4 (cluster membership) and indirect chains H4 is only set when a pair has both direct similarity evidence (H2 or H3) and cluster membership of size ≥ 3. A pair with no direct evidence is never upgraded through cluster membership alone — such "telephone game" chains are reported separately as an indirect-chain warning on the dashboard, and the direct pair's own result is left unmodified.

---

⚠️ Known Limitation: Threshold Calibration Is Japanese-Specific

All four thresholds (H1 = 0.35, H2 = 0.80, H3 = 0.45, cluster size ≥ 3) were calibrated using real-data testing on Japanese text (roughly 1,300+ pairs across unrelated documents, paraphrase pairs, and formal academic prose).

The H1 density metric in particular (len(set(text)) / len(text)) behaves very differently across scripts: Japanese text draws from a large set of kanji, so unique-character ratios tend to be high, while English and other alphabetic-script text reuses a small ~26-letter alphabet and will systematically score lower on this metric, independent of how generic or templated the content actually is. As a result, non-Japanese text may be under- or over-classified against the current defaults.

Multi-encoding file reading (UTF-8, Shift_JIS, CP932, EUC-JP) and NFKC normalization work for any text regardless of language. What has not been separately validated is the semantic quality of H2/H3 splitting, or the appropriateness of the default thresholds, for languages other than Japanese. If you use this on non-Japanese corpora, re-run the false-positive check described in ARCHITECTURE.md against your own data before trusting the default thresholds.

Additionally, the reason string returned by every EvaluationResult is currently hard-coded in Japanese regardless of the input document's language, as shown in the example output above. There is no English (or other-language) localization of these messages yet.

---

📚 Documentation

• 🔌 API & Integration Guide — using sieve_referee as a Python library or FastAPI microservice
• 🚀 Deployment Guide — Docker and Terraform for Google Cloud Run
• 🔐 CI/CD & Security Guide — keyless GCP deployment via Workload Identity Federation
• 🏛️ Architecture Specification — the mathematical model and safety invariants in detail

📄 License

MIT License. See LICENSE.