English | 日本語

Sieve Lens v0.15.0

💡 Concept

Modern AI evaluation systems parse documents as text. Humans read documents as rendered output. These two views are not the same, and the gap between them can be exploited:

• Zero-width characters (U+200B...) that a model reads but a human cannot see.
• CSS rules (display:none) that hide text from a human but not from a parser.
• Out-of-band channels (comments, metadata, headers, footnotes) that a parser reads but a human never notices.

Sieve Lens observes this divergence.
It does not judge intent. It surfaces what is invisible, with location and decoded content, so that a human can decide what to do.

The name "Sieve Lens" comes from the design philosophy of seeing through a document the way a lens reveals what the naked eye cannot.

🧩 The Sieve Series

Sieve is a family of deterministic, zero-dependency engines that share the same design principles (deterministic output, no telemetry, fully local) — each engine observes or masks one specific risk, and they compose:

Observation → masking in one pass: Lens finds what a human can't see, Redact removes it with a receipt that proves nothing else changed.

📐 7-bit Observation Space

Mask format: H1H2H3H4-H5H6H7 (e.g., 1000-100).

📦 Supported Formats

Not supported: audio, video.

🔥 Key Features

• 7-bit observation space (H1–H7)
  Independent, deterministic detection of each class of invisible content.
• Zero dependencies
  Python standard library only. Runs offline, no telemetry, no network.
• Deterministic
  Same input always yields the same observation, on any platform.
• Explainable
  Every observation carries location and decoded content, so a human can verify.
• Observation, not judgment
  The engine reports evidence. It never declares "this is an attack".
• Self-contained HTML report (since v0.1)
• PDF support via pypdf (since v0.2)
• Image OCR support via Pillow + pytesseract (since v0.3)
  OCR language is configurable: install(engine, ocr_lang="eng+jpn")
• Interactive HTML dashboard (since v0.5)
  Batch-analyze a directory and produce a single self-contained dashboard with summary cards, mask distribution, hypothesis activation charts, and a sortable file table. Requires Jinja2 + Plotly.
• Printable PDF report (since v0.6)
  Generate an A4 PDF report from a directory of documents. Includes summary, hypothesis table, mask distribution, and a color-coded file list. Requires weasyprint (and optionally kaleido for charts).
• Image layer analysis (since v0.7)
  Detects pixel-level concealment in images: transparent RGBA layers carrying non-zero RGB values, and low-contrast text with a WCAG ratio below 2:1. Requires Pillow.
• Full CSS selector resolution (since v0.8)
  Extends the HTML extractor with full CSS selector matching via cssselect2: descendant, child, sibling, attribute, and pseudo-class selectors. Hidden rules are only reported when they match an element.
• Report internationalization (i18n) (since v0.9)
  Dashboard and PDF reports are available in English and Japanese. Auto-detection selects the language based on input document content.
• PDF embedded image analysis (since v0.10)
  Extracts images embedded in PDF pages and applies OCR, transparent layer detection, low-contrast detection, and metadata analysis to each image. Requires pypdf + Pillow.

• PDF Form XObject text scanning (since v0.15)
  Text inside Form XObject content streams, including invisible (Tr 3) text, is scanned with the same semantics as page content streams; nested forms are walked with cycle protection (depth cap of 8).

📦 Installation

# Core (zero-dependency)
pip install git+https://github.com/neguseatama/sieve-lens.git

# From PyPI (for released versions)
pip install "sieve-lens[pdf]"
pip install "sieve-lens[ocr]"

# Or the latest from GitHub (may include changes not yet on PyPI)
pip install "sieve-lens[pdf] @ git+https://github.com/neguseatama/sieve-lens.git"
pip install "sieve-lens[ocr] @ git+https://github.com/neguseatama/sieve-lens.git"

💻 Quick Start

from sieve_lens import SieveLensEngine, format_report, format_report_html

engine = SieveLensEngine()
obs = engine.observe("resume.docx")

print(obs.mask)                    # e.g. "1000-100"
print(obs.h_states)                # {'H1': 1, 'H2': 0, ..., 'H7': 0}
print(format_report(obs))          # human-readable text report

html = format_report_html(obs)     # self-contained HTML report

To write the HTML report to a file:

from sieve_lens import write_report_html
write_report_html(obs, "report.html")

PDF (requires the pdf extra)

from sieve_lens import SieveLensEngine
from sieve_lens_ext.pdf import install

engine = SieveLensEngine()
install(engine)                    # attach .pdf support
obs = engine.observe("resume.pdf")
print(obs.mask)

Image OCR (requires the ocr extra)

from sieve_lens import SieveLensEngine
from sieve_lens_ext.ocr import install

engine = SieveLensEngine()
install(engine, ocr_lang="eng+jpn")    # tesseract language packs
obs = engine.observe("screenshot.png")
print(obs.mask)

🔬 Testing

# v0 test suite (26 tests)
python -m unittest discover -s tests -p "test_sieve_lens.py" -v

# v0.1 test suite (14 tests)
python -m unittest discover -s tests -p "test_sieve_lens_v0_1.py" -v

# v0.2 PDF extension (7 tests, requires pypdf)
python -m unittest discover -s tests -p "test_sieve_lens_pdf.py" -v

🔗 Redact Bridge (v0.12.0)

sieve_lens_ext.redact_bridge exports an observation catalog that Sieve Redact consumes directly:

python -m sieve_lens_ext.redact_bridge input.txt catalog.json
python -m sieve_redact input.txt -o clean.txt --from-lens catalog.json

• Catalog format (version 1): observations[] with kind, codepoint, count, positions (first-seen order).
• Look-alike characters (Cyrillic/Greek -> Latin, full-width digits and letters -> ASCII, 102 entries) carry an optional twin field, which Redact converts into a replacement rule; invisible characters become deletions.
• Deterministic, stdlib-only. The bridge imports the core character tables, so the two engines can never disagree on what counts as invisible.
• End-to-end check: python3 probe_redact_roundtrip.py (synthetic input -> catalog -> redact -> re-observe clean).

v0.11.0 Highlights

• Fixed critical regressions: PDF embedded-image analysis and HTML body text extraction (which had silently disabled H2/H3/H6/H7 detection), and EXIF UserComment/SubIFD reading.
• Security: report output is now HTML-escaped (untrusted specimens can no longer inject markup into dashboards/PDF reports), and local stylesheet resolution is confined to the specimen's directory tree.
• Deterministic PDF reports: byte-identical output across builds, verified in CI (fixed dcterms dates + pinned XMP DocumentID).
• Chained extensions: sieve_lens_ext.image_layers now composes with sieve_lens_ext.ocr automatically; new SieveLensEngine.get_extractor().

Invisible Prompt Observation Engine

Sieve Lens is not a tool that distrusts AI.
It is a tool that helps AI receive trustworthy input.

Zero-dependency, deterministic, explainable observation of invisible content in documents (resumes, submissions, reports) that may be intended to influence AI-based evaluation systems.

Extends the Sieve series to the domain of machine-readable vs. human-visible divergence.

⚠️ Known Limitations

1. Observation, not removal. Concealment found in PDFs and images is reported; masking is Sieve Redact's domain (deliberate role split).

2. Semantic intent is not evaluated. A visible but manipulative prompt is outside the scope of this tool.

3. Only inline CSS is inspected. External stylesheets and CSS classes are not resolved.

4. Character-table coverage is fixed. New invisible techniques require updating the tables.

5. This is an observation engine, not a detector. Findings should always be confirmed by human review.

6. Homoglyph twins are non-ASCII -> ASCII only — lookalike pairs inside ASCII (O/0, l/1) and CJK lookalikes are out of scope

7. Missing OCR language packs fail silently. If the configured language pack is not installed, OCR yields no text without an error; metadata extraction is unaffected

8. PDF Form XObject nesting is capped. Text inside Form XObjects is scanned with a cycle-safe walk; nesting deeper than 8 levels is skipped.

📄 License

MIT License. See LICENSE for details.

👤 Author

• Kai IWASAKI