Scority

A ship / hold / rollback gate for automated changes — one that refuses to declare a win.

You changed something. Did it help? Most tooling answers that by comparing two numbers and calling the difference an effect. This does not:

from scority_engine.modules.audit.safe_fix.measure import evaluate

# (before, after) per unit — pages, accounts, anything you can pair
result = evaluate(test_pairs, control_pairs=control_pairs)

result.decision.action       # 'expand' | 'hold' | 'rollback'
result.decision.reason       # 'CI straddles zero — effect not distinguishable from zero'
result.decision.confidence   # 'high' | 'medium' | 'low' — from the number of pairs
result.ci                    # CI(point, low, high, n, alpha) — difference-in-differences,
                             # 2000-iteration bootstrap, seeded

Four things have to be true before it will say expand: no watchdog regression, enough paired units (min_pairs, default 8), a confidence interval that does not cross zero, and a viable control cohort. Anything else is hold or rollback. There is no code path that turns "the numbers moved" into "we caused it".

measure.py and holdout.py are 210 lines and import nothing outside the standard library — no numpy, no scipy. The holdout split is sha1(salt|url), so the counterfactual is reproducible without carrying RNG state. Nothing in either file knows what SEO is.

The reference implementation: SEO and AI visibility

The gate is not a thought experiment here — it runs against a real workload. Scority audits a page or a whole site, proposes concrete fixes, applies them to a holdout cohort, and puts the result through the decision above. The SEO and answer-engine modules are the proof that the discipline survives contact with a domain where wishful measurement is the norm, and they are useful on their own if that is what you came for.

The same refusal runs through the rest of it. Contested claims of the field — site authority as a standing factor, the "sandbox", CTR as a direct ranking factor — cannot be labelled verified no matter what produced them. A rate computed from a thin sample is contested, not a fact. See The rules, and where they live in the code.

Agent-facing contracts — what an agent may be, what one unit of work may ask of it, when a cached model answer may be reused — live in scority_engine/contracts/ with their own README.

Everything below is in this repository and runs offline unless a section says otherwise. Source paths are written relative to scority_engine/modules/ — audit/safe_fix/measure.py is scority_engine/modules/audit/safe_fix/measure.py on disk.

What this is not

• Not a rank tracker or a SERP data provider. serp/ is an adapter layer. You bring provider keys (Google/DataForSEO, Yandex/XMLStock) or you use the modules that need no SERP at all.
• Not an autopilot. The open core stops at proposals, gates, and verdicts. The default policy decision for anything unmatched is REQUIRE_APPROVAL, and OUTREACH_SEND / FIX_CONFIRM can never be auto-allowed — an ALLOW rule targeting them is downgraded and the downgrade is written into the reason (policy/engine.py). Scheduling, delivery to production, and the human queue UI are not here.
• Not a traffic guarantee. There is no ROI projector and no "expected lift" estimator. predictive_ranking.py is a gate, not a predictor: it refuses to let a caller present a prediction until the evidence base clears explicit thresholds (200 observations, 60 days, 20 query/page pairs, 1000 impressions), and its declared policy string is literally hypothesis_ranker_with_ci_not_oracle. It has no callers in this repository, and that is the honest reading of it: the model it guards does not exist here, so the module is a contract for work that has not been written rather than machinery you can watch fire. Wiring it to the rule-based opportunity scorer would make it look active while gating nothing, which is worse than leaving it plainly inert.
• Not calibrated on a public gold-set. Weights and thresholds in the scoring modules are directional defaults. They are tagged # CALIBRATE in the source and versioned (THRESHOLDS_VERSION = "cg-v2", READINESS_VERSION, VV_VERSION) so a recalibration is a visible, diffable event rather than a silent one.
• Not a replacement for Search Console / Metrika. It reads them.
• Not fully offline. The technical audit, schema audit, citability, compliance scan, linking, llms.txt generation and the measurement math need no accounts and no keys. SERP, GSC, Yandex Webmaster, Metrika and AI-engine probes need credentials that are yours.

Quick start (60 seconds)

Python 3.12+.

git clone https://github.com/vitaliyino/scority-core.git
cd scority-core
uv venv && uv pip install -e ".[dev]"

The base install is nine packages. The technical audit, schema audit, citability, the compliance scan, linking, llms.txt generation and the measurement math all work with it and need no accounts. Heavier surfaces are extras — [service] (HTTP), [db] (Postgres), [llm] (embeddings), [mcp], [browser], or [all] — and a module that needs one says which, instead of failing with a bare ModuleNotFoundError. tests/test_base_install_surface.py holds that line: it imports each keyless module in a subprocess and fails if an optional dependency appears.

Check what the install can actually do before you rely on it. doctor lists every capability as OK or NO_CREDENTIALS — it never reports an unconfigured connector as an empty result, and it names any SCORITY_/PUBLIC_ variable you have set that nothing reads:

uv run scority doctor

One page, no accounts, no keys, no network beyond the page itself:

from scority_engine.modules.audit import audit_url

res = audit_url("https://example.com/")   # swap in your own page
print(res["score"], res["counts_by_severity"])
for f in res["findings"]:
    print(f["severity"], f["code"], f["title"])

69 {'MEDIUM': 3, 'LOW': 5}
MEDIUM meta_desc_missing Missing meta description
MEDIUM schema_missing No structured data (JSON-LD)
MEDIUM thin_content Thin content (21 words)
LOW canonical_missing No canonical tag
...

The whole site — crawl, technical audit sample, internal linking, UX/a11y, structured data, trust signals, local SEO and citability in one command:

uv run python -m scority_engine.modules.site_review https://example.com --pages 50 --lang en --out report.md

Is this page structurally citable by AI answer engines?

uv run python -m scority_engine.modules.geo.readiness --url https://example.com/guide

site_review degrades section by section: if one analyzer throws, that section becomes a note in result["notes"] and the rest of the review still ships. A malformed page does not kill the run.

What a verdict looks like

This is the honesty layer, and it is a pure function — deterministic, seeded, no network, no database. Paste this and you get exactly these numbers.

from scority_engine.modules.audit.safe_fix.measure import evaluate

# (before, after) CTR per page. test = got the fix, control = deliberately left alone.
test = [(0.021, 0.028), (0.019, 0.024), (0.031, 0.030), (0.012, 0.019), (0.026, 0.033),
        (0.017, 0.022), (0.023, 0.029), (0.014, 0.018), (0.028, 0.031), (0.020, 0.027)]
control = [(0.022, 0.023), (0.018, 0.017), (0.030, 0.031), (0.013, 0.014),
           (0.027, 0.026), (0.015, 0.016), (0.021, 0.022), (0.019, 0.018)]

print(evaluate(test, control_pairs=control).as_dict())

{
  "ci": { "point": 0.00475, "low": 0.00305, "high": 0.00625, "n": 10, "alpha": 0.05 },
  "method": "did",
  "decision": {
    "action": "expand",
    "reason": "CI entirely above zero — measured lift",
    "confidence": "low",
    "reason_code": "ci_above_zero"
  },
  "notes": []
}

Now drop the control cohort. Same test pages, same lift, different honesty:

{
  "ci": { "point": 0.005, "low": 0.0033, "high": 0.0063, "n": 10, "alpha": 0.05 },
  "method": "paired",
  "decision": { "action": "expand", "reason": "CI entirely above zero — measured lift", ... },
  "notes": ["no control cohort — paired estimate cannot separate the fix from a site-wide shift"]
}

And a change that did nothing — same control cohort, test pages that barely moved:

flat = [(0.021, 0.0215), (0.019, 0.0185), (0.031, 0.0312), (0.012, 0.0119), (0.026, 0.0261),
        (0.017, 0.0172), (0.023, 0.0228), (0.014, 0.0141), (0.028, 0.0279), (0.020, 0.0202)]

print(evaluate(flat, control_pairs=control).as_dict())

{
  "ci": { "point": -0.00021, "low": -0.00085, "high": 0.00051, "n": 10, "alpha": 0.05 },
  "method": "did",
  "decision": {
    "action": "hold",
    "reason": "effect not distinguishable from zero — keep gate",
    "confidence": "low",
    "reason_code": "ci_crosses_zero"
  },
  "notes": []
}

Note "confidence": "low" on a positive verdict. n=10 is below the module's own high bar (n≥30). The engine reports the lift and the thinness of the evidence in the same object.

The rules, and where they live in the code

Read the files. The claims below are one-line checks.

A verdict is computed, never asserted. audit/safe_fix/measure.py bootstraps the mean before→after delta (2000 iterations, seed=12345, alpha=0.05). With a control cohort it switches to difference-in-differences, so a site-wide algorithm shift that moves test and control equally nets out to approximately zero. The module docstring states plainly why it is not CausalImpact: real CausalImpact wants ~100 days and 40–50 pages, and a short window does not have that — so it reports "not distinguishable from zero" instead of claiming lift.

No signal, no expansion. evaluate() has exactly four exits before expand: any watchdog regression → rollback; n < min_pairs (default 8) → hold; CI entirely below zero → rollback; CI crossing zero → hold. expand requires ci.low > 0. loop.py adds one more: a cohort whose holdout is not viable is forced to hold — "autonomy never expands on a cohort it couldn't measure".

The counterfactual is real. audit/safe_fix/holdout.py splits a cohort by sha1(salt|url), so the same cohort always splits the same way and the split is reproducible without RNG state. The split is by page, never by user: one HTML per URL, no user-agent branching. viable is false unless both sides are non-empty and the cohort clears min_cohort.

Thin samples are never facts. provenance/confidence.py computes a Wilson score interval (asymmetric near 0 and 1, correct at the boundary) and derives trust status from sample size: n == 0 → unsupported, 0 < n < 30 → contested, n ≥ 30 → verified. proportion_provenance() is the one call a producer makes to ship a rate honestly — it will not stamp verified on a thin sample.

Contested SEO folklore cannot be laundered into fact. provenance/firewall.py holds a small versioned list of concepts that vendors and leaks present as ranking facts — site/domain authority as a standing factor, the "sandbox", CTR-and-dwell-time as a direct ranking factor, brand-mention causality. Any finding whose title, recommendation or evidence text matches is forced to status = "contested". A verified status cannot survive a match. Leak-derived heuristics ship as speculative from a tier-5 leak source, because a leak documents which attributes exist, not how ranking uses them (provenance/schema.py).

Provenance, and exactly how far it reaches. The canonical shape is {status, sources[], max_tier, ci, sample_size, method, prompt_version, contested_concept, measured_at}, with status ∈ {verified, contested, vendor-claim, speculative, unsupported} and method ∈ {self-measured, retrieval, coalition, probe}. It rides inside Finding.evidence["provenance"], so adding it broke no existing consumer.

This README used to say every number carries it. That was not true. Counting by the rule stated in tests/test_provenance_coverage.py — a module emits findings iff it constructs a Finding, or a dict literal carrying severity + title + recommendation that reaches a user under a findings key — 33 modules emit findings and 18 attach provenance. The other 15 attach nothing. The list is in that test file, and a new emitter fails the suite until it is classified, so this paragraph cannot quietly drift again. Closing one entry is a well-scoped contribution; they carry the provenance-gap label.

Everything the technical audit produces is stamped by construction (Finding.as_dict), as are the GEO access checks and the content gates. The unstamped set is mostly the analysers that grew before the schema existed.

The causal claim waits for the window to close. outcome/journal.py records an intervention at apply time with a measure_after timestamp (default 28 days — one Search-Console-comparable window) and status PENDING. The delayed pass closes it as MEASURED or INCONCLUSIVE. outcome/metrics.py recomputes CTR from click and impression totals rather than averaging per-row CTRs, weights position by impressions, and reports a metric that exists on only one side of the window as only_before / only_after instead of silently treating the missing side as zero.

Windows are fixed, not "recent". baseline/snapshot.py builds two adjacent 28-day UTC windows and ends the current window three days before the seed date, because Search Console data arrives late. as_of can be pinned for replay.

Autonomy fails closed. policy/engine.py matches flat conditions (cost, risk class, URL/domain substring, complaint rate, UTC hour window). A condition containing any key the engine does not recognize never matches — an unrecognized rule can only fall back to the human gate, never grant autonomy. No matching policy at all returns REQUIRE_APPROVAL with reason no matching policy — default human gate. An auto-ALLOW still writes an approval record, so "the policy did it" is as auditable as "a human did it".

Fixes are reversible and TOCTOU-guarded. audit/safe_fix/changeset.py compiles engine fixes into a structured, adapter-agnostic, reversible operation set. Anything the engine cannot model structurally degrades to manual_action with applyMode: "manual" — never guessed. edge/rewrite.py applies a ChangeSet to HTML: only auto changes apply, a before value that no longer matches live HTML is a stop (skip, do not clobber), application is idempotent, and unknown ops are skipped with a reason.

Verification refuses convenient answers. pr/verify.py treats rel=nofollow, rel=sponsored and rel=ugc as a failed placement, not a verified one — because sponsored and guest-post links are exactly what outreach buys, and counting them as equity-passing would be a reporting false positive against the person paying for the report.

AI visibility is a separate axis

Scority measures AI-answer visibility as its own dimension and never folds it into a single site health score. In audit/audit.py the GEO readiness result is returned as a separate geo_readiness field with its own geo_findings list; it does not move the technical score. The reason is in the source: citability and trust are high-bar composites that most technically clean pages score poorly on, so folding them in would put noise on every page.

That gap has a name in the code — clean_but_invisible: a page that passes the technical audit and is still structurally uncitable.

• geo/ai_access.py — does robots.txt let answer engines in? It distinguishes retrieval bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Google-Extended — blocking these genuinely removes you from live AI answers, so it drives severity) from training bots (GPTBot, CCBot, anthropic-ai, ClaudeBot, Applebot-Extended — blocking these is a common, deliberate opt-out and is reported as INFO, not HIGH). Most tools conflate the two and inflate the finding.
• geo/bot_traffic.py — what AI crawlers actually fetch, parsed from server access logs. Most AI agents fetch without executing JavaScript, so client-side analytics never see them; logs are the only ground truth.
• geo/answer_monitor.py — repeated waves of queries against the answer engines you configure. The engine decides whether a domain was cited and whether it was recommended (endorsement ≠ mention), by deterministic matching over the response text; the model is not trusted to score itself. Rates carry Wilson intervals, and wave-over-wave change is a paired bootstrap on the difference in citation rates — an interval covering zero is stable, not "improving".
• citability/score.py — an offline 0–100 scorer for "would an answer engine lift this?" (answer-first block, structure, question headings, first-hand signals, heading quality). No keys, run it on your own drafts before publishing.
• llms_txt/generate.py — spec-valid llms.txt from a sitemap, stdlib-only, output round-trips through the validator. The docstring says out loud that Google states this is not a ranking signal; treat it as cheap crawler inventory.
• edge/agent_markdown.py — clean Markdown for AI crawlers, with an explicit guard: classic search crawlers are excluded from the AI user-agent list, because serving Googlebot a different stripped page would be cloaking — the exact harm this project exists to prevent.

Compliance packs

Generated text passes a deterministic guard before it can be published. compliance/claims.py ships the Russian pack: forbidden advertising claims tied to the article of law they violate — guaranteed approval, "no refusal", instant money, income and employment guarantees, guaranteed debt relief, promises aimed at pensioners and borrowers with bad credit history, "100% exam pass". Each pattern carries a block or verify severity and a message naming the statute (ФЗ «О рекламе», ст.28 / ст.5).

Two details that matter more than the pattern list:

• A negation guard. "we do not guarantee approval" is a disclaimer, not a violation. Negation is clause-scoped, so a distant "не" in another sentence cannot suppress a real claim.
• Invented numbers are flagged separately. flag_unverifiable_specifics catches the figures a generator tends to fabricate about a specific brand — approval rate, client count, star rating, years in business, decision time — and marks them verify rather than blocking them. Grounding is value-aware: a number clears the flag only if it matches a fact you supplied, so a wrong figure still flags (compliance/facts.py).

The pack is a rule set, not a hardcoded jurisdiction. A second locale is a second pattern table plus its statute references. A Chinese pack (广告法 art. 9 superlatives as a hard block rather than a provable claim, art. 25 risk disclaimers, art. 28 unverifiable-evidence claims) is designed but not in this repository yet — it is on the roadmap, not in the code.

What is in the box

schema_audit/ deserves a specific note: it catches the defect class that every JSON-LD flattener hides — blocks that fail json.loads and are silently skipped, so a page "has schema" that no engine can read. It also catches duplicate single-per-page types and a missing @context, on top of per-type validators for FAQPage, Product, BreadcrumbList, HowTo, Review, VideoObject, Event and JobPosting.

render/ matters for the same reason: fetch_or_render tries a cheap static fetch first and escalates to Chromium only when the response looks like a bot interstitial or a client-rendered shell. Without it, an SPA or a site behind bot protection produces a page full of confident false defects. The audit detects that case explicitly and returns blocked_or_js_shell with score: None instead of reporting "missing title".

Architecture

fetch → (escalate to render if stub) → parse → checks ─→ findings + provenance
                                                          │
                                                          ├─→ fixes → ChangeSet → preflight gates
                                                          │             (reversible, TOCTOU-guarded)
baseline windows ─→ holdout split (test / control) ─→ apply ─→ measure (paired | DiD)
                                                          │
                                            expand | hold | rollback
                                                          │
                                            intervention journal → delayed re-measure (28d)

Two invariants hold the whole thing together. The engine owns data, determinism and arithmetic; a language model, where one is used at all, owns wording and judgment and never computes its own verdict. And nothing becomes a fact without a sample size behind it.

The optional judgment layer (fable/) talks to any OpenAI-compatible chat endpoint you configure via base_url + API key. Its gates are composed by the engine, not by the model: ragas_gate passes a draft only if it is both faithful to its sources and relevant to the query — an AND over two separately thresholded scores, with the failing reason returned. Nothing in the audit, scoring, measurement or compliance path requires a model to be configured.

Open core vs hosted

Same split as Supabase: the engine is the product, the hosting is a convenience.

You can run every capability in the left column yourself, forever, without an account. Nothing in the core phones home, and no module requires a Scority-hosted service to function.

Locales and markets

• Reports render in Russian or English (--lang ru|en).
• Text analysis is bilingual by construction — the YMYL classifier, the citability scorer and the first-hand-signal detector carry RU and EN pattern sets, not an EN-only list with a translation layer.
• Search data: Google Search Console alongside Yandex Webmaster and Yandex Metrika. The Yandex half of that pairing — connector plus analyzer — is hard to find in open source at all.
• SERP: Google via DataForSEO, Yandex via XMLStock; the adapter interface is small enough to add another.
• IndexNow pings Yandex, Bing, Seznam and Naver after a fix, to shorten the gap between applying a change and being able to measure it. There is deliberately no Google ping: Google does not support IndexNow and retired its sitemap-ping endpoint in 2023, so shipping one would be cargo cult.
• Compliance: Russian pack shipped, Chinese pack designed and not yet implemented (see above).

Status

Extracted from a production system that has been running against live client sites. This repository has no history from that system by design — it starts clean.

The measurement, provenance, policy and compliance layers are pure functions with offline tests and no fixtures. Scoring weights are directional and tagged for calibration; when we calibrate, the version string changes and you can diff it. If you find a claim in this README that the code does not hold, that is a bug and we want the issue.

Contributing

House rules, in order of how much we care:

1. A scorer is a pure function. No network, no database, no clock inside anything that produces a score. If it needs data, it takes data as an argument.
2. No new verified without a sample-size gate. If you emit a rate, emit it through provenance.proportion_provenance or explain in the PR why the existing gate does not apply.
3. Thresholds are tagged and versioned. New weights get # CALIBRATE and bump the module's version constant.
4. Fail closed. An unrecognized rule, an unmodellable fix, a cohort too small to measure — each falls back to the human gate. Never to autonomy.
5. Tests run offline. No fixtures fetched at test time, no live endpoints.
6. Don't add a claim the code cannot hold, in the code or in the docs.

Issues and pull requests welcome. If you are adding a locale pack, open an issue first with the statute references — the citation is the hard part, the regex is not.

License

Apache-2.0. See LICENSE.