SiteTriage

SiteTriage takes a list of domains, usually one or two hundred, scans each site politely and ranks the ones that deserve a full manual audit. Every ranked site comes with a few sentences, in Serbian or English, that you can paste into a message, use on a call or type into a contact form. It is built for web studios and consultants who have a long list of prospects and want to know where to spend their time. It reads the home page and a small sample of pages and tells you which sites have problems their owners would care about. The audit itself is still your job, and the tool does not look for vulnerabilities.

The screenshot shows a report built from the hand-written test set; the full HTML is docs/sample-report.html. Reports and the sentences for site owners are in English by default. --lang sr switches them to Serbian, and so does the rs profile: see docs/sample-report-sr.html and its screenshot. Each report is one file that loads nothing from a CDN, so it opens offline.

What it deliberately does not do

• It does not scan ports, try to log in, look for vulnerabilities or fill in forms.
• It does not go deeper than the home page and a sample from sitemap.xml. When there is no sitemap, it takes internal links instead.
• It does not request any path that robots.txt disallows. The home page and two probes for a fake 404 page are the exceptions: they are always opened.
• It does not guess what a business does. The industry is an input, not a conclusion.
• It does not fix anything. The tool reads; it never writes.

Ethics and responsible use

User-Agent: SiteTriage/3.0.0 (+hello@studio.example; Example Web Studio)

Every request carries the tool's name, who is scanning and how to reach them. Without that you are an anonymous bot visiting two hundred sites from one address, and an administrator who blocks you is doing the right thing. So the User-Agent names the operator, the person running the tool, not the author of the tool. The tool refuses to send a single request until the operator's name and contact are set.

On level 2 the browser sends the User-Agent that Chromium reports by itself, with the same identity appended (Mozilla/5.0 … HeadlessChrome/141… Safari/537.36 SiteTriage/…). Ad networks and Google Fonts choose what to send by User-Agent, and the page should be measured the way a visitor gets it (BUG-019).

• Never two requests to the same site at the same time, and www.x.example and x.example count as the same site. A global semaphore limits you; a per-site semaphore (always 1) protects them. That is the line between a tool and an attack.
• Between two requests to the same site there is a pause of 0.5 to 1 s, longer if Crawl-delay asks for it (up to 10 s).
• Disallow in robots.txt is respected.
• A hard budget per domain of 19 requests and 40 seconds: the home page, robots.txt, the three other host variants (http/https × with and without www), two probes for a fake 404, up to four sitemaps and seven sample pages. A 25 s budget was not enough for slow shared hosting, where a single page can take three seconds.
• On 429 or 503 the tool gives up on the domain for that run, without retrying.
• An administrator who writes "do not scan us" to the contact in the User-Agent is respected from the next run. Their domain goes into an exclusion file (one per line, # for comments), set with [operator] exclude_file or --exclude. Every subdomain is excluded too: an entry business.example also covers www.business.example and blog.business.example. An excluded domain gets no request at all, and the report shows only how many domains were excluded, not which.

Responsible use. The tool tells you which sites have problems. How you contact their owners is your responsibility. Cold outreach by email is regulated in many places (in the EU by the GDPR and the ePrivacy rules, and by national law), so check what applies to you before you send anything. Also note that each run with level 2 downloads a probe file from speed.cloudflare.com to measure your own connection (see load time); set [network_profile] probe_url to use another server.

Private addresses are never opened

The tool opens public addresses only. localhost, 10.x, 192.168.x, 169.254.169.254 and the other private and special addresses are never opened, not when a public name resolves to them and not after a redirect. A domain given with an IP address (in any notation, 0177.0.0.1 and 127.1 included), a port or a user name gets no request at all. Without this, a web application built on the scanner would let its users look inside the server (SSRF).

Staging with a port works only when you allow it explicitly:

[net]
allowed_private = ["staging.example:8123"]

In the browser (level 2) the protection cannot see the target of a redirect or a WebSocket, so a web application in production also needs an egress filter at the network level (ADR-006). Level 1 and the link probe before level 2 (probe_url) do not read HTTP(S)_PROXY from the environment, because a proxy would bypass the check. A probe_url on a private address is not opened either, unless allowed_private lists it: the link is then not measured, and load time is unknown.

Quick start

Docker (recommended)

The image contains exactly the Chromium that the Playwright version in pyproject.toml expects, so browser measurements behave the same on every machine.

git clone https://github.com/lazicmilos/sitetriage.git
cd sitetriage
mkdir work
docker compose build
docker compose run --rm sitetriage pytest -q        # every test must pass

work/ is mounted into the container as /work: put your input there, and the reports land there too. On Linux, set SITETRIAGE_UID=$(id -u) and SITETRIAGE_GID=$(id -g) first, so the files in work/ belong to you. Docker Desktop on Windows and macOS handles that by itself.

pip

You need Python 3.11 or newer.

pip install 'sitetriage[browser]'
playwright install chromium

To work on the code, install from a clone instead, as described in CONTRIBUTING.md.

Without the browser extra, level 1 works and level 2 is skipped with a message.

Tell it who you are

Before the first scan, set who is scanning. Every site's administrator sees the User-Agent in their server log, and it has to tell them who is visiting and how to reach them. Create work/my.toml:

[operator]
name = "Example Web Studio"
contact = "hello@studio.example"

The environment variables SITETRIAGE_OPERATOR_NAME and SITETRIAGE_OPERATOR_CONTACT do the same, and compose.yaml passes them into the container. The name may be written in Cyrillic or with letters like "š" and "đ"; the tool transliterates it to plain Latin for the header, because an HTTP header accepts only ASCII.

The defaults ship with the tool; your own TOML passed with --config overrides only what it sets. All defaults, with comments, are in sitetriage/sitetriage.toml. A value must have the type of its default (a number where the default is a number, a whole number for counts) and a sensible range (points, timeouts and ratios above 0, delay_ms as [min, max], severity tiers that grow); otherwise the tool stops before the first request and names the key and the value. A key or section the defaults do not have is an error too, with the closest known key when there is one (html_size_kbb → did you mean thresholds.perf.html_size_kb?), so a typo cannot silently leave a default in place.

Profiles

A profile is a set of settings for one market that ships with the tool, in sitetriage/profiles/. It is merged between the defaults and your own file: defaults, then the profile, then --config. Choose it with --profile NAME on scan, recheck and record, or with a top-level profile = "NAME" in your --config file. The flag wins, and --profile "" turns off a profile your file chooses. An unknown name fails and lists the profiles that exist.

There is one profile so far, rs, for sites in Serbia. It turns on Serbian content detection (thresholds.i18n.content_languages = ["sr"]), and with it i18n.lang.mismatch: Serbian content on a page whose lang names another language. Without a profile that check is "not applicable", because its only detector recognises Serbian and would mistake Croatian or Czech text for it (ADR-023). With the profile, the report and its sentences are in Serbian too (report.lang = "sr"); add --lang en to keep them in English. Every other check works the same with or without a profile.

sitetriage scan leads.csv --config my.toml --profile rs --out report/

The profile a run used is in report.json (profile), in the header of the HTML report and in the summary line at the end of the run.

First scan

docker compose run --rm sitetriage sitetriage scan /work/leads.csv --config /work/my.toml --out /work/report

or, without Docker:

sitetriage scan leads.csv --config my.toml --out report/

The recommended machine has 16 GB of RAM and a connection of at least 50 Mb/s. On it, the target for a list of 200 domains is 15 minutes or less (ADR-019; not yet measured on such a machine). The tool also works on a weaker machine, just more slowly: the light level 2 starts as many browsers as the machine can take, one per free GB of memory, at most two per CPU core and at most 8. You can fix the number with [browser] light_concurrency.

Input and output

The input is a CSV with a header, not a bare list of domains:

domain,industry,note
hotel-planina.example,hotel,mountain
restoran-centar.example,restaurant,
klinika.example,healthcare,clean control site

industry is one of hotel, restaurant, healthcare, ecommerce, b2b, institution and other. An empty or unknown value falls back to other. The industry changes how much each finding is worth: a link-preview image matters a lot to a hotel and hardly at all to a b2b site.

The list usually comes out of Excel, so the tool also accepts what Excel writes on a Serbian Windows system: columns separated by ; and a file saved in windows-1250 instead of UTF-8. Header names ignore letter case and the spaces around them (Domain, Industry, Note works), and a column the tool does not know is ignored with a warning, so a typo like industy does not go unnoticed. A site that appears twice is scanned once, and the first occurrence wins. Two rows are the same site when they differ only in letter case, the scheme, a leading www., a trailing . or /, or a port that is the default for the scheme: x.example, https://www.x.example/ and http://x.example:80 are one site, while blog.x.example, x.example:8443 and x.example/en (another entry page) are not. Errors and warnings name the row, the way Excel numbers it.

The output folder holds:

--format chooses which of the first four are written (html, csv, json or several separated by commas, e.g. --format html,json; default all three formats). Case and spaces do not matter, and any other name is an error before anything runs. Snapshots are always written. --lang en|sr sets the language of the report and its sentences. Without it the tool uses report.lang: en by default, sr with the rs profile. --csv-bom writes the CSV files as UTF-8 with a BOM, for Excel.

Only working sites are ranked. A site that does not open on the second attempt either, at least 60 s after the first (http.second_attempt_after_s), goes to the "Not working" section, with its own suggested text. That happens only when the state is certain: the name does not exist in DNS or has no address, or the server accepts no connection on either https or http (ADR-013). A domain without a home page for any other reason (DNS temporarily unavailable, an interrupted TLS handshake, any HTTP response, an address that is not public) goes to "Not scanned", with the reason, because the site may work and the tool just did not see it. That section also shows the number of excluded domains.

The JSON report

report.json is the contract for a web application, sitetriage diff (planned), a CRM, or whoever builds on the code. Its shape is described by the schema sitetriage/schema/report-3.json, and the schema version is the first field of the document (schema_version, now 3.0). Next to it are the tool version, the run time in UTC, the duration of each level, a configuration digest (config_digest: the sha256 of the thresholds, multipliers and points, without the operator identity and concurrency), the profile (profile, null without one), the environment (OS, Python, Chromium, network profile, measured link speed) and totals by status. Domains are in three lists: ranked, unreachable ("Not working") and not_scanned (failed, with the cause in reason).

A domain is partial for one of two reasons, and partial_causes says which: the budget ran out (budget), or a check could not run (unknown). The totals count them separately, and budget_share and unknown_share give the share of working domains that ran out of budget and the share of unknown for each check. The denominator is the working domains on which the check ran: for a level-2 check only those that went through level 2, and for load time only those measured under the network profile (ADR-015). The HTML report shows both shares under the totals.

A finding in the JSON has no sentence. It has a check_id, the evidence and a variant, and the sentence is built from the evidence, in the reader's language.

Rules for changing the schema (ADR-011, ADR-012):

• an added field raises the minor version (3.0 → 3.1), because an older reader can skip it;
• a removed field, a changed type or meaning of a field, and a renamed check_id raise the major version (3.x → 4.0).

The schema does not allow unknown fields, so a new field has to go into the schema first. A test fails if the output does not validate against the schema.

Other commands

sitetriage recheck report/snapshots/ --out new/                       # score saved snapshots again, no network
sitetriage recheck report/snapshots/ --config trial-thresholds.toml --out new/
sitetriage recheck report/snapshots/ --lang sr --out serbian/         # the same run, report in Serbian
sitetriage recheck report/snapshots/ --profile rs --out rs/           # Serbian content detection and report
sitetriage record my-domains.csv --out recordings/                    # record snapshots as fixtures
sitetriage explain seo.canonical.duplicate                            # what a check does, its threshold and sentences
sitetriage explain --all --markdown                                   # the table of checks below
sitetriage scan domains.csv --level 1 --concurrency 4 --only institucija.example --debug-domain institucija.example

recheck is the main calibration tool. Change a threshold, run it over saved snapshots and see the difference in a second, without downloading two hundred sites again. Try thresholds in a separate file that holds only what you change: --config overrides only what it sets.

scan also takes --level 1|2|auto, --max-level2 N (the cap for level 2, default 60), --snapshots DIR, --exclude FILE and --debug-domain DOMAIN (DEBUG log for that domain only: every request with its status or error, time, redirect hops and whether the body was cut off, and on level 2 the navigation, load and the measured requests and bytes). Run sitetriage <command> --help for the full list.

How it works

The split into level 1 and level 2 matters less than another decision: code that touches the network and code that draws conclusions live in two separate layers. A check never opens a connection. It receives data that was already collected and returns results.

domains.csv → Fetcher L1 (httpx, async) → SiteSnapshot to disk → L1 checks → findings
                                                                      ↓
                                                              escalation policy
                                                                      ↓
                                     Fetcher L2 (Playwright) → BrowserSnapshot → L2 checks
                                                                      ↓
                                               scoring → ranking → CSV + HTML + JSON

Level 1 reads raw HTTP: the home page, robots.txt, host variants, sitemaps and a sample of pages. Level 2 opens the home page in a real Chromium, under a fixed simulated network profile, and measures weight, requests, load time, headings, images and console errors. Sites with a reason to look closer get the full level 2, up to a cap of 60; every other working site gets a lighter level 2 without the network profile (ADR-019).

Because the layers are separate, the tests run without a network and always give the same result, calibration needs no new scan, and when a domain gives an odd result, the exact input that produced it is saved on disk.

Two architecture rules are enforced by tests/test_architecture.py:

• checks.* does not import httpx, playwright or fetch.* (except fetch.urls), nor pipeline, messages or report. If a check needs something from the network, the snapshot is missing a field, so the field gets added.
• cli does not import fetch.http, fetch.browser or score. The order of phases exists only in pipeline, so a web application calls it instead of copying it (ADR-008).

Checks

The table is generated from the check registry with sitetriage explain --all --markdown. When a threshold changes, the table is generated again; CI checks that every check in the registry appears in it.

Four states

Every check returns one of four states, and the last two always come with a reason:

• ok: checked, no problem.
• finding: checked, a problem, with evidence.
• unknown: the check could not give an answer. Without this state you cannot tell "clean" from "not checked", and that difference is exactly what would cost you trust in your own report a month later. For example, when a site answers every request with 403 because it blocks bots, the tool does not claim that robots.txt is missing. It says it does not know.
• not_applicable: there is nothing to check. The duplicate checks give it when a site really has fewer than three pages. A small sample is not enough to say so, because the sample is also small when the budget runs out or when links are drawn by JavaScript. So the tool counts addresses from the sitemap, from links on the home page and from links the browser sees, and at least one source must see content drawn by JavaScript. When any of these conditions does not hold, the check is unknown. Unlike unknown, not_applicable does not make a domain partially scanned. When the sitemap lists fewer addresses than the sample needs, the sample is topped up with links from the home page (ADR-015). i18n.lang.mismatch gives it, with the reason no_content_language, when no profile turns on content language detection.

Sentences for the site owner

Client sentences claim only what can be verified, and every number in them comes from measured evidence (ADR-022). Tests in tests/test_messages.py guard both rules, along with number and noun agreement ("4 glavna naslova", "5 glavnih naslova", "4 main headings").

A finding carries only evidence: numbers, URLs and technical codes. The catalog in sitetriage/messages/ turns it into a sentence at display time, in the language chosen with --lang sr|en (ADR-009). That is why the same run can be shown in either language. The English sentence claims exactly what the Serbian one claims, no more and no less, and sitetriage explain prints both. The same goes for the reasons why something was not checked and why a domain was sent to level 2.

Why the thresholds are what they are

"List A" and "list B" below are two private calibration lists of 100 real Serbian websites each, mostly small businesses. Thresholds were set from the distribution on list A, then frozen and validated on list B, which does not overlap with A (ADR-020). The lists are not published.

Ranking is not by total but by the pair (most severe single finding, total). A site with one disaster is a better lead than a site with ten small issues: the first has one clear reason to answer you, and few people answer a list of ten items. Points are multiplied by an industry multiplier per category ([industry_multipliers.*]); the multiplier changes the weight of a finding, never its severity.

All thresholds live in sitetriage/sitetriage.toml, not in code, so git diff on it is a readable history of calibration decisions. The calibration log behind them is docs/sr/kalibracija.md (in Serbian); the decisions themselves are summarized in English in the ADRs.

Both lists come from one country and are mostly small businesses and associations, and they were scanned with what is now the rs profile. For other industries and markets the distributions may differ, so repeat the calibration on your own lists with recheck.

Tests

docker compose run --rm sitetriage pytest -q               # offline, with a real Chromium, about 4 min
docker compose run --rm -e SITETRIAGE_LIVE_CSV=/work/live.csv sitetriage pytest -m live -v   # real sites from your own list; needs a network

The first layer does not depend on the network and runs in CI on every push. It covers every check on snapshots, scoring, URL normalization and sitemap parsing. It also has integration tests against a local HTTP server that serves broken sites on purpose: a slow response body, a stream that never ends, bot protection, a sitemap in nine parts. That is how the per-domain budget, the pauses, the semaphores and the browser measurements are tested. The second layer goes to real sites and is off by default; otherwise CI would fail every time some site's certificate expired. The repository names no real site: you give the list through SITETRIAGE_LIVE_CSV, a CSV in the format of domains.example.csv (see tests/test_live.py for the optional column of expected findings).

The baseline is a clean site on which no check may find anything. Each positive test breaks exactly one thing and checks that no other check woke up. Thresholds have boundary value tests (limit − 1, limit, limit + 1), and rules with several conditions have decision tables. The test technique is in the name of each case: gv- boundary value, ke- equivalence class, tab- decision table, st- state transition, pg- error guessing (the prefixes come from the Serbian names of the techniques).

CI fails if line and branch coverage drops below 95 %. Mutation testing measures the quality of the tests themselves: mutmut breaks the code on purpose, and the tests have to notice every change that alters behaviour. mutmut 3.2.3 does not create mutants inside decorated functions, so it does not mutate check bodies (@check). It runs through Docker, because mutmut does not work on Windows, and takes about 20 minutes:

docker compose run --rm sitetriage python scripts/mutation.py

The fixtures in tests/fixtures/ are hand-written, not recorded from real sites. They describe what the specification says about those eight domains and check that the chain of checks works on that state, with the specification's thresholds. The domains are made up (.example). Real sites are checked by pytest -m live, over the list in SITETRIAGE_LIVE_CSV.

How to set up, run the tests and add a check is in CONTRIBUTING.md. Before a release that changes fetching or thresholds, run the real-site verification and the manual check. The history of how v1 and v2 were tested (test plan, three test reports, calibration log and an ISO/IEC 25010 review) is in docs/sr/, in Serbian.

Contributing, security, code of conduct and license

• CONTRIBUTING.md: development setup, tests, architecture rules, adding a check.
• SECURITY.md: report a vulnerability privately, not in a public issue.
• Code of Conduct applies to everyone taking part in the project.
• CHANGELOG.md: what changed in each release.
• docs/: architecture decision records and procedures.
• The code is under the MIT license.

Roadmap: not in v3

• sitetriage diff between two runs and sitetriage outcomes (which leads answered and turned into work). They come once there is a real subscription. "Since March the site has become 3 MB heavier" is a good opening for an email.
• A check for "home page with almost no content": a home page that has almost no text even after JavaScript (a splash page with one button, a parked domain) is the strongest argument for an email, and today it is not a finding. Of 160 home pages on lists A and B, 4 have fewer than 200 characters; the limit will be calibrated on A and B.
• Core Web Vitals (LCP, CLS): useful numbers, but they need more careful measurement.
• CPU emulation, so that load time no longer depends slightly on the machine.
• A check for structured data (schema.org), which has real value for restaurants and clinics.
• CMS detection (WordPress, Wix, custom-built), because it changes how an offer is written.