fhir-scorecard

A plain-language operational scorecard for publicly observable FHIR (Fast Healthcare Interoperability Resources) endpoints.

Live site: fhir.chelseakr.com — every endpoint in the registry rescored daily, each grade with the findings and spec citations behind it, plus the machine-readable dataset and API. Published by .github/workflows/pages.yml; tools/verify_live_site.py fails if what is served stops matching what this checkout builds — unless no publish has run since the change, which it reports as pending rather than as a fault.

CMS interoperability rules require regulated payers to stand up FHIR R4 APIs (Patient Access, Provider Directory, and, under CMS-0057-F, more to come). Whether those endpoints are actually reachable, honestly documented, and interop-ready is publicly observable today, and nobody grades it in language a non-engineer can act on. This project does for FHIR endpoints what transit data-quality tooling did for GTFS feeds: fetch what is public, run deterministic checks, and publish a letter grade with a short, prioritized list of findings and spec citations.

Quick start

make sync                         # uv sync --locked: exactly the toolchain uv.lock pins
make verify                       # lock check, lint, format, strict typecheck, tests + coverage floor, audit
.venv/bin/fhir-scorecard grade --registry data/registry.json --out site/

Offline mode (no network) grades the discovery documents captured under tests/fixtures/, which are real /metadata and SMART documents with a capture date on them, not hand-written examples:

.venv/bin/fhir-scorecard grade --offline \
  --fixtures tests/fixtures --registry tests/fixtures/registry.json --out .cache/offline-site

An offline run writes its availability history to .cache/ unless you name a path, and refuses to write into a history file a live run wrote. Fixture observations in the real availability record would report a day nobody measured. See tests/fixtures/README.md for what each capture covers and how to refresh one.

Check a single endpoint without the registry, the history, or the site. Nothing is published and nothing under data/ is written or read:

.venv/bin/fhir-scorecard check https://fhir.example.org/r4 --kind payer --min-grade B

With a --min-grade the command exits 1 when the measured grade is below it, which makes it a build gate for the operator of the endpoint being checked. Without one it reports what it saw and exits 0, because a finding about a published document is data, not a failure of the program that read it. An endpoint no vantage reached is reported as not observed, never as F; with a threshold set that fails the gate, on the stated ground that the threshold could not be evaluated. The same command is packaged as a composite GitHub Action in action.yml — see docs/ci-action.md.

An operator with several endpoints — a payer with Patient Access, a provider directory and a sandbox — can check them in one run by pointing check at their own list:

.venv/bin/fhir-scorecard check --registry operator.json \
  --junit fhir-junit.xml --sarif fhir.sarif --min-grade B

The file is the same shape as data/registry.json minus the verification blocks. An operator checking endpoints they already run publishes no attribution, so there is no verification claim to record, and a verification block is refused rather than ignored. Each entry may carry its own min_grade, which overrides the run-wide one, so a sandbox and a production API can sit in one file and be held to different bars:

{
  "endpoints": [
    {"id": "patient-access", "name": "Patient Access", "kind": "payer",
     "base_url": "https://fhir.example.org/r4", "min_grade": "B"},
    {"id": "sandbox", "name": "Sandbox", "kind": "payer",
     "base_url": "https://sandbox.example.org/r4"}
  ]
}

--junit writes a testsuite per kind and a testcase per endpoint and dimension; --sarif writes SARIF 2.1.0 with one result per finding, each carrying the specification passage it cites. Neither carries a timestamp, so two runs over the same documents produce the same bytes and a diff between yesterday's artifact and today's is a change in the endpoints.

An endpoint this run did not reach is reported as not measured, never as a failure. Its testcases are skipped naming the vantage, its SARIF results are note with a severity of not observed, and the run's summary counts it separately from the endpoints that were graded, so a run that reached nothing cannot summarize as a clean one. The exit code still answers only the thresholds you set, exactly as the single-endpoint check does: an unreachable endpoint is a fact about the network path between your runner and it as much as about the endpoint, and a build that goes red for that is blaming the endpoint for the runner. To make an unreached registry fail on purpose, pass --min-grade F — a threshold cannot be evaluated without a grade, so it fails.

What it observes, and what it never touches

Everything graded here is public, unauthenticated surface:

• [base]/metadata , the FHIR CapabilityStatement every server must expose
• [base]/.well-known/smart-configuration , SMART on FHIR discovery

This project never accesses patient data, never authenticates, and never probes beyond the public discovery surface. Two documents per endpoint per probing run, an identifying User-Agent with a contact address, HTTPS only, and conservative timeouts. HTTPS and the two-path scope hold on every hop: a redirect is followed only when it still names one of those two paths over HTTPS, on any host, because a stock urllib opener would happily have taken /metadata to a patient search or onto a plaintext connection. That is a tested property, not a claim: tests/test_probe_contract.py runs the real fetcher against a loopback server and fails if the second request is ever made. The site is rebuilt on a schedule and on demand, never on a commit, so a scheduled day normally costs an endpoint six requests: two documents from each of three probing runs, and none from the run that publishes. Following a redirect costs another GET, so the published ceiling is four requests per document, eight per endpoint per run, and 24 per endpoint per scheduled day.

Where it measures from

Every published grade reconciles probes from more than one vantage (vantage.py), on a deliberately asymmetric rule: one vantage reaching an endpoint proves it is reachable; one vantage failing proves nothing. That rule exists because a live payer endpoint was once recorded as dead when a middlebox on the probing network intercepted TLS.

What the vantages are, precisely: three GitHub-hosted runner images (Ubuntu, macOS, Windows). They are three hosts on one provider's network, not three independent networks, and nothing this project publishes calls them that. Three hosts catch a fault local to one host or one TLS trust store, which is the failure that prompted the mechanism. They cannot catch a source-address rule, bot filter, geo rule, or rate limit applied to that provider's address space, because such a rule reaches all three at once. A run where every vantage failed is therefore published as not reached from that network on that day, with the reason, rather than as an endpoint being down.

Each vantage counts once. The publishing run makes no probe of its own and grades the documents the probing runs retrieved (--from-probes); before that it re-probed under a label one artifact already carried, and every card reported four vantages when three had reported. Adding a genuinely independent vantage — a residential or other-provider runner posting a probes-*.json — is an open item in ROADMAP.md, and until one exists the published wording stays "one network."

What it grades (v0.1)

Grades are deterministic. Every finding carries a citation to the FHIR R4 or SMART App Launch spec, and every finding describes a document this project actually retrieved.

An endpoint no vantage could reach is published as not observed, with the reason and the vantages that tried, and is not graded: its content dimensions carry no score, and the checks that read a CapabilityStatement do not run. It never drops out of the dataset, and it never acquires findings about what its publisher did not publish. F means the opposite and only the opposite: the endpoint answered, and what it declares falls short across the checks. The two used to share a letter, and the site rendered both with one sentence about a network.

Between them sits a third case that real servers produce often: /metadata answers with HTTP 200 and the body is not a CapabilityStatement — an OperationOutcome, a sign-in page from an authenticating gateway, a search Bundle. That endpoint is reachable and it is graded, because answering that path with that document is a fact about it. But the checks that read fields inside a CapabilityStatement have nothing to read, so they report the parse failure once (T0, I0) and carry the weight they always carried, rather than reporting the parser's empty defaults as things the publisher declined to declare. Same score, no invented claim.

How this relates to Inferno and Lantern

• Inferno (ONC) is a conformance test kit run against a server you control, primarily for certification. This project runs no test suites and asserts no certification status; it grades the public operational surface of endpoints in the wild.
• Lantern (ONC) monitors FHIR endpoints of certified EHRs on the provider side. This project's target registry is the payer side, which has no equivalent public monitor.

Complementary, not competing. If you need conformance testing, use Inferno.

Registry honesty

data/registry.json ships with a small set of live-verified public reference servers so the tool is runnable out of the box. Payer endpoints are added only after live verification (fetch the CapabilityStatement, confirm the publisher matches the claimed organization, record the method and date in the entry). Unverified entries are never shipped. See CONTRIBUTING.md.

Attribution follows the publisher's own words and never a URL path segment. Where a vendor runs a multi-tenant payer platform the conformance document often names the vendor, or nobody at all, and attribution then rests on the plan publishing the base URL on its own site, which the entry's verification record says outright. Where only a vendor or a path segment connects a server to a plan, the endpoint is excluded instead.

An entry records on what basis it is listed and when it was last checked, as two separate facts a machine can read:

• verification.basis is live_capability when a conformance document was retrieved, or publisher_documented when the organization publishes this base URL in its own materials and the document was not retrievable on that date. The second requires the source that printed the URL and what the probe observed, and such an endpoint publishes as not observed rather than F. It is listed on purpose: an unreachable endpoint is a finding, not a reason to drop it, and a registry pruned of its failures hides exactly what this project exists to detect.
• verification.reverified is a later dated re-check, never an overwrite of the curation date. An entry with no re-check block has not been re-checked, and its page says so in words. A stale date must not be able to read as a fresh one.

Re-checking used to be entirely manual. fhir-scorecard reverify does the retrieval and leaves the judgment where it belongs:

# Re-check every entry nobody has looked at in 90 days. Writes a proposal, edits nothing.
fhir-scorecard reverify --older-than 90d --out reverify-2026-09-06.json

# Read it, set "accepted": true on the rows whose attribution you have confirmed, then:
fhir-scorecard reverify --apply reverify-2026-09-06.json

Three properties hold it to the rule above:

• An entry whose document was not observed gets a row saying so and no proposed block at all, so there is nothing --apply could write. Marking such a row accepted is refused with the reason, rather than skipped quietly; a skipped row reports success for a date that was never earned.
• Outcomes are match, unconfirmed and not_observed. There is deliberately no "mismatch": a vendor-hosted platform that does not repeat the plan's name is the ordinary case, not evidence against the entry, and the verb has no standing to say otherwise.
• --apply with nothing accepted writes no byte, and --apply reaches no network. Producing a proposal and applying one are separate runs because the decision between them is a person's.

Reading an inbound claim

The add-endpoint issue form collects exactly what an entry needs. fhir-scorecard claim reads one submission, retrieves the same two discovery documents every other verb retrieves, and writes a proposal plus the text of a comment. It edits no registry and opens no pull request.

# The rendered issue body in a file; the registry is opened read-only, to refuse a duplicate.
fhir-scorecard claim issue-body.md --out claim-2026-09-07.json --comment-out comment.md

The submitted base URL is the one URL in this project that an unauthenticated stranger chooses, so the boundary runs before anything is requested: the scheme must be https, the URL must carry no credentials and must not already be a discovery path, and every address the host resolves to must be public — one loopback, private, link-local or reserved address disqualifies the host even when others are public, and a name that will not resolve is refused rather than attempted. The tests drive the whole verb with a fetcher that fails the run if it is called at all, so "nothing was requested" is asserted rather than assumed.

Two limits are deliberate and are stated in the output rather than worked around:

• The submitted documentation page is not retrieved. It is the evidence that the publisher is who the claim says, and reading it would mean requesting a page that is not one of the two discovery documents — a different activity from the one this README describes, and an open decision recorded on #118. Until it is settled the URL is carried through unchecked and published as not retrieved, in those words, and no proposed verification block rests on it. The scope sentence above therefore still holds exactly as written.
• A claim whose CapabilityStatement was not retrieved proposes nothing at all. The other basis, publisher_documented, is defined by that documentation page, so an unretrievable claim cannot honestly reach either basis. It gets a comment naming what happened and no entry.

A document that does not repeat the submitted organization's name is attribution_review, not a rejection and never a "mismatch" — the same vocabulary and the same comparison reverify uses. Where an accepted claim belongs, data/registry.json or data/CANDIDATES.md, is the maintainer's open call; the proposal takes no side on it.

Cohorts and the sampling frame

A cohort is a named view over the registry whose membership comes from a public roster rather than from whatever was easy to find, which is what lets a hit rate mean anything: the denominator is fixed before any probing starts. Every member either points at registry endpoints or carries an exclusion with a reason, a review record, a date, and a source, so the plans that publish nothing discoverable are part of the published result rather than an absence in it.

The rule that decides which organizations get looked for at all is written down in docs/SAMPLING-FRAME.md, including what was considered and rejected as a roster - among them the guessed-hostname method that produced 0 verified endpoints out of 18 probes and a hit rate that meant nothing.

The first is the California payer cohort at /california/: the Medi-Cal managed care plans DHCS lists plus the Covered California qualified health plan issuers, deduplicated to 27 organizations. Eight publish a base URL this project verified from their own documentation, on the dates in data/registry.json; how many of those endpoints answered on any given day is a separate number, measured by the run that generated the page and printed beside the curated one. For the Medi-Cal managed care plans, those APIs are required to exist by the federal CMS Interoperability and Patient Access rule (CMS-9115-F), through its Medicaid managed care prong, which is the only obligation this project claims about them. The four members that are on the Covered California list alone are not obliged by that membership, because the rule's qualified-health-plan prong reaches the federally-facilitated exchanges (more on that below); each member's obligation_basis in the cohort file says which case it is. The rule does not require a plan to print its base URL where an unregistered visitor can read it; California's Data Exchange Framework runs through the DSA and QHIOs and requires none of these surfaces; and CMS-0057-F's additional APIs are not in force until 2027 and are not graded.

The second is the Texas marketplace issuer cohort at /texas-marketplace/, and it exists because California's exchange is state-based. CMS-9115-F's qualified-health-plan prong (45 CFR 156.221) reaches issuers on the federally-facilitated exchanges, so this cohort takes that prong from the regulator's own file: every issuer selling an individual-market QHP on HealthCare.gov in Texas for 2026, as enumerated by CMS/CCIIO's QHP Landscape PY2026 Individual Medical dataset - 13,013 plan-county rows, 18 HIOS issuer IDs, 15 issuer organizations, frozen before any URL was probed. Six publish a base URL this project could verify from their own documentation. Three of the fifteen publish a Patient Access base URL and one of the three answers a stranger; six publish a Provider Directory base URL and four of the six answer.

One attribution this cohort should not make, and used to: the unauthenticated-provider-directory duty is not this prong's. 45 CFR 156.221 is a Patient Access API section and says nothing about a provider directory; the API that must answer without authentication is 42 CFR 422.120, which reaches Medicare Advantage organizations, with parallel provisions for Medicaid and CHIP. Several of these issuers run MA plans too, so a directory endpoint one of them publishes may well exist for that reason. This project measures the public surface either way and does not decide which obligation, if any, a given endpoint answers to.

The third is the Florida marketplace issuer cohort at /florida-marketplace/: the same federal prong in the largest HealthCare.gov state. From the same CMS file - 7,569 plan-county rows, 16 HIOS issuer IDs, 15 issuer organizations, frozen before any URL was probed, with the per-issuer roster committed beside the cohort as florida-marketplace.roster.csv. Nine of the fifteen publish a base URL this project could verify from the organization's own documentation, which inverts the Texas result, and the smallest issuers are among the cleanest publishers: a brand-new one-county plan prints its base URLs in the open while three national names publish nothing a stranger can reach. The frame behind the twenty marketplace cohorts is national - 176 state-issuer organizations across 30 federally-facilitated-exchange states, committed under data/frames/ - of which those cohorts have reviewed 133; the other 43 are not yet reviewed, a statement about this project's progress that is never rendered as "publishes nothing" (see docs/SAMPLING-FRAME.md).

Findings

docs/findings/ holds dated write-ups of what pointing this tool at real endpoints produced, with the evidence beside each one and every published figure recomputed from that evidence by tests/test_findings_evidence.py rather than typed.

• What 27 California health plans publish about their FHIR endpoints (curation review of 2026-08-07): 8 of 27 organizations on a public roster publish a base URL this project could verify. The other 19 are split four ways rather than counted as one number, because a plan publishing nothing and a plan publishing a URL that returns 404 are different results.
• One URL, three brands: three consecutive requests to one payer's documented base URL returned CapabilityStatements naming three different brands, and what that means for anyone building a payer endpoint registry. Re-checked 2026-08-19: that URL now answers 401, so the rotation is no longer observable from outside, while the shared Provider Directory address in the same document still answers and is stable. The re-check is a dated section on the write-up, not an edit to what was seen on 08-07.

Neither is a compliance determination, and both say so.

The site

Every endpoint, organization, category, and cohort gets its own indexable page with a canonical URL, description, and structured data, plus a sitemap and a methodology page that every finding links into. Each endpoint also gets a single-endpoint report at /endpoint/<id>/report/, linked from the endpoint page: what this project observed, what it did not observe and which of the two kinds of nothing that was, every reporting vantage's own result, and what a publisher would change to score better, ranked by the points each change would recover. It is free, unauthenticated, indexable, and built to print; nothing about it is behind a sign-in and no action is ever listed for a check that did not run. Every endpoint's observation record is browsable at /history/: an index saying what window the record covers and how many endpoints are still below the reporting floor, and one page per endpoint listing every observation with its date and whether the endpoint answered. An endpoint with no observations says so; it does not render a zero. No rate is published below fourteen observations, on the same ground the grades use: a percentage off two data points is noise dressed as a metric.

/over-time/ reports what changed across the record, one section per calendar month, computed on every publish from the observation record and stored nowhere. It says who was observed, who answered every check, and what changed in what they declare. It does not report grade changes, and says why on the page: the record retains availability and a capability fingerprint and has never retained a grade, so no run can look up last month's letter, and deriving one from the fingerprint would invent a measurement nobody took.

/coverage/ answers the question the frame was built for: how much of the federal marketplace has a publicly checkable FHIR endpoint at all. Every one of the 176 state-issuer organizations lands in exactly one of four populations, and no two of them are ever added together: publishes a base URL a conformance document was retrieved from (80), publishes one that did not answer (6), was reviewed and publishes none a stranger can read (47), and not yet reviewed (43). The fourth is a fact about this project's progress, never about an issuer, and publishing_rate raises rather than compute a rate over a set that contains one.

/availability/ orders the endpoints that have enough recorded observations to state a share, within each kind and never across one, and names the endpoints that do not with their counts and how many more observations each needs. On the live record on 2026-08-27 that is 30 endpoints ordered and 15 named. A position orders one measurement - whether a public address answered a request for its /metadata document, from three hosts on one provider's network, over the recorded window - and the page says so where the numbers are. It is not a service-level measurement and not a statement about an organization.

/cms-0057-f/ is a readiness report for payers and API vendors, built on every publish from observations the run already holds. For each payer-side endpoint it prints what the two public files declare about SMART Backend Services and Bulk Data (the app-to-server block on the endpoint page), counted per question in five states that are never merged: declared, not listed, absent, could not be read, not retrieved. It also groups the endpoints by the software name their CapabilityStatement declares. It is not a grade and not a compliance determination, it changes no grade, and it does not grade the APIs CMS-0057-F adds, which are not in force yet. Every sentence it says about the regulation is a closed list in src/fhir_scorecard/readiness.py, each with the words in this repository that support it, and each marked as unverified against the rule text, because this repository does not cite or retain that text (ADR 0009).

Each record also carries the endpoint's declaration timeline: every recorded change to what its CapabilityStatement says about itself, dated and in order, with the declarations it has gone back to listed separately and counted rather than repeated. One hostname in front of two backends produces a return every time a probe lands on the other one, and listing those as releases would bury the releases that are real.

Any of that can be subscribed to rather than visited. /feed.xml carries every event the record holds across every endpoint, /endpoint/<id>/feed.xml carries one endpoint's, and /<cohort>/feed.xml carries a cohort's, as Atom 1.0 files rebuilt from the same history.json the pages are built from. No accounts, no email, no service. Four kinds of entry, never merged: an endpoint entering the record, a declaration change, a return to a declaration already on record marked as a return, and a move between answered and did not answer. Every entry carries the date the record holds and never the build's, and its id is a function of (endpoint_id, kind, date, digest) rather than of the address the site is served from, so a rebuild republishes nothing and a hosting change republishes nothing. Every feed states how many events it carries out of how many the record holds and over what window: the site feed is capped at the newest 100 and says so, because a truncated dataset published as a complete one is the defect this project is organized against. An endpoint with no recorded event gets a feed with zero entries and a sentence saying which of the two reasons applies, not a 404. Two things a feed will not say: it reports no grade, because the record retains none; and an availability entry names no vantage, because an observation in the record is a date and a reconciled answer and the vantages behind it were never retained.

Each endpoint is also published as what its CapabilityStatement declares, not only as a grade. /endpoint/<id>/capabilities/ lists every declared resource with its interactions, search parameters, operations and declared profiles, and api/capabilities/<id>.json carries the same declaration as flat rows, documented by capabilities.schema.json. It answers "does this payer support Coverage search by patient?" from a document this project already retrieves, and the page says where the table is that a declaration is not a test: nothing listed was requested or exercised, and none of it is graded. Four states are kept apart: no document was retrieved on this run, a document arrived and could not be read, a CapabilityStatement declares nothing, and one declares something. The first two publish null rows and a sentence, never an empty table. Pages are split by resource and never truncated. Over the 68 documents the live probes retrieved on 2026-09-10, 51 fit on one page and 17 run to more, 86 pages in all, the largest 49,692 bytes against the 65,536-byte budget. A resource too large for a page by itself is written with counts, and every name stays in the data. The declarations are not in the dated snapshot, because snapshot copies api/endpoint and these live in api/capabilities. What a signed dated release carries is the maintainer's decision.

Each cohort page also links a census of what its listed endpoints declare, one page per category: how many of the endpoints with a readable declaration name each resource, and each interaction on it. The counts are out of that number and never percentages, because a category in one state can have a single readable declaration. Endpoints whose document was not read on the run are named beside the count and never folded into it. An endpoint nobody read has not declared an absence of Patient. Two plans that publish through one server count as one endpoint, and an interaction code outside R4's nine is listed with its count rather than dropped.

Each endpoint page also says what its two documents declare about app-to-server access, the fields SMART Backend Services and Bulk Data run on. It asks whether the token endpoint accepts private_key_jwt, whether the client_credentials grant, the client-confidential-asymmetric capability and any system/ scope are declared, and whether the CapabilityStatement declares an export operation or instantiates the Bulk Data Access guide. None of it is graded. Five answers are kept apart: declared; not listed; not declared because the field is absent; unreadable; and not retrieved. The field-absent answer matters most, because token_endpoint_auth_methods_supported is OPTIONAL, so its absence is not a refusal. The same block is in api/endpoint/<id>.json, in check's terminal report and result JSON, and in the Action's job summary. The published scorecards.json does not carry it.

What the site promises about itself is checked rather than asserted. fhir-scorecard audit-site site/ reads a built directory and reports every page the sitemap omits, every sitemap entry no file answers, every missing or misaddressed canonical, every structured-data block that does not parse or omits a field this site promises, every internal link pointing at a path the build never wrote, every page no path of internal links reaches, and every way a feed breaks its own contract - not listed in the sitemap, not readable as Atom, two entries sharing an id, an entry linking at a path the build did not write, or a page pointing feed autodiscovery at something that is not a feed. The same command also runs twelve mechanical accessibility rules - seven naming the WCAG 2.2 Level A criterion they implement, five saying plainly that they are this project's own rule and not a criterion - and two transfer-size budgets. A fourth family checks the published grades: that every grade is one the grader can produce, that no dimension carries a score for a check no vantage was able to make, that the CSV, the per-endpoint JSON, the API index and scorecards.json publish the same letter for the same endpoint, and that the page a reader opens shows what the data for that endpoint says. It decides nothing about whether a grade is right, which is the grader's business; it decides that what was published is a shape the grader can produce and that every surface agrees. The publish workflow runs all four families before the artifact is uploaded, so a site that fails any of them is not deployed. Its first run against a site carrying an organization page found twelve published, sitemapped /org/ pages that nothing on the site linked to.

The pages are styled with the U.S. Web Design System, vendored at a pinned version and served from the site's own origin rather than a CDN; this is an independent open-source project, not a government website, and the site's own footer says so on every page.

Analytics. The HTML pages use Google Analytics 4 to count visits (ADR 0006); /privacy/ on the site says what it records. The loader in src/fhir_scorecard/analytics.py runs only on fhir.chelseakr.com, never in a local, test or CI build, and loads nothing when the browser sends Global Privacy Control or Do Not Track or the visitor has used the footer's "Opt out of analytics" control. Google signals and ad personalization are off, the advertising consent settings are denied everywhere, and analytics storage is denied by default in the EEA, the UK and Switzerland, where Google still receives cookieless pings. Page addresses are sent without their query string. The compliance bundle pages also record three purchase steps (plans shown, checkout followed, purchase returned), each carrying only a plan and its price, and a hashed order number for the purchase (ADR 0007). The data files, feeds, API tree and badges carry no script. Setting GA4_MEASUREMENT_ID to "" removes GA from every page on the next publish. See ROADMAP.md for what a production public service still needs and, more importantly, for the constraint that governs it: search traffic scales with registry size, and registry size is gated on payers publishing base URLs.

Use the data

A dated copy of the dataset, with a manifest a reader can check with sha256sum:

fhir-scorecard snapshot site/ --out snapshots/2026-08-27 --date 2026-08-27
fhir-scorecard verify-snapshot snapshots/2026-08-27

The snapshot holds the machine-readable files and not the pages, because a page is a rendering that changes when the templates do and the dataset is what somebody would cite. Two builds of one site produce byte-identical output. verify-snapshot fails on a changed byte, a changed length, a deleted file, a file added after the fact, and a manifest that is missing, unreadable, or records nothing. Signing and tagging a snapshot as a release is deliberately not automated: releases here are cut only from an SSH-signed tag verified against .github/allowed_signers, and a path that published an unsigned artifact would skip that control.

What changed between two of those, or between two retrieved documents, is diff:

fhir-scorecard diff yesterday/metadata.json today/metadata.json
fhir-scorecard diff yesterday/scorecards.json today/scorecards.json --format json

It reads what it is given, requests nothing, and stores nothing. Each side may be a retrieved CapabilityStatement, a SMART discovery document, a scorecards.json, or a probe artifact; what they are is decided from their content rather than their filename. A removed interaction reads as Coverage search-type: interaction no longer declared, with the R4 clause beside it, rather than as a fingerprint that moved. The same inputs always produce the same words.

Three things it will not say. A document that could not be read diffs as unreadable against anything, with no field-level claims at all, because an unreadable response is not a server withdrawing what it is still serving. A key absent on both sides is never mentioned. And two artifacts recorded from different vantages are reported as exactly that, never as a change the endpoint made, because reconciling vantages is vantage.reconcile's rule and subtraction is not a substitute for it. The fingerprint half of the comparison is the same function the declaration timeline records with, so the timeline and the verb cannot disagree.

--fail-on-regression exits 1 when the later side no longer has something the earlier side had: a resource, an interaction, a declared profile, or a check that used to pass. It is opt-in because it is an operator's policy rather than this tool's judgment, and it is deliberately narrow. An addition never trips it. Neither does a dimension that stopped publishing a score, which is reported as not comparable rather than as a fall, because treating a lost measurement as a fall would score an absence. Nor does a document this run could not read.

A read-only MCP server exposes the same data to an assistant:

fhir-scorecard mcp --site site

It reads only the published dataset files. There is deliberately no tool that probes an endpoint: a model deciding to fetch arbitrary URLs is a much larger security surface than one reading a file this project already publishes. Its grading_method tool returns the documented limits, so an assistant can be told what the numbers do not mean. Its cited_passages tool returns, for one endpoint, each finding with the verbatim passages of the specification page it cites, quoted from the copies retained under corpus/; no model is called, and an assistant that explains a grade can quote the specification rather than recall it. Its declared_capabilities tool returns one endpoint's published declaration (#102) exactly as api/capabilities/<id>.json serves it, including the state that says when there was no readable declaration at all.

The one command that does call a model is narrate (ADR 0003):

uv sync --all-extras                                     # adds the `ai` extra: the public anthropic SDK
FHIR_AI_PROVIDER=bedrock fhir-scorecard narrate --endpoint cms-blue-button-2 --language es

FHIR_AI_MODEL overrides the model; each provider's default is one that provider can actually be asked for, and they are deliberately not the same. The Anthropic API defaults to claude-sonnet-5. Bedrock defaults to global.anthropic.claude-sonnet-4-6, because the AWS account this project's evals run on is not entitled to Sonnet 5 there - InvokeModel returns AccessDeniedException while the entitlement API reports the model AUTHORIZED, so entitlement can only be established by invoking. The recorded runs in evals/ai/results/ are all on the Bedrock default.

It explains one published scorecard in plain language. The grade and findings are inputs it cannot change; every sentence it prints quotes a passage of the cited page that was verified against the retained copy, and a sentence whose quote does not verify is withheld and counted. The output is labeled AI-generated, describes what the endpoint published rather than the organization, and is not on the site. A record that offers no passage a claim could cite (no graded dimensions, no findings, or findings whose cited page is not retained) is refused before the model is called: the result is status: not_narrated with the reason, model_called: false, and zero tokens, rather than a narration whose every claim was withheld. python -m fhir_scorecard.ai.eval measures the grounding rate; the recorded runs are in evals/ai/results/, and refused records are counted there as records_not_narrated, outside the grounding fractions.

Status

v0.3.0, the version this tree declares and the one a consumer pins. Changes merged after a tag are collected under Unreleased in CHANGELOG.md; what a version names is the composite GitHub Action and the distribution behind it, not the daily site rebuild. Eighty-eight endpoints across payers, payer provider directories, a federal provider API, EHR vendor sandboxes, and reference servers, in twenty-one curated cohorts (California payers and the marketplace issuers of Texas, Florida, Ohio, Wisconsin, Arizona, Michigan, Missouri, Oklahoma, Iowa, Kansas, Louisiana, North Carolina, Nebraska, Alabama, Alaska, Wyoming, West Virginia, Delaware, Indiana and Mississippi). Seventy-six were verified from a retrieved conformance document; twelve are listed on the organization's own publication of a base URL that does not answer, which is a finding about the public record rather than a gap in this one. Registry curation continues one roster at a time - see docs/SAMPLING-FRAME.md - because payer base URLs are not predictable from company names. Grades are observational snapshots of public surfaces, not audits, rankings of care quality, or statements about any organization's compliance.

Provenance

Personal open-source project, built on personal time and equipment, unaffiliated with any employer or client, past or present. Built with AI assistance (Claude Code); every change passes the make verify gate (ruff lint and format checks with security rules, mypy strict, pytest with a branch-coverage floor, and a pip-audit of the locked dependency set).

License: Apache-2.0.

Standards Conformance

Per the portfolio standards set. N/A rows are backed by a committed declaration or ADR; there are no blank rows and no silent skips.

Support

This is independent, unpaid work. If it has been useful to you, you can