ration
A ration is a bounded claim on a finite shared pool, granted for a period, and refused rather than overdrawn.
A coordination substrate, and the lease, pool, addressing and publication services built on it, with the conformance suite that decides whether a store can carry them.
What is here: the contract in src/ration/contract.py; a suite that
discriminates between a store that keeps it and stores that are wrong on
purpose, one per property; a Postgres backend that passes every property, and
a naive Postgres one that is rejected; src/ration/lease.py, which grants one
holder per key; src/ration/pool.py, which grants a bounded number of holders
over a set of keys; src/ration/addressing.py, which answers where a role is
by reading the lease that says who holds it; src/ration/publish.py, which
advances a versioned pointer to published state under the same
compare-and-set; and src/ration/frame.py, the checked carrier beneath claims
of authority and new payloads between nodes, with its own suite of frames built
wrong on purpose. Treat every statement below as intent unless it points at a
passing test.
Three of the services, built after the suite, each found something every green
run before it had agreed with: the lease, the pool and the publication pointer.
The lease found a defect in the contract's own remedy for an
ambiguous write. The pool found that a retrying holder takes a second slot
unless the scan looks for its own grant first, and that the obvious test for it
cannot fail, which only a deliberately wrong implementation revealed. See
What the first consumer found and src/ration/pool.py.
What this offers, and what it does not
The mechanism, and the suite that judges it. Nothing here is a guarantee,
and nothing is supported beyond what SECURITY.md says. The suite is the
contract: where this README and the suite disagree, the suite is right and this
file is the defect.
The suite is evidence about ration's own code and about the stores it was run
against, and about nothing else. A store's guarantee is shown by running the
suite against that store; if you put ration in front of a store of your own,
that run is yours to make. A store's own vendor test suite is evidence about
the vendor, not about your deployment, and the same is true of this one.
Running the suite against your own store, at the end of this file, is how to
make that run.
Postgres passing is not Postgres being chosen. The contract makes the store
a conformance result, and a conformance result is evidence, not a decision.
Nothing in pyproject.toml's runtime dependencies names a driver, and in this
repository ration.postgres is imported only by the suite. Which store a
deployment uses is that deployment's decision.
What this is
Two things, in one repository because they own the same state:
- The coordination substrate: an atomic compare-and-set against a monotonically increasing version, on a single key, plus the backends that implement it and the suite that decides whether a backend qualifies.
- The services built on exactly that primitive:
lease.py, a bounded, expiring claim on one key;pool.py, a bounded number of simultaneous claims over a set of them;addressing.py, which answers where a role is by reading the lease that says who holds it; andpublish.py, which advances a generation pointer to published state. They are one thing rather than four: a pool is N keys, each one of these leases; an address is a lease's payload; and a publication is the same key discipline with the expiry refused. None of them adds an operation of its own.
And a third that owns no state at all: frame.py, the carrier a claim
of authority or a new payload rides in between nodes, checked before the
payload is looked at. It is here because it leans on the contract exactly once,
for the epoch a restarted sender needs, and a lease version already is one. See
The frame, and the suite of frames built wrong on purpose.
The repository is decomposed by state ownership: the substrate and the
services share it because they own the same state. The contract they all rest
on is stated once, in src/ration/contract.py, and decided by
src/ration/conformance/:
| Property | What it fixes |
|---|---|
| The operation | compare_and_set on a single key applies only at the expected version, the comparison and the write are one step, and a conflict is reported without writing |
| 1 | The version is monotonic per key and never reused, not even after the key is deleted |
| 2 | Expiry is evaluated by the store, when it is asked; a TTL is a duration, never a caller's deadline |
| 3 | A caller that cannot reach the store has acquired nothing: unreachable means refuse |
| 4 | A write whose answer was lost is resolvable by reading, because the caller's token is in the store |
Addressing a role, and why it is not a name record
A role must not be pinned to a host by accident, and a caller that reaches a service by hostname has done exactly that. So services address each other by role, and something resolves the role to a host.
That something introduces no new state, which is why it is a module here rather than a service of its own. Who holds this role right now is already lease state. The address rides as the lease's own payload, so a role is announced and held in one operation and there is nothing to keep in step.
announce(store, "producer", "node-a.example:8080", holder="node-a", ttl=30.0)
resolve(store, "producer")
# Address(role='producer', endpoint='node-a.example:8080',
# holder='node-a', version=7)
The alternative was to publish the address into a name service and let callers resolve by DNS. It loses on a lesson that is not new. The lease already decides who holds a role, so a name record is a second copy of that fact, and it goes stale in exactly the moment that matters: a failover. Two copies of a fact is one fact and one stale claim.
The concrete form of the same objection is the version field above. It is
the lease generation the answer was read at, so a caller can be refused by a
callee that has seen a newer one. A cached name has no generation, so a
stale holder keeps being reachable and nobody can tell. An address without a
generation is a guess with good manners.
The network still carries the packets. It is the transport; it is not the register of who holds what, and those are different jobs.
What it costs. Service addressing now depends on the coordination store being reachable. A store that refuses under partition already meant a peer could take no lease while it was unreachable; it now also means the peer can resolve no role. That widens the blast radius of a failure a deployment has already accepted rather than introducing a new one, which is a smaller claim than "no new risk" and is the honest one.
ttl is required here where the lease layer allows None. A claim with no
expiry keeps answering for a holder that stopped existing, which is the
stale-name failure arriving through the door this module was built to close.
The expiry assertions are backed by a control that reproduces the defect.
Most of tests/test_addressing.py asserts an address stops answering, and a
test asserting an absence can pass while looking at nothing. So a deliberately
caching resolver is required to keep answering for an expired role; if it
ever stops, the tests above it are measuring the passage of time rather than
the store.
Publishing a generation pointer, and why it is not a lease
A producer publishes an artefact, consumers act on it, and a reader needs to
tell a stale artefact from a current one. publish.py gives the pointer a
generation: a compare-and-set on a key holding a generation and a pointer,
where the generation is the store's own version, the same number
addressing.py carries as Address.version.
made = publish(store, "config", "sha256:9f2c...", "producer", generation=UNPUBLISHED)
current(store, "config")
# Publication(topic='config', publisher='producer',
# generation=1, artefact='sha256:9f2c...')
Only the pointer lives here, because the artefact half has nothing to coordinate. Once published bytes are named by content they are immutable, and immutable data needs no lock: lockfiles and atomic swaps exist only for an artefact that mutates in place.
The differences from a lease are the design, and both run in the refused
direction. A publication takes no TTL, structurally: there is no
parameter to pass, where announce requires one. An address is a claim
about a live process, a publication is a fact about content, and an expiring
pointer would vanish without a successor and read as a topic never published.
And there is no retract: a publication is superseded, never deleted, which
also keeps dormant the resolve-after-deletion edge lease.py records as "a
real edge for any consumer that does need the distinction." The test for
non-expiry calibrates its own clock:
a leased address under the same store must expire across the same advance, so
the publication standing through it is a decision rather than a default.
What building it found: recovery from an ambiguous advance needs two checks,
and each has a case the other cannot decide. A publisher is its own
predecessor as the routine case, since every advance rewrites a key already
holding its token, and it rewrites with different payloads, which the lease
never did. So resolve's APPLIED is no longer enough to hand back a handle: a
publisher identity racing itself (two replicas, or a retry racing its original)
can have its lost write attributed to the write that survived, and only the
payload distinguishes them. The other case: a recovery built purely on the
follow-up read, token and artefact both matching, decides one case wrongly, an
ambiguous republish of the standing artefact, where nothing in the store
distinguishes the lost write and only the unmoved version says it never
landed. Both cases are pinned in tests/test_publish.py. The contract is
untouched by both: the information was in the store the whole time.
Consumers poll. The contract offers no watch, so "has the pointer moved?" is a read and a comparison, repeated. Staleness is detectable, not pushed; a cache added to hide the cost would be the defect the addressing section above documents refusing.
The limit, stated before it is needed: a pointer that needs two keys to move together, an artefact and its index published atomically, needs a guarantee the contract deliberately refuses. That would be a change to the contract, not a quiet widening here.
The frame, and the suite of frames built wrong on purpose
The rule the frame serves: a payload that carries a claim of authority from one
node to another (a fencing claim, a vote, a command one node sends another)
rides inside a frame, and so does every new payload between nodes; a payload
already framed stays framed. The rule says where a payload belongs, not where
every one is today: a consumer moves a route onto the frame in its own change,
so a claim can still cross outside a frame while that move is under way. The
frame is checked before any payload is decoded, and this repository owns the
frame. src/ration/frame.py is that and nothing beyond it: no route, no
transport, no consumer.
Two kinds of existing payload are outside the rule on purpose. A record in
the coordination store can carry a claim, since its version is a fencing
token, but it crosses between nodes through the store rather than as a
message, and property 1 already orders it (src/ration/conformance/test_version.py);
a frame's sequence beside it would be a second ordering of one record. The
telemetry an existing shipper sends carries no claim, and the frame's
sequence rule would break it: a receiver accepts exactly the next count
(src/ration/conformance/test_frame_sequence.py), so one lost batch would leave
every later batch refused as a gap until the sender started a new epoch or the
receiver a fresh ledger.
A frame is an eighteen-byte big-endian header (the frame's own version, the payload kind, a 64-bit per-sender sequence, the body length, and a CRC-32 over the header and the body) around a body the module never looks inside. The checks run in a stated order, and the order is the contract a v2 leans on: a frame of a version this reader does not speak refuses as exactly that, before anything assumes v1's layout. The layout table and the order are in the module docstring, once.
A frame can also travel as text, in a request header, for a route that
takes no HTTP body. The frame inside is an unchanged v1 frame, and its seven
checks run on its bytes as they would on a body's; only the carrier differs.
The text form is base64url (RFC 4648, section 5) with no padding, so a frame
of n bytes is ceil(4n / 3) characters and the header-only frame is 24. The
decode is strict, and strict means canonical: a frame has exactly one text,
the one encode_text writes, and any other string is refused as
malformed-text, the ninth control, before any frame check runs. Which header
carries the text is the consumer's transport choice and is named on its side,
not here.
The strictness is the property, because the obvious decoders are not strict.
Python's default base64.b64decode drops a character outside its alphabet and
returns the bytes the text held without it. Its strict base64url setting
(validate=True with base64url's altchars) still takes padding and the
standard alphabet's + and /. And every setting, binascii.a2b_base64 with
strict_mode=True included, reads a last character whose unused bits are set
as the same bytes as the canonical one, so two texts decode to one frame; the
text property asserts that reading before it relies on it. The reader
therefore checks every character against base64url, decodes, re-encodes, and
refuses any text that does not come back as itself.
The sequence is the check with a live failure mode, because HTTP frames one
request and neither orders nor de-duplicates requests from a sender: a retried
POST is a duplicate the transport delivers happily. The rule for a sender
that restarts is written into the sequence itself. The high word is the
sender's epoch, the low word its count within it, and a receiver accepts
exactly the next count or a higher epoch at count zero. A lease version is an
epoch already: property 1 makes it monotonic and never reused, so a sender
that re-announces its role after a restart holds an epoch strictly above any it
held before, without persisting anything. An older epoch is refused as
replayed, which is the stalled holder a fencing token exists to refuse,
arriving on the wire. And the ledger moves on acceptance only, so the retry
of a frame that was refused carries the same sequence and is taken for the
first time, while the retry of a frame that was accepted is refused as the
duplicate it is. The two retries are byte-identical; the ledger is what tells
them apart.
The suite is the price of the word checked: every hostile case has a known-negative that fails without the check beside a known-positive that passes with it. The known-negative is not a bad frame; it is a reader without the check, one per check, each the shape a shortcut takes:
| Frame property | What it fixes | Hostile cases | The codec that must fail it |
|---|---|---|---|
| 1 | The bytes present honour the declared length, exactly | truncated mid-header, truncated mid-body, length larger than present, length smaller than present | unchecked-length - trusts the transport, so a short header is an exception and a short body is misnamed; ignores-trailing - takes the declared body and discards the rest |
| 2 | A flipped bit anywhere is refused as corruption | a bit flipped in the body, in the sequence, in the epoch, in the kind byte | unchecked-crc - carries the CRC and never compares it; body-only-crc - a CRC over the body alone, which leaves the header bare |
| 3 | An unknown version is refused as that, before its layout is assumed | version ahead of the reader, version ahead with the CRC moved, version zero | unchecked-version; crc-before-version - every check present, only the order wrong |
| 4 | An unregistered kind is refused | kind unknown to the registry, kind zero | unchecked-kind |
| 5 | Exactly the next sequence, or a fresh epoch at zero | duplicated, skipped, behind, restart at a fresh epoch, an older epoch replayed, a new epoch not at zero, retry of a refused frame | unchecked-sequence; counter-only - 32 bits and no restart rule, so it turns the honest restart away; sequences-before-checking - the right rule, applied before the CRC has spoken |
| 6 | Only a frame's canonical text is read, and any other string is refused as text | a character outside the alphabet, whitespace, a character outside ASCII, padding, the standard alphabet's + and /, a length one past a multiple of four, a second text of the same frame at every unused bit of both length classes; and a well-formed text of bad bytes, refused by the frame rather than the text |
lenient-text - the standard library's default decode, which drops what it does not recognise; library-strict - its strict base64url setting, which still takes padding and both alphabets; skips-text-decode - hands the header to the frame reader undecoded; accepts-non-canonical - every character checked against base64url and the round trip skipped; drops-dangling-character - reads a dangling character as nothing, as Node's Buffer does; collapses-layers - answers the frame's refusals as the text's |
The rows are Frame property 1 to 6, in this order, and that is the
number each module in src/ration/conformance/test_frame_*.py cites. It is not
the order in which src/ration/frame.py runs its seven checks, which is a
different list and says so.
One hostile case is separated by no registered reader: a character outside
ASCII, which every reader here refuses as text because the standard library
raises on it. A reader that normalised the text to NFKC first would fail it;
none is registered, and src/ration/conformance/test_frame_text.py says why.
Same mechanism as the store suite, not a copy of it: one marker, one strict
xfail, one set of bookkeeping assertions in
tests/test_the_suite_discriminates.py, parametrised over two registries. A
run reports one xfailed line per codec per property it drops, which is one per
class in src/ration/conformance/frame_violators.py
(grep -c '^class .*(FrameCodec)' src/ration/conformance/frame_violators.py); the run is the
count, not this sentence.
What building it found. The first run of the suite had
sequences-before-checking failing the four properties it does not claim,
beside the one it does. Every property test ends each refusal with the healthy
frame accepted, and a reader that consumes a sequence on refusal breaks the
retry after every refusal, so it failed length, integrity, version and kind
for a defect that is entirely about the ledger. A violator that fails
everything proves nothing about which check caught it. The fix is in
src/ration/conformance/support/codecs.py: every property that is not about sequence is
decided against a ledger that has seen nothing, so a sequence refusal cannot
masquerade as theirs, and the one property that is about the ledger names its
sender on every decode. The same run had unchecked-crc red on the
sequence property, because that test's "refused frame" was refused by the
CRC; it is now a truncation, which every reader refuses whichever check it is
missing, so the only thing decided there is what a refusal does to the ledger.
Both are the same lesson the store suite records about conflict-writes,
arriving from the other direction: the property, not the assertion, is the
unit, and a property test must hold every other property still.
The text property's first run taught it a third time. Its arm for a fault in
the bytes under a well-formed text asserted crc-mismatch, so unchecked-crc
and body-only-crc failed a text property for a CRC defect. That arm now
asserts only that the refusal is a frame control rather than the text's, and
tests/test_frame.py pins the exact control on the reference reader. The
second run had body-only-crc red on the standard-alphabet arm: the frame's
only - had been in the CRC's characters, which that reader writes
differently, so the arm went vacuous for it and its own guard said so. The
frame's epoch is now chosen to put a _ in the sequence field, in bytes every
reader writes alike.
Three costs stated rather than hidden. A ledger that has seen nothing takes
the first frame as its baseline, so a replay against a fresh receiver is not
caught; the alternative would turn every receiver restart into a forced restart
of every sender. The header carries no sender field: which sender a frame
belongs to is the receiver's knowledge, and a sender identity in the header
would be a claim the frame cannot verify. And the kind registry is closed and
short: five kinds, a consumer's fingerprint envelope by the envelope's own
name, one per body-carrying route of a consumer's service, and role-claim.
Their codes are on the wire and their names label them, so they do not change.
role-claim is a claim-only body, the fencing claim a sender presents and
nothing beside it: the kind a route with no HTTP body takes, its frame carried
as text. It is one code for every such route rather than one per route. A code
means one body shape and this is one shape, and the frame carries a CRC, not a
MAC, so anyone who can build a frame can build it with any registered kind: a
code per route would catch only a sender wired to the wrong route. The claim
rides in the body for the same reason the header carries no sender: the
header carries only fields the frame itself checks, and a claim there would be
one the frame cannot verify. As body bytes it is opaque to the frame like any
body, and the CRC and the sequence say of it only what they say of every
frame, that it arrived as sent, once; whether the sender may present it is the
consumer's to judge.
A sixth kind lands here before anything sends it. A code nobody sends is inert,
while a decoder selecting on a code this registry does not name has every
framed request refused as unknown-payload-kind; so the row comes first, and a
consumer that installs this package at a commit sees it when it moves to that
commit. src/ration/frame.py carries the argument beside the registry.
The name
A ration is a bounded portion of a finite pool, issued for a period. The word already carries the property the system needs most: rationing over-refuses rather than overdraws. The error direction is fail-closed, and here it is in the name rather than in a comment.
The first piece of work, and why it is not the service
The conformance suite comes before any backend, and before the lease service. The contract's central claim is unproven by construction: that a backend can be held to the contract, and that a suite can reject one.
So the suite's job is not to validate a backend. It is to discriminate between backends, and the design follows from that: a deliberately violating implementation per property, each breaking as little as it can:
| Property | The implementation that must fail it |
|---|---|
| Compare-and-set applies only at the expected version | create-always - absence is not a version, so creation is an upsert |
| ...and a conflict does not write | conflict-writes - reports the conflict honestly, writes anyway |
| ...and a conflict carries a version the key has held | echoes-expected - hands the caller's expected version back, which the key may never have held |
| The comparison and the write are one step | naive - read-then-write, exactly what the operation must not be |
| Version monotonic, never reused, not even after deletion | recycling - drops the key's counter when the value goes |
| ...and per key, not per store | global-counter - monotonic, never reused, and a statement about unrelated traffic |
| Expiry evaluated by the store | client-expiry - asks the caller what time it is, so skew is a lease held twice |
| ...when it is asked, not when it last looked | stale-clock - the store's own clock, fetched and then acted on later |
| Unreachable means refuse | fail-open - a layer in front of a correct store that treats silence as "nothing there" |
| An ambiguous write is resolvable by reading | token-dropping - keeps the payload, substitutes the token |
| ...and all of the above, in a store that is not a fake | postgres-naive - SELECT then UPDATE, exactly as the operation reads in English |
If a violating implementation passes, that property is not being tested.
The suite states that as a mechanism rather than a hope: a violator is marked
xfail(strict=True) against its own property, so a violator that passes is a
failing run. Same standard as the guards under scripts/, each of which
runs its own self-test before it runs in CI: proof it can fire and proof it can
go quiet.
Two of those rows are one implementation short of the ideal, and it is worth
knowing which. conflict-writes is declared against two properties rather
than one, because a store that writes when it should not destroys the winner's
value in any race it loses: there is no interleaving in which it is atomic. A
violator that broke only the reply would be better and does not exist.
The version property is still short its most likely violator. Restoring a store from a backup can replay or invalidate lease versions, so a restore is not an ordinary database restore. Both violators above are one process misbehaving; reuse-after-restore is an ordinary recovery, run by someone who believes they are restoring a database. It is harder to reach, a lifecycle event between store instances rather than a defect inside one, so no in-process implementation expresses it, and the suite does not cover it. Stated here so the gap is a known absence rather than a discovery made during a recovery.
A claim the suite did not cover was false for as long as nobody looked.
Conflict and the Postgres compare_and_set both stated that a conflict
reports the version the key had moved to. It does not. Under concurrency
PostgresStore reports the version the caller came from: two writers from
version 1 leave the loser holding Conflict(1) while the key is at 2; two
creators leave the loser holding Conflict(0) while the key is at 1. The
statement that decides the outcome and the statement that reports the version
share one snapshot, and only the UPDATE half gets EvalPlanQual.
Not a contract defect. The contract requires a conflict to be reported
without writing, which holds, and asks of the version it carries only that
the key has held it, which holds too.
src/ration/conformance/test_compare_and_set.py decides both, accepting any version
the key has held, and echoes-expected is the store it rejects for reporting
one the key never held. The acceptance has a store of its own.
previous-version (src/ration/conformance/permitted.py) is registered with no violations and,
once a key has been written, answers every refusal with the version the key
held before its current one. Every other correct store answers a refusal that
races nothing with the current version. So a test tightened back to the
current version fails previous-version and no other registered store, and
loosening the assertion is a change the suite can see being undone. To watch
it, replace the assertion after the stale-expectation refusal with
assert refused == Conflict(created.version) and run
pytest -rA src/ration/conformance/test_compare_and_set.py. Making a conflict's
version exact costs either a second round trip or an
ON CONFLICT DO UPDATE that writes in order to read, which is precisely
what conflict-writes exists to fail. So the claims were corrected and the
behaviour pinned by tests/test_postgres_conflict_version.py,
deterministically, with a held row lock rather than a race. A caller needing
the current version must read.
The lesson is about the suite, not the store. Two docstrings agreed with each other and disagreed with the server, and every test passed throughout, because nothing asserted in that area at all. A property the suite does not decide is a property the prose is free to be wrong about, and prose is what a reader trusts.
The hard part is concurrency. A lost update only appears under concurrent
access, and a concurrency test that has never been run against an
implementation known to lose updates can run green while not looking. So the
instrument here is calibrated against a known-bad implementation before it is
trusted. The calibration is what turned "the hammer passes" from a result into
a question: at the interpreter's default switch interval the probe ran green
against naive, the one implementation that must never pass, at the rate
recorded under The first calibration below. A suite that had merely run
it would have shipped that green as evidence.
Threads or a deterministic interleaving? Both, doing different jobs. The
interleaving is the gate: it parks one operation at the last moment before its
atomic step, runs another to completion, and, the part that matters,
asserts that the interleaving happened. It is needed by two properties rather
than one, and the second is stale-clock: a store that reads its own clock and
then evaluates expiry somewhere else defeats clock injection entirely, because
the reading it used was never the caller's.
The seam sits outside every lock, and that placement is the reason the stepper is deterministic rather than lucky. A parked operation holds nothing, so the second one cannot be blocked by it and is waited for rather than given a window and assumed. Giving it a fixed fraction of a second and carrying on either way would reintroduce the failure this section argues against: a run that missed its own interleaving, reporting green.
A hammer can tell that it raced, and not whether it raced inside the
window. A writer refused because another moved the version between its read
and its write was refused by a genuine overlap, so the threaded test counts
refusals and reports "the probe never raced" instead of a verdict when there
were none. What it cannot do is land inside the read-to-write window on
demand: a violating implementation that is never made to race inside it has
not been shown to survive one. So threads are a probe, not a gate: they
reach non-atomicity inside a real store that no client-side stepper can, and
they carry a false-pass rate the suite states rather than implies. The order of
the two checks is in src/ration/conformance/test_atomicity.py's docstring.
The first calibration is the finding worth keeping. On the interpreter's
default switch interval, the hammer never once caught the naive backend: it
ran green against the implementation that must never pass. The window between a
naive read and its write is about a microsecond and the interpreter was
switching every five milliseconds, so the probe was not unlucky; it was not
looking. Forcing preemption to a microsecond, the same counts caught it every
run. The thread count, the iteration count and both rates live in
src/ration/conformance/test_atomicity.py, beside the probe they describe, and are
deliberately not repeated here. That rate is a measurement, not a
guarantee: no false pass was observed, which is not the same as none being
possible, and it is precisely why the property does not rest on it. The naive
backend stays in the hammer run: if the hammer ever passes it, the hammer is
measuring nothing and fails on that basis.
Neither of these is written into the contract as a requirement on future
backends. A seam invented before any backend exists would be a contract clause
with no caller, and it does not need to be a clause, because AGENTS.md
already covers the case: a backend that needs a special case to pass is a
finding about the backend or the contract, not a fixture to be written.
Every service is one compare-and-set
- The lease:
acquire,renewandrelease, each of them one compare-and-set, and no new method onStore. A contract is tested by what its first real consumer needs that it lacks, and the answer was nothing: the refusals held, at a cost of one extra read on the ambiguous path. - The pool, which is what the name promises. The contract forced the design
rather than leaving it open: expiry is evaluated by the store, and a store
expires keys rather than elements inside a value, so a pool of N grants on
one key would have its deadlines evaluated by whoever read it, which is
client-expiry, a violator the suite already rejects. So a pool is N keys, each one of these leases, and capacity is the number of keys rather than a number written down anywhere. The evidence is whatpool.pydoes not contain: norenewand norelease, because a pool grant is a lease. - The generation pointer:
publish.py, versioned published state as a compare-and-set on one key, the same primitive as a lease rather than a second one. - The frame adds no key and no operation, and leans on the contract exactly once, for the epoch a restarted sender needs.
What the real backend cost, and what it found
Two things came out of running the suite against something that is not a fake, and both were about the suite rather than about Postgres:
- A fake clock is not a clock, and an in-process seam does not cross a
socket. The harness had been written when only fakes existed. Time is now
passed by
advance_store_time(), moving a fake clock or actually waiting, and the deterministic stepper skips where a store's atomic step is one SQL statement. That skip removes an instrument, never a verdict:postgres-naivehas no seam either and is still expected to fail, on the threaded probe. - A store that passed every property still could not survive being asked.
The first backend opened a connection per operation, which is the obvious
shape and reads as the careful one. The threaded probe put about 1,600
connections through it, and the next run failed on
Address already in use: every one of them held an ephemeral port inTIME_WAIT. Nothing in the contract catches that; there is no property about exhausting the machine. It was found by running the real thing under real load, which is the whole argument for deciding a backend by conformance result rather than by comparison.PostgresStorereuses one connection per thread.
What a second real backend found, and why it does not qualify
duckdb is registered as a case alongside postgres, with an empty violates
set. It passes every conformance test. That is the result, and on its own it
would qualify the backend.
It cannot be a coordination store for more than one process, and the suite is
structurally unable to say so. DuckDB is embedded: the database is a file and
one process owns it. A second process cannot open it. That is measured from a
real interpreter under spawn and asserted, not described, in
tests/test_duckdb_single_process.py, with a control proving the same child
succeeds once the holder lets go. A coordination store exists so that other
processes, and other hosts, can reach it, and there is nothing here to reach. It
does not fail at the second host. It fails at the second process, which is
where a pool's contenders live.
This is not a defect the contract can name, and the contract should not grow
one. Every violator in src/ration/conformance/violators.py is wrong logically: it computes
an answer the contract forbids, in one process, where the suite is looking.
DuckDB is wrong about deployment topology, which the contract deliberately
does not govern. Adding a property for it would be inventing a clause with no
caller. So the suite's verdict on DuckDB is correct and incomplete, and the
single-process test is the part that completes it.
What it was actually worth: it found a hole in the suite, and a consumer found
it first. DuckDB's creation path raised Duplicate key ... violates primary key constraint under load, from tests/test_lease.py, after the conformance
run had pronounced the same path correct. The cause was in the suite's own
instruments: every concurrent test seeded its key single-threaded and then
raced from that version, so the suite had only ever raced updaters. The
INSERT branch had never been entered by two callers at once.
test_two_creators_racing_from_absence_do_not_both_apply closes it, and the
control is the usual one: with the defect reintroduced the new test goes red
while the entire pre-existing suite passes green.
The second-order lesson is about probes, not about DuckDB. The first draft
of that test was threaded only. It caught naive every run in isolation and then
let it through inside the full suite, because a creation can be raced only once
per key and a loaded machine serialises the racers. A single shot per round is
not something a strict xfail can stand on, so the test uses the deterministic
stepper wherever a seam exists and the probe only where none does, the split
the rest of the atomicity file already uses.
What the first consumer found
Both findings are the same lesson as the Conflict correction above,
arriving from a new direction: a property nothing asserts on is a property the
code is free to be wrong about, and the prose will agree with it the whole time.
resolve decided by token before version, and that is wrong for a caller
that is its own predecessor. Property 4's remedy recognises the caller's own
write by finding its token in the store. A renewing lease already has its token
there, from the grant before, so a renew whose answer was lost resolved as
APPLIED on the strength of the earlier write, and the holder was told it had
a TTL the store had never been given. Fixed by testing the version first, which
property 1 makes decisive: every write advances the version, so a key still
standing at the expected one has not been written by anybody, and no token can
argue otherwise.
- The suite as it then stood could not have caught it.
test_ambiguity.pycreated and never rewrote, and the token stops identifying a write only once a caller writes the same key twice. Nothing did until the lease did;test_a_rewriters_lost_answer_is_resolvednow reaches that case. - The contract is untouched. The information needed was in the store the whole time and the procedure was reading it in the wrong order. This is a defect in the remedy, not in the contract.
- Calibrated before it was trusted: the regression test was run against the old ordering and fails there, returning a lease at the unchanged version. A test that has only ever been seen to pass has not been shown to test anything.
The exclusivity check was green while never racing, and the negative
control is the only reason that is known. Starting eight threads in a loop is
not a race: an in-memory acquire finishes in microseconds, so the first claimant
was done before the last was started. Exclusivity passed, on a window it never
entered. The control, the same contention against naive, which must
double-grant, failed, which is what a control failing is for.
It then took two more shapes to get right, and the second failure is the instructive one:
- A
threading.Barrierfixed it on a machine with many cores and failed on a smaller one. A barrier synchronises arrival, and arrival is not overlap: on a smaller machine the winner finishes its whole compare-and-set before the loser is scheduled, so sampling has to catch the overlap in the instant it exists. - Counting a violation per operation rather than sampling a moment does
work, catching
naivein roughly half the rounds.
So it was replaced by a deterministic gate anyway, and not because it
failed. naive's seam sits exactly in its gap, so one claimant can be
parked there while another runs to completion. Where a store has a seam, a
gate beats any per-round rate: a probe that catches a known-bad store half
the time is evidence about the probe; a parked thread is a verdict. The
threaded probe keeps its place because postgres has no seam, and there it
is the only instrument there is. The gate is proven both ways: the identical
interleaving against the correct store grants one holder.
Threads or processes, answered with a number instead of a paragraph
The threaded probe is eight threads in one interpreter, and a lease service's real population is processes, often on more than one host. Whether that difference matters is a question the suite answers rather than assumes.
tests/test_postgres_cross_process.py is the same arithmetic, eight contenders
with one hundred increments each, run as eight operating-system processes
under spawn, each building its own store from nothing but a DSN.
For detection it makes no difference, and that is a measurement. Both
populations were pointed at postgres-naive, which has no version predicate in
its write and must lose increments. One set of runs on a development machine:
| Population | Increments lost of 800 | Runs |
|---|---|---|
| 8 threads, one interpreter | 673, 687, 689 | 3 |
8 processes, spawn |
662, 668, 669, 671, 673, 686 | 6 |
The ranges overlap. Against the real store, postgres returned 800 of 800 in
every run of both. The thread count was never the interesting variable
here: every operation is a round trip, so the writers already interleaved at
the socket without the interpreter needing to be persuaded to switch, which is
the reason the threaded hammer catches this store so much more reliably than it
catches the in-memory naive.
What the process version does buy is structural, and it is not the
contention. The threaded hammer shares one PostgresStore instance across
its eight workers; that instance holds a threading.local() connection
registry, so the workers are eight clients of one store object. A deployment is
not that. Any defect living in state accidentally shared inside a store
instance is invisible where there is only one instance to share it, and this
probe has one store per process, so it cannot inherit that blindness. The claim
is narrow on purpose: it removes a caveat, it does not raise the catch
rate.
The harness is calibrated before either verdict is read, because a spawn
pool that quietly ran every task in the parent would produce a perfect counter
and prove nothing: it would be the threaded result wearing this test's name.
Each worker reports its PID and the test asserts it saw eight distinct ones.
Still a probe. It has no seam to park in either, so a run that finds nothing
has not shown there was nothing to find. What keeps it honest is that the
identical harness is required to catch postgres-naive, as a plain
assertion rather than an xfail, because there the subject is the harness.
A refusal that arrives late
PostgresStore reuses a thread's connection while conn.closed is false, and a
connection the server killed still reads closed == False until the client next
talks to it. A store restart therefore cost each idle reader one refusal
whenever it next called, which can be hours later, reported as a store
failure about a store that had been back the whole time.
read()redials once when the statement fails on a reused connection. A store that is really down still refuses, because the redial is a real connect: property 3 is not traded for it.compare_and_set()never does, including on a connection that was dead before the statement left. The client cannot tell that from a write that applied and lost its answer, and repeating the second is a double write, which property 4 exists to prevent. The caller resolves it by reading, and the read is what redials.- Pinned by the event itself.
tests/test_postgres_killed_connection.pykills the store's backend withpg_terminate_backendagainst the server CI already runs, so the sequence is replayed on every pull request. Its fakes cover only what no server can stage. - And every one of its tests runs against four stores built wrong on purpose, under the same strict xfail the contract's violators get: the store as it stood before, one that redials until it works, one that forgets which connections it opened itself, and one that gives a write the read's courtesy. A test that stops rejecting is a failure on the next run.
Installing
The distribution is catincloud-ration, and it imports as ration.
.github/workflows/publish.yml uploads each release to PyPI from v0.2.1 on.
An earlier release installs from a clone at its tag, as under Versions and
releases. From PyPI:
pip install catincloud-ration
pip install "catincloud-ration[conformance]"
The first installs the services and stores alone. The second adds pytest, the
one dependency of the suite the package ships as ration.conformance (Running
the suite against your own store, below).
pip install ration installs a different project. The bare name on PyPI
belongs to an unrelated window manager for Python 2.6, last released as 0.1.3
in 2010. Its setup.py installs a top-level ration package of its own, so the
two cannot share an environment.
publish.yml uploads a release from its signed tag only once the checks under
Versions and releases have passed and the stranger-shaped projects in
.github/strangers/ have judged the wheel it uploads. Each file it uploads
carries a PEP 740 attestation. After the upload, the same run reads each file's
provenance back from the index. A provenance that names any publisher but that
workflow, in this repository, in its pypi environment, turns the run red, and
the run then creates no GitHub release. v0.1.0 and v0.2.0 are GitHub
releases only, and their trees name the distribution ration.
Versions and releases
Releases are listed at https://github.com/catincloud-labs/ration/releases,
and the first is
v0.1.0.
Install a release from a clone at its tag, then as in Running it, below; for
another release, put its tag in place of v0.1.0:
git clone https://github.com/catincloud-labs/ration.git
cd ration
git switch --detach v0.1.0
A clone at a tag holds this README as it stood at that tag. The one at
v0.1.0 still says no release had been cut, which is what it said before the
tag existed; CHANGELOG.md at main records that sentence and when it died.
A GitHub release carries no files of its own. The source archives GitHub
attaches to every release hold no .git, so the tag cannot be checked from
one; use the clone. A release is a signed v* tag, its upload to PyPI from
v0.2.1 on, and its GitHub release, both made by
.github/workflows/publish.yml from the tag only after
.github/workflows/release.yml, which it calls, has verified the tag's SSH
signature against .github/allowed_signers, read the tag's name as exactly v
and the version pyproject.toml carries at the tagged commit, and run the
suite there. The same workflow runs both checks on tags built wrong on
purpose (unsigned, lightweight, a gpg signature, signed by another key, a
signature moved from another tag, no summary line; a name that is not the
version, only begins with it or is cut short of it, a version from another
table, a commit with no version, and a parser that cannot run) and fails unless
each is refused, on every pull request and before every release.
v0.1.0's release run predates the version check and its arms; see below.
The version in pyproject.toml at a commit no tag names is not a release.
While the major version is 0, a change to what the suite decides moves the
minor version and may break a caller; a patch release never changes what the
suite decides.
A tag is checked from a clone at that tag, with the script the workflow runs:
bash .github/release/verify-tag.sh v0.1.0 .github/allowed_signers
A bare git verify-tag is not the same check: it passes a signature made for
another tag and moved to this one. The script's exit 0 shows that a key
.github/allowed_signers names signed a tag object carrying this tag's own
name. The file travels in the same tree as the tag it vouches for, so it says
nothing on its own about who holds that key.
The version is checked with bash .github/release/check-version.sh <tag>,
which reads pyproject.toml from the commit the tag names, not from the
working tree. v0.1.0 was released before that script existed, so its tree
does not carry it and its release run did not compare its version; from a
clone at main, bash .github/release/check-version.sh v0.1.0 admits it.
Do not install by the bare name. ration on PyPI is an unrelated project.
.github/workflows/publish.yml uploads this package's releases to PyPI as
catincloud-ration from v0.2.1 on (Installing, above). Its wheel carries the suite that
judges a store, as ration.conformance, and the suite's one dependency, pytest,
is the conformance extra rather than a dependency of the package. The one
module of the suite that needs more is ration.conformance.postgres_violators,
the Postgres violator, which needs psycopg and is imported only by a caller
that registers it.
Running it
pip install -e ".[dev]"
pytest # the fakes only
The Postgres cases are registered only when a server is named, so a run without
one silently drops the only store in the suite that more than one process can
share. That is fine
locally and unacceptable in CI, where RATION_REQUIRE_POSTGRES=1 makes its
absence a failure:
docker run -d --name ration-pg -e POSTGRES_PASSWORD=ration \
-e POSTGRES_USER=ration -e POSTGRES_DB=ration -p 55432:5432 postgres:17-alpine
until docker exec ration-pg pg_isready -h 127.0.0.1 -U ration -d ration; do sleep 1; done
RATION_POSTGRES_DSN=postgresql://ration:ration@localhost:55432/ration pytest
The wait is load-bearing, and -h 127.0.0.1 is what makes it one. On a
new container the image's entrypoint first runs a temporary server for its init
with listen_addresses='': a unix socket and no TCP (docker-entrypoint.sh,
docker_temp_server_start). It then stops that server and starts the real one.
Over TCP, pg_isready answers only once the real server listens. Over the
socket it can answer during the temporary one; that is read from the
entrypoint, not observed. Without the wait, a run started against a new
container can fail with server closed the connection unexpectedly, while the
same suite against a server that has finished starting is green.
A run reports xfailed once per violator per property it is expected to break.
Those are the suite working. A passing violator is reported as a failure,
which is the only outcome that means the suite has stopped measuring anything.
Running the suite against your own store
The suite ships in the package as ration.conformance, and a store of yours
joins it through a pytest plugin, ration.conformance.plugin. The plugin has no
entry point, so installing ration never enables it: a project that holds a
lease with ration runs its own tests as it did. A project that judges a store
installs the package with its conformance extra, from a clone at a release's
tag as in Versions and releases, and enables the plugin by name. From the
directory holding the clone:
pip install "./ration[conformance]"
From v0.2.1 on, the same package and extra install from PyPI, with
pip install "catincloud-ration[conformance]".
A tag whose tree has src/ration/conformance/plugin.py has the plugin;
v0.1.0 does not.
Every run through the plugin judges the package's own cases beside yours.
They are the in-memory store, the correct store with an unusual answer, every
violator in the table under The first piece of work but postgres-naive,
which needs a server, and every frame codec.
Each violator is strict-xfailed against the property it breaks, so a run in
which one passes in your environment is red, whatever your store did. Frame
codecs take no case of a caller's: the frame is this package's wire format and
ration.frame its reader, and a hook for them would be a public name with
nothing yet to carry.
In the conftest.py at the root of the directory you run pytest from, enable
the plugin and answer its one hook, pytest_ration_cases, with your cases.
This is the shape .github/strangers/common/stranger.py builds for the
installed wheel job, for a store in one process:
import time
from ration.conformance.support.backends import BackendUnderTest, Case
from ration.conformance.support.faults import FaultInjector
from yourpackage import YourStore # a `ration.contract.Store`
pytest_plugins = ["ration.conformance.plugin"]
def your_case() -> Case:
def build() -> BackendUnderTest:
faults = FaultInjector(YourStore())
return BackendUnderTest(
case=case,
store=faults,
advance_store_time=time.sleep,
skew_client_clock=lambda _seconds: None,
partition=faults.partition,
ambiguity=faults.ambiguity,
)
case = Case(name="your-store", violates=frozenset(), build=build)
return case
def pytest_ration_cases() -> list[Case]:
return [your_case()]
Then run the suite the package ships, naming the case the run must judge:
pytest --pyargs ration.conformance --ration-require your-store
The report header names the package's cases, yours and what is required, and each test's id names the case it ran.
--ration-require NAME is the switch that keeps a lost store from reading
green. Repeat it for each case the run must judge; it is modelled on this
repository's RATION_REQUIRE_POSTGRES. If no collected test runs a case of
yours by that name, because none was registered (a CI job that lost the
variable naming your server, say), because the suite was not collected, or
because -k, -m or a node id left it out, the run ends as a usage error,
exit 4, before any test runs. A name of the package's own cases does not
satisfy it, and a case named like one of the package's is refused, because the
name is how the report and the switch tell cases apart.
The case contract
A Case and the BackendUnderTest its build returns are in
ration.conformance.support.backends; FaultInjector is in
ration.conformance.support.faults.
| Field | What it must do |
|---|---|
Case.name |
Name one case in the run. It is each test's id and what --ration-require takes. |
Case.violates |
Name the properties the store is expected to fail, as each suite module's conformance marker names them; empty for a store you claim is correct. Each test of a named property is strict-xfailed, so a store that passes one is a failing run. A name no test declares marks nothing, and the check that catches one, tests/test_the_suite_discriminates.py, is this repository's and not in the package. |
Case.build |
Return a fresh BackendUnderTest, called with no arguments for each test that runs the case. key() hands each build keys nothing else has touched, so a persistent store may serve every build. |
case |
The Case it was built from. |
store |
Your store, wrapped in FaultInjector, so ambiguity can lose a write's answer. |
advance_store_time(seconds) |
Return once at least seconds of the store's own time has passed: time.sleep for a store that reads its own clock. The suite's leases last REAL_TTL, half a second, so the waits stay short. |
skew_client_clock(seconds) |
Move the clock on the caller's side of the store. A store never told the caller's time takes a no-op, and that the expiry tests then hold trivially is the result, not a gap. |
partition() |
A context manager inside which every read and write raises Unreachable, with the store reachable again after it. For a store with a server, point it somewhere that refuses a connection, as postgres_case in the same module does, so the suite sees your store translate the failure. FaultInjector.partition refuses on the store's behalf and shows only that the layers above it refuse. |
ambiguity(applied=...) |
A context manager that makes the next write raise Ambiguous, having applied or not as applied says. Every registered case uses FaultInjector.ambiguity. |
_stepped |
Leave it out. The stepped build parks an operation in the memory store's own _staged seam, which no other store has. A case without one skips the test that needs it, and atomicity is decided for it by the threaded probe, at the false-pass rate test_atomicity.py states; a probe that never raced reports a skip rather than a verdict. |
What a green run shows, and what it does not. Your case passed every test of every property, in your environment, in a run in which the package's correct stores passed and each of its violators failed its own property. It shows nothing the suite does not decide: the gaps stated above, such as a version reused across a restore and a store only one process can open, are gaps for your store too.
Licence
Apache-2.0. See LICENSE.
Metadata
Release files for catincloud-ration 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| catincloud_ration-0.2.1.tar.gz | 133.7 kB | With provenance | Details for catincloud_ration-0.2.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| catincloud_ration-0.2.1-py3-none-any.whl | Python 3 | none | any | Details for catincloud_ration-0.2.1-py3-none-any.whl |
Total release size: 262.4 kB
Release files / catincloud_ration-0.2.1.tar.gz
| Download URL | catincloud_ration-0.2.1.tar.gz |
|---|---|
| Size | 133.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7dd9d14916427124b06f91f522bf3c7786d54b15418a38d7400bde3a74ac9280
|
|
BLAKE2b-256 checksum How to use checksums |
64fef73191cd78cd1b43bdad575304731ed74ec2dac107a9e1471bb29f437f3f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logRelease files / catincloud_ration-0.2.1-py3-none-any.whl
| Download URL | catincloud_ration-0.2.1-py3-none-any.whl |
|---|---|
| Size | 128.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0d328d8c51e88c5ca63a684c888103aa8afdc85d609e97b27475ff52c3feaf59
|
|
BLAKE2b-256 checksum How to use checksums |
b30d1f7c71bb065f8c251b15035c83b94ab2e978e9f73c048fd1a4508ee093a1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logSecurity for catincloud-ration 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for catincloud_ration-0.2.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.