fiftyone_pipeline_did

Strongly typed Python reader and cloud client for the 51Did (51Degrees Identifier) returned by the 51Degrees Cloud service. Mirrors the .NET FiftyOne.Did package.

Terminology

• The 51Did (51Degrees Identifier) is the identifier as a whole.
• The envelope is the data model that carries it: a signed OWID holding the version, domain, date, payload and signature. It changes byte-for-byte every time the cloud issues one.
• The match key is the stable, comparable part of the payload after the Flags and License Id, being a 32-byte SHA-256 for Probabilistic and HashedEmail identifiers, or 16 GUID bytes for Random. Two 51Dids for the same inputs share the same match key even though their envelopes differ.
• The Terms is the byte after the match key that says which terms document the identifier was created under, so the terms travel with the identifier instead of alongside it. The package turns that byte into the address of the document.

Comparing two 51Dids means comparing their match keys, never their envelopes.

Payload layout

The bytes a 51Did payload holds, and what each bit of them means, are specified once for every language in identifier-layout.md, and the surface each package offers over those bytes in package-surface.md. Those two pages are the authority, so read them rather than this summary where the two ever disagree.

In short, the payload opens with a header carrying the flags byte and the License Id, and the identifier type in that byte then fixes the length of the match key that follows, being 32 bytes for PROBABILISTIC and HASHED_EMAIL, 16 for RANDOM, and whatever remains for RESERVED. The Terms byte follows the match key, and the bytes after it are a creator context section. Identifiers issued before the type tag existed decode as PROBABILISTIC, and a payload that ends at the match key reads as terms that are not stated.

This package does not publish the offsets or the raw flags byte, and it does not need to, because every field has a typed accessor that reads it correctly. The usage is the clearest reason why, as its bits are cumulative rather than exclusive, so a caller masking the byte for the non-marketing bit alone reads every marketing identifier as non-marketing, which is exactly backwards for a rule that says a non-marketing identifier must never be passed to a demand source. fod_id.usage answers with the highest usage granted and that mistake cannot be made.

OWID dependency

FodId builds on the OWID envelope library (SWAN-community/owid-python, package owid), consumed via the 51Degrees/owid-python fork, which is a git submodule of this repository and will move to upstream once that is published. Owid is composed, not subclassed, so FodId holds an Owid and delegates OWID-level concerns to it.

The published package does not take OWID from PyPI. The 51Degrees fork is not published there, and the name owid on PyPI belongs to an unrelated project, so a declared dependency would install the wrong thing. Instead ci/copy-owid-source.ps1 copies the fork's source into the package as the private module fiftyone_pipeline_did._owid before the distribution is built, carrying the Apache-2.0 licence and a notice naming the source commit with it. The leading underscore keeps the top level name owid free on the consumer's machine. The only third party requirement the package declares is cryptography, which OWID uses for the signatures.

Nothing changes for a developer working in this repository, because the submodule stays where it is and the same script puts the copy in place. Run pwsh ./setup.ps1 from the repository root, or git submodule update --init --recursive followed by pwsh ./ci/copy-owid-source.ps1, before building or running the tests. The tests and examples import the fork under its own name, owid, which is how they build signed envelopes to test against.

The OWID types the public API refers to are re-exported from the package itself, so a caller never has to reach into the private module. Catch fiftyone_pipeline_did.OwidError for an OWID level failure raised by the raising readers, use fiftyone_pipeline_did.Owid for the envelope that FodId.from_owid takes, and read fiftyone_pipeline_did.SignatureStatus from FodId.signature_status. An Owid only ever comes from a successful parse or from an OWID Creator that signs one into being, so there is no way to hold an unsigned or partly built envelope.

Usage

from fiftyone_pipeline_did import FodId, IdType, Usage

fod_id = FodId.from_base64(base64_from_cloud_service)   # either alphabet

type_ = fod_id.type          # IdType.PROBABILISTIC / RANDOM / HASHED_EMAIL
usage = fod_id.usage         # Usage.NON_MARKETING / STANDARD / PERSONALIZED
indirect = fod_id.usage_is_indirect  # True when worked out from another
                                     # signal, today a consent string
license_id = fod_id.license_id
match_key = fod_id.match_key  # SHA-256 or GUID bytes, see type
terms = fod_id.terms          # address of the terms document it was
                              # created under, None where it names none
                              # this package knows

# Delegated OWID-level fields and operations.
domain = fod_id.domain
created = fod_id.date            # aware UTC datetime, to the minute
verified = fod_id.verify(public_key_pem)
base64 = fod_id.as_base64()      # standard alphabet, padded, as the cloud
url_safe = fod_id.as_base64_url()  # URL-safe alphabet, no padding, for a link

from_base64 accepts the standard alphabet the cloud issues and the URL-safe alphabet a page puts in a link, with or without padding. On an identifier carrying a creator context the License Id field holds an encrypted value that only 51Degrees can turn back into a licence identifier, so license_id is the field's raw value and identifies nothing outside 51Degrees.

The usage an identifier was created for

fod_id.usage says what the identifier may be used for, as a Usage carrying NON_MARKETING, STANDARD or PERSONALIZED. Those are the only three values. An identifier with no usage bit set did not come from the cloud, so reading one is refused with FodIdParseStatus.NO_USAGE rather than answered with a fourth value. The usage decides where the identifier may go, because one created for non-marketing must never be passed to a demand source, and one created for standard or personalized marketing may be passed only to a recipient that has accepted the applicable terms. usage.id_usage gives the cloud's own id.usage wording, being non-marketing, standard or personalized.

The three usages are cumulative in the bits that carry them, so every marketing identifier also carries the non-marketing bit. fod_id.usage answers with the highest usage granted, which is why it is the only supported way to read the usage and why the package does not hand out the byte. fod_id.usage_is_indirect says whether the usage is indirect, being worked out by the issuer from a signal the caller sent, rather than direct, being stated by the caller. A consent string is the only indirect signal today. Both are legitimate ways to arrive at a usage and the answer says nothing about which usage was reached. The accessor was called usage_from_consent before, and that name has been removed rather than kept beside the new one.

The terms an identifier was created under

fod_id.terms answers with the address of the terms document the identifier was created under, so the terms travel with the identifier instead of alongside it and a receiver can tell which document was in force when the identifier was made. The byte behind it is an index into a table of terms documents in the layout specification, and the package turns the index into the address so a caller never handles the byte. The byte is an index and not a version number, so that a later document can live at any address rather than only at one the specification could compose from a number, and an index is never reused or repointed once published, because an identifier issued under it has to stay readable years later.

The answer is None where the identifier does not state its terms and where it states an index added after this package was released, using absence rather than an empty string. No address is ever built from an index this package cannot name, because that would name a document nobody wrote and a receiver would record having accepted terms that do not exist. A caller therefore cannot tell those two apart, which is deliberate, since both lead to the same place. This package answers with the address and never fetches it, because what to do with the document is the receiver's decision.

No address does not mean the identifier is unrestricted. It says only that this identifier does not carry the answer, so the answer has to come from somewhere else, being the Terms Document Locator in an OpenRTB request or whatever the surrounding protocol provides.

A payload with no byte after the match key reads as index 0, so absence and zero mean the same thing and no presence flag exists to tell them apart. The Usage and the Terms answer different questions and a receiver needs both, because the Usage says where an identifier may go and the Terms says under which document it was created.

The payload version

Bits 4 and 5 of the flags byte say which payload layout the identifier follows, and this package reads version 0. A payload naming version 1, 2 or 3 is refused with FodIdParseStatus.UNSUPPORTED_PAYLOAD_VERSION, and the raising readers name the version they found in the message.

No field is read under the layout this package knows once the version says otherwise. A later version exists precisely because a field moved, so reading such a payload here would answer with values that are wrong rather than absent, which is worse than refusing. A version that nothing checks protects nothing.

The version is not exposed. Either this package read the layout, in which case the accessors are the answer, or it did not, in which case there is no identifier to read fields from.

No identifier already issued is refused by this. The two bits held no field before the version was defined and every issuer wrote them as zero, which is version 0, so an identifier from before the field existed reads exactly as it did.

The raw flags byte, the byte layout constants and the old hash names are not part of this package. fod_id.flags, fod_id.hash, fod_id.date_minutes, FodId.MATCH_KEY_OFFSET and every other offset and length were removed, in this package and in the .NET, Java, Node, PHP and Rust ones together, so the surface stays the same in every language. Read type, usage, usage_is_indirect, license_id, match_key and date instead, which read the same bytes and cannot get the cumulative usage bits wrong. The layout is still specified, in identifier-layout.md, for anyone implementing a reader rather than using one.

Parsing without exceptions

An identifier arriving from outside, in a query string, a header or a form field, may be anything at all, and failing to be a 51Did is an ordinary outcome rather than a fault. try_from_base64 and try_from_byte_array read such input without raising and answer with a FodIdParseResult, a small immutable tuple carrying three facts:

• ok, whether the parse succeeded;
• value, the FodId on success and None on failure, never a partly read identifier;
• status, a FodIdParseStatus, which is PARSED on success and the specific reason otherwise.

The result is truthy on success, so if result: reads naturally.

from fiftyone_pipeline_did import FodId, FodIdParseStatus

result = FodId.try_from_base64(text_from_the_request)   # either alphabet
if result:
    fod_id = result.value
else:
    reason = result.status      # for example FodIdParseStatus.INVALID_BASE64

Parsing and verifying are separate steps. A successful parse says the bytes have the shape of a 51Did and nothing about whether the signature is genuine, so a parsed identifier is not known to be genuine until fod_id.verify(public_key_pem), fod_id.signature_status(public_key_pem) or a DidClient check says so. signature_status answers in the OWID SignatureStatus vocabulary, where only SIGNATURE_VALID and SIGNATURE_INVALID are about the signature. KEY_UNAVAILABLE, INVALID_KEY and VERIFICATION_ERROR say the question could not be answered, which must never be read as a forgery, and the boolean verify raises for a key it cannot use rather than answering False for the same reason.

Status meanings

The FodIdParseStatus vocabulary is the OWID one, member for member and value for value, plus four members for the payload rules this package applies once the envelope has been read. A failure inside the envelope is carried through with the OWID status unchanged, so the reason reads the same whichever language parsed the bytes.

Lower bounds and no upper bound

The payload must hold the 5 byte header before the type can be read, and the type then says how many match key bytes must follow, being 16 for RANDOM and 32 for PROBABILISTIC and HASHED_EMAIL, as the payload layout specification says. The Terms byte follows the match key, and a payload ending at the match key reads as terms that are not stated. RESERVED keeps the best-effort reading, being the header fields and whatever bytes follow, and because no match key length is defined for it there is no byte left over to read as its Terms. Anything beyond the Terms is a creator context section whose lengths belong to the cloud, so a longer payload, a longer creator domain (a self-hosted container may sign with one) or a longer envelope is accepted and this package places no upper bound of its own on any of them. An older reader meeting a context section of a version it does not know still reads the header, the match key and the Terms.

DidClient refuses text longer than 4096 characters before it parses it, fetches a key or calls the cloud. That figure is client policy, deliberately arbitrary and generous, and it is not a statement of how long a 51Did can be. The parser answers such text with an ordinary result, whilst the client raises its usual ValueError.

Expected results and exceptions

Every FodIdParseStatus other than PARSED is an expected data result from the try_ readers and never an exception. The raising readers, from_base64, from_byte_array, from_owid and the constructor, read through the same logic and keep their documented exceptions for callers who prefer them, being TypeError for None or a wrong input type, ValueError for PAYLOAD_TOO_SHORT, INVALID_TYPE_PAYLOAD_LENGTH and UNSUPPORTED_PAYLOAD_VERSION, and OwidError for every other status, with the message naming the status. The three raising ValueError are the three the payload rules produce, whilst every other status comes from the envelope. Signature verification against a key that cannot be decoded, a key list that cannot be fetched, and a cloud answer other than the one asked for remain exceptions, because they are faults in the surroundings and not properties of the identifier.

Migrating from the removed OWID API

The OWID library no longer offers a throwing parse or a public constructor, so an envelope cannot be assembled by hand, and code that used those through this package changes as follows.

# Before the hardening, external input was read by catching what the
# reader raised.
from fiftyone_pipeline_did import FodId, OwidError
try:
    fod_id = FodId.from_base64(text)
except (OwidError, ValueError):
    fod_id = None

# After the hardening, ask for the result and its reason.
from fiftyone_pipeline_did import FodId
result = FodId.try_from_base64(text)
fod_id = result.value if result else None

# Before the hardening, an envelope was built by hand and signed
# afterwards, as the tests and the offline example did.
owid = Owid(domain=domain, payload=payload)
creator.sign(owid)

# After the hardening, the creator signs a new envelope into being from
# the payload.
owid = creator.create(payload)

from_base64, from_byte_array, from_owid and the constructor keep working and keep their exception types.

Comparing two 51Dids

a = FodId.from_base64(idprobglobal_a)
b = FodId.from_base64(idprobglobal_b)

# The envelope (date, signature, base64) differs across reissues.
# The match key inside the payload is stable, so compare match keys:
same_match_key = a.match_key == b.match_key

Verifying on your server

DidClient handles every manipulation of a 51Did a server needs against the cloud, so server code never builds a cloud URL or handles a key itself. Every method that reaches the cloud is a coroutine, so a server awaits it on its event loop and there is no blocking form. One instance serves a whole server, and its key cache is guarded by an asyncio lock so that concurrent awaits for the key list share one fetch. It uses asyncio, urllib and json from the standard library, so this package gains no dependency the pipeline does not already carry.

import asyncio
import os
from fiftyone_pipeline_did import DidClient, FodId

client = DidClient(
    os.environ["_51DEGREES_RESOURCE_KEY"],
    os.environ.get("_51DEGREES_LICENSE_KEY"),   # optional, see below
    # endpoint defaults to FOD_CLOUD_API_URL, then the public cloud
)

async def check(text):
    fod_id = FodId.from_base64(text)
    return await client.verify_signature(fod_id)

# Inside an asyncio server, await the client directly. A script or a
# synchronous framework drives one call with asyncio.run.
valid = asyncio.run(check(fifty_one_did))

The default transport runs urllib on a worker thread through asyncio.to_thread, which keeps blocking I/O off the event loop rather than making the I/O itself non-blocking. For a fully non-blocking client supply a transport built on aiohttp or httpx, being an async callable that takes the prepared urllib.request.Request and answers (status, body_bytes) whatever the status. Neither library is a dependency of this package. With httpx, for example:

import httpx

async def httpx_transport(request):
    async with httpx.AsyncClient(timeout=30) as http:
        answer = await http.request(
            request.get_method(), request.full_url,
            content=request.data, headers=dict(request.header_items()))
        return answer.status_code, answer.content

client = DidClient(resource_key, licence_key, transport=httpx_transport)

Every request carries a User-Agent naming this package and its version.

1. Parse. The identifier arrives from a page in the URL-safe alphabet and from the cloud in the standard one. from_base64 takes either, with or without padding, and as_base64_url() gives the form to put in a URL. Input that may not be a 51Did at all is better read with try_from_base64, which names the reason instead of raising (see "Parsing without exceptions" above). Neither checks the signature.

fod_id = FodId.from_base64(fifty_one_did)

2. Verify the signature offline. The client fetches the published signing public keys from the cloud once, caches them for a day, and picks the key in force when the identifier was created, being the entry whose start is latest on or before the identifier's date (a key stays in force until the next one starts, and keys are published up to three months ahead). Near a period boundary the neighbouring key is tried as well. No earlier key is ever tried. The envelope version must be the one the cloud signs and the payload at least the base length for its type, and a longer payload carries a creator context and is accepted, its exact lengths being for the cloud to judge.

valid = await client.verify_signature(fod_id)            # bool
check = await client.verify_signature_detailed(fod_id)   # SignatureCheck
# check.valid is False and check.reason is SignatureReason.NO_KEY when
# no published key covers the identifier's date
keys = await client.public_keys()    # [PublicKeyEntry(starts_at, public_key)]
key = await client.public_key_for(fod_id)  # the entry in force, or None

These reach the cloud only when the key list needs fetching, and they are coroutines all the same so the fetch never blocks the loop.

3. Verify the signature through the cloud. The open verify endpoint, one use against the resource key and no licence key needed. The identifier is sent under both the 51did and owid query names, so the call works with hosts that read either parameter. A string that does not parse as a 51Did raises DidArgumentError (a ValueError) before any request is made, with the message naming the FodIdParseStatus, and a value the cloud itself refuses raises the same error carrying the cloud's message and status_code 400.

valid = await client.verify(fod_id)   # bool

4. Redeem a sealed creator context result. The verify-context and verify-full endpoints are browser calls, because the creator context describes the browser's own connection, and they return the verdict only as an encrypted result the browser cannot read or forge. The party that acts on it redeems it on the server, with the licence key, against the 51Did it knows independently.

redeemed = await client.redeem(fod_id, result, challenge)
redeemed.context                  # ContextResult: VERIFIED, MISMATCH,
                                  #   NO_CONTEXT, MISCONFIGURED,
                                  #   INVALID_DATE, EXPIRED, REPLAYED,
                                  #   UNREADABLE, UNCONFIRMED
redeemed.signature                # SignatureResult: VERIFIED, INVALID
                                  #   or UNKNOWN
redeemed.factors                  # only where there is something to
                                  #   diagnose, name to FactorResult
                                  #   (VERIFIED, MISMATCH or
                                  #   MISCONFIGURED) or None where nothing
                                  #   was compared, for transport, device,
                                  #   browserip, connectionip, asn,
                                  #   platformname, platformversion,
                                  #   browsername and browserversion
redeemed.verified_at              # datetime, on the redeemed and expired
                                  #   outcomes
redeemed.seconds_since_verified
redeemed.status_code              # 200, or 503 for UNCONFIRMED, which may
                                  #   be retried
redeemed.raw                      # the body as received
redeemed.to_dict()                # the cloud's own response shape, for
                                  #   relaying to a page

A context string this package does not know maps to UNREADABLE, so an unrecognised outcome is never mistaken for a good one, and context_raw keeps the string as sent. Every cryptographic failure comes back from the cloud as the one word unreadable by design, a missing licence key included, so the client does not try to tell them apart either. A string that does not parse as a 51Did raises DidArgumentError before any request is made, a cloud that refuses the 51Did raises the same error with HTTP 400, a host that does not offer the creator context raises DidNotSupportedError (HTTP 404), and any other status raises DidClientError carrying status_code and body. A transport failure raises the OSError the transport raised, which is urllib.error.URLError by default. Each of these is raised when the call is awaited, as with any coroutine.

Examples

The examples folder holds an offline example and a web demo that calls the 51Degrees cloud.

fodid_example.py builds a sample 51Did in process and parses it back with this package, so it needs no resource key and makes no cloud calls.

creator_context_web/ is a small demo web app, server.py serving page.html, that runs the 51Did creator context flow the way production does. The creator context only makes sense from a browser, because a server verifying its own connection would be checking itself against itself, so there is no console example.

1. Create a 51Did by loading the 51Degrees client script, which runs the snippets the service asks for, sends what they collected, and hands the page the answer with the identifier in it. The browser does all of that, so the identifier is created for the browser's own connection. The service issues an identifier only once those values are in, so a page asking for one by itself is told the page has not finished and is given nothing.
2. Verify it with verify-full, which returns both the signature outcome and the creator context verdict only as an encrypted result that the caller cannot read or forge. (A deployment holding no context secret answers in the open instead.) The browser makes this call too, so the cloud observes the browser's live connection, and it sends the snippet values with it so that this browser can be compared with the creator. The page then hands the encrypted result to its own server.
3. Redeem the encrypted result with redeem, presenting the 51Did, the encrypted result and the account's licence key, and receive the true creator context verdict, when the verification happened (verifiedAt) and how long ago that was (secondsSinceVerified). The server makes this call, as the only party holding the licence key.

A fresh challenge is issued per page load and bound through both steps by the cloud. A production server would also remember the value it issued and reject a redemption carrying any other, which the demo keeps out of scope.

The server-side step to copy into your own server

The one part that belongs on your server is the redeem call with the licence key, which is what the /redeem handler in server.py does with the DidClient from this package. It parses the identifier the page sent, checks its signature offline against the published public keys, then redeems the encrypted result with the challenge, adding the licence key the browser never sees. The essential lines are these.

import asyncio
from fiftyone_pipeline_did import DidClient, FodId

# Once, at start-up. RESOURCE, LICENCE and API come from the
# environment variables below.
client = DidClient(RESOURCE, LICENCE or None, API)

async def check(client, fod_id, result, challenge):
    signature_valid = await client.verify_signature(fod_id)
    redeemed = await client.redeem(fod_id, result, challenge)
    return signature_valid, redeemed

# In the /redeem handler, with 51did, result and challenge from the
# page. The identifier arrives in the URL-safe alphabet, which
# from_base64 accepts alongside the standard one. The demo sits on the
# standard library's synchronous http.server, so it runs the check on a
# short-lived event loop. An asyncio server awaits check() directly.
fod_id = FodId.from_base64(query.get("51did", [""])[0])
signature_valid, redeemed = asyncio.run(check(
    client, fod_id, query.get("result", [""])[0],
    query.get("challenge", [""])[0]))
body = redeemed.to_dict()
body["serverSignature"] = "verified" if signature_valid else "invalid"
self.send_json(redeemed.status_code, body)

The handler answers the page with the cloud's status and a body in the cloud's own shape (signature, context, factors when present, verifiedAt, secondsSinceVerified) built from the typed result, plus serverSignature, the server's own offline check of the identifier's signature. The page ignores fields it does not know, so page.html is the same for every language.

A verdict of nocontext is a normal outcome and not an error. A self-hosted container may be configured not to emit the creator context, so an identifier it issued has nothing to check and redeems as nocontext with no factors, and the page shows it the way it shows any verdict. A 404 from verify-full or redeem means the host answering does not support the creator context at all, which is a service without the feature rather than a failed check. The client raises DidNotSupportedError for that case, the handler answers 404 with a text body, and the page shows "not supported by this host". Any other answer the cloud gives is relayed with its status and body, so the page reports whatever the service said readably, and an unreachable cloud answers 502 with { "error": ... }.

Environment variables

Running

With the resource key set as above:

cd fiftyone_pipeline_did/examples/creator_context_web
python server.py

then open http://localhost:5100/. To demonstrate across two devices, serve on an address both can reach and open the copied link on the second device.

What a run costs

Every call the demo makes to the cloud is one use against the subscription behind the resource key. Checking a 51Did from the browser makes two, verify-full from the page and redeem from the server, so a browser-based context check is two uses every time. Checking only the signature with verify is one use.

The copy-and-paste proof

Once the 51Did has fully validated, the page shows a copy-and-paste section with a link carrying the same 51Did, and an explanation of what will happen next. Open that link in a different browser and the same page loads with the same identifier. The signature still verifies and the identifier unpacks, because it is genuine, but the creator context does not validate, because the context binds the identifier to the browser and connection it was created on. That visible failure is the demonstration that matters, a copied or stolen identifier caught at presentation with nothing stored server side. Opening the link in the same browser is not the demonstration, since the same browser presents the same context and may still verify.

The stylesheet

examples-main.min.css beside the demo is the design system build and is refreshed by common-ci's update-example-assets step.

Non-goals

• No signature verification on parsing. A parsed 51Did is not known to be genuine. Call verify(public_key_pem), signature_status(public_key_pem) or a DidClient check when needed.
• No fetching of a terms document. fod_id.terms answers with the address and nothing more. What to do with the document is the receiver's decision.
• No upper bound on the size of an identifier. The lengths beyond the header, match key and Terms belong to the cloud. The 4096 character figure in DidClient is client policy against obviously malformed text, not a format limit.
• No creation of new 51Dids. This is a parser; new 51Dids are issued by the 51Degrees cloud / on-premise hashing engines.