Bookshelf Python SDK

The bookshelf package is the official Python SDK for the Bookshelf data platform. It provides synchronous and asynchronous facades for consuming published data, producing managed resources, and running record and replay publishing workflows. It also includes the bookshelf command line interface for authentication, discovery, and local cache management.

• Documentation
• Migrating from 0.4
• Changelog

Installation

Install the SDK from PyPI:

uv add bookshelf

The SDK requires Python 3.12 or newer. It ships with pandas and PyArrow, so as_df() and as_arrow() work out of the box. as_polars() uses Polars if you have it installed.

Install optional integrations as needed:

uv add "bookshelf[scmrun,publish]"

Consuming published data

The bookshelf package provides synchronous and asynchronous facades. Book coordinates resolve the latest published edition unless edition= pins one. Indexing a Book returns a BookEntry with book scoped exploration helpers.

search_volumes() finds volumes by free text plus discovery filters, and volume() resolves one, carrying the versions and editions it has published. Both are available on either facade, and neither needs credentials for public data.

from bookshelf import Bookshelf

with Bookshelf() as bs:
    found = bs.search_volumes("emissions", deprecated=False)
    volume = bs.volume("primap-hist")
    versions, latest = volume.versions, volume.latest

with Bookshelf() as bs:
    entry = bs.book("rcmip-emissions", "v5.1.0")["magicc"]
    frame = entry.as_df(year_min=2020, year_max=2100, filters={"region": "World"})
    sample = entry.preview(year_min=2020, top_n=5, drop_constant=True)
    facets = entry.facets()

as_df() returns every selected row as pandas, using wide indexed form for timeseries resources. The converter family also includes as_long_df(), as_scmrun(), as_polars(), and as_arrow(). They all take the same selection:

• filters maps a column to a value, or to a list of alternative values.
• year_min and year_max bound an inclusive year window on a timeseries.
• server_side=True has the platform select, rather than the verified cached file.

as_df(), as_polars() and as_arrow() also take order, a list of columns to sort by, each prefixed with - to sort descending.

The cached download is shared, so a second conversion costs no network work. An external pointer has no cached file, so the platform selects it on every call. An unknown filter column raises SelectionError, which is also a KeyError, on either route.

preview() returns a bounded DataPreview instead. It adds limit, top_n and drop_constant, and takes order on dimension columns. Its completeness says whether the sample holds every selected row.

Use bs.resource(tracking_id) for an exact machine or provenance path. fetch() verifies the declared SHA256 before storing bytes in the local content cache. download(destination) copies the verified file to a path you own. as_path() returns the cached file itself, which the cache may evict later.

The asynchronous facade has the same capabilities with awaited I/O:

from bookshelf import AsyncBookshelf

async with AsyncBookshelf() as bs:
    book = await bs.book("rcmip-emissions", "v5.1.0", edition=2)
    frame = await book["magicc"].as_df()

Producing and curating data

Managed resources are produced only inside an activity. The activity derives a stable config hash, records runtime provenance, materialises the object, and sends explicit Usage and Generation lineage to the API. Bare strings and UUIDs in used= are tracking ids. Use Used(name=...) to resolve an input by the name another resource in the same request was given. Pass role="plan" to register for the plan the activity followed, such as a method card. A plan needs a name, takes no used=, and is linked to every output of the activity rather than derived from its inputs.

from bookshelf import Bookshelf, Used, models

with Bookshelf() as bs:
    source = bs.book("rcmip-emissions", "v5.1.0")["magicc"]
    with bs.activity(code_ref="github.com/example/model@abc123", config={"scenario": "ssp245"}) as activity:
        output = activity.register(
            transform(source.as_df()),
            type="timeseries",
            name="model/ssp245/output",
            used=[source, Used(name="model/constants")],
        )

    draft = bs.draft_book("model-results", version="v1.0.0")
    draft.attach(
        output,
        name_in_book="ssp245",
        data_dictionary=[
            models.DataDictionaryEntry(name="region", role="dimension"),
            models.DataDictionaryEntry(name="value", type="number", role="measure"),
        ],
    )
    draft.publish()

register_external() is available on both Bookshelf and an activity. The former catalogues an existing pointer. The latter attributes an external output to the current run. Book drafting, attachment, and publication remain separate editorial calls. Each tabular or timeseries entry can declare its own column descriptions through draft.attach(..., data_dictionary=...). Omitting the argument preserves the entry's existing dictionary on re-attach, while an empty list clears it.

Use activity.register_many() with RegisterItem values for a batch. An atomic batch over 1000 items raises before any upload begins. A larger non atomic batch is split into requests of at most 1000 items. If any item fails, the facade finishes every chunk and raises PartialRegistrationError. The error retains indexed successful outcomes, usable committed resource handles, and each failed index with its typed ItemError. Index -1 identifies a batch level lineage failure reported by the server. The server decides: a registration aliases onto a resource your organisation already holds with the same bytes, unless a bundle pins its tracking id. The first canonical resource keeps its name, even when later items supply a different one. Returned producer handles expose registration_status and registration_outcome, so callers can detect this aliased result.

A failed multipart PUT can leave an unfinished upload because the server has no abort endpoint. Registration does not begin after that failure. A retry reuses the content addressed upload path and safely resumes the workflow.

Publishing a recorded bundle

bookshelf record runs a build file offline and writes a bundle, and bookshelf publish replays that bundle to the platform. bookshelf.publisher.replay_bundle and replay_bundle_sync do the same from Python.

A replay uploads the managed bytes, then sends the whole bundle to POST /v1/bundles/replay as one transactional request. The server registers the resources, mints the recorded activity's provenance edges, drafts the book, attaches every entry and publishes it, and rolls all of it back on a failure anywhere.

Every resource travels under its bundle-local name. A resource that a book entry names takes that name inside the book, and used= lineage cites the name of a resource recorded earlier in the same bundle. An input the platform already holds is cited by its digest instead. The server computes the seal from the request, so replaying the same bundle again converges on the one edition and reports converged rather than minting a rival.

Storing a pull request preview

bookshelf preview upload BUNDLE... stores the books a feedstock pull request would publish as one preview, so a reviewer can compare them with what is published before the pull request merges. The feedstock CI workflow runs it once per build, passing one bundle per candidate book.

bookshelf preview upload bundle-v1.0.0 bundle-v2.0.0 \
  --repository "$GITHUB_REPOSITORY" --pr "$PR_NUMBER" --pr-url "$PR_URL" \
  --head-sha "$HEAD_SHA" --main-sha "$MAIN_SHA" --tree "$TREE_SHA" \
  --run-id "$GITHUB_RUN_ID" --json

• It creates the preview, uploads every book's bytes under the preview, attaches each book and seals it.
• A bundle that fails validation is still a target, so the preview is failed with the reason and the command exits 7.
• Any error once the preview exists fails it before the command exits, so the check run never waits for a timeout.
• It prints the preview id, the proposal and preview links, the state and each book. The platform owns the check run and the pull request comment.

The command authenticates with the job's GitHub Actions OIDC token for the bookshelf audience. The usual credential chain is never consulted, so the job holds no Bookshelf credential. The workflow therefore needs the id-token: write permission, and without it the command exits 3 and names the permission.

permissions:
  contents: read
  id-token: write

Uploading a file that cannot be checked in

bookshelf upload FILE --type TYPE puts a file on the bookshelf as an input that belongs to no book, and prints the bookshelf://sha256/<hex> URI a recipe declares it by. Bookshelf.register_file does the same from Python, and Bookshelf.resource_by_hash resolves the digest back into the resource.

The command leaves the file hidden, so it is readable by the uploading organisation alone.

Credentials

Bookshelf and AsyncBookshelf authenticate through the credential providers in bookshelf.auth. Each is an httpx.Auth, so one provider object serves both the sync and async surfaces:

• StaticToken: a fixed bearer token, no refresh.
• RefreshTokenExchange: a WorkOS user access/refresh pair from bookshelf auth login. The refresh token rotates on each use and an on_rotate callback persists the new pair.
• ClientCredentials: an OAuth2 client_credentials machine credential. A refresh is a plain re-POST, there is nothing to persist.
• BsatAssertion: an agent identity assertion re-exchanged via the jwt-bearer grant against the API's POST /oauth2/token. It is explicit-only and never resolved from the environment.
• ActionsOidcToken: the running job's GitHub Actions OIDC token, minted for the read audience. It mints on first use and mints again once the API refuses the one it holds.

Refresh mechanics are shared: proactive refresh five minutes before expiry, one refresh-and-replay after an unexpected 401 (a second 401 raises AuthenticationError), and single-flight refresh behind per-surface locks. There is no background refresh task. A token handed in with no known expiry is refreshed before first use, because it may already be dead server-side.

When auth= is omitted, ambient credentials resolve in this order (explicit beats ambient, machine beats human):

1. $BOOKSHELF_TOKEN as a static bearer
2. a GitHub Actions OIDC token, only when $BOOKSHELF_AUTH=github-actions
3. $BOOKSHELF_CLIENT_ID + $BOOKSHELF_CLIENT_SECRET, minted at $BOOKSHELF_TOKEN_URL
4. stored bookshelf auth login credentials
5. unauthenticated (public reads)

auth= also accepts a provider instance or a bare token string, and an explicit auth=None stays unauthenticated. base_url resolves as argument, then $BOOKSHELF_URL, then its alias $BOOKSHELF_API_URL, then the production URL.

Client lifecycle in an embedded service

The client is long-lived by design: token state lives in the provider and each client pools connections. Construct one client at startup, inject it as a dependency, and close it at shutdown. In FastAPI that is a lifespan:

from contextlib import asynccontextmanager

from fastapi import FastAPI, Request

from bookshelf import AsyncBookshelf
from bookshelf.auth import ClientCredentials

@asynccontextmanager
async def lifespan(app: FastAPI):
    async with AsyncBookshelf(
        auth=ClientCredentials(client_id, client_secret, token_url=token_url),
    ) as bs:
        app.state.bookshelf = bs
        yield

app = FastAPI(lifespan=lifespan)

@app.get("/co2")
async def co2(request: Request):
    bs: AsyncBookshelf = request.app.state.bookshelf
    book = await bs.book("rcmip-emissions", "v5.1.0", edition=2)
    frame = await book["magicc"].as_long_df(filters={"variable": "Emissions|CO2"})
    return frame.to_dict(orient="records")

Do not open a client per request (async with AsyncBookshelf(...) inside a handler): that churns the connection pool and discards the cached token on every call. Context managers are optional. Notebooks can construct a client plainly and never close it.