pylakekeeper — Lakekeeper Python Client

Please visit https://docs.lakekeeper.io for Documentation!

pylakekeeper is the official Python client for Lakekeeper, an Apache-Licensed, secure, fast and easy to use implementation of the Apache Iceberg REST Catalog specification.

It covers the two things a plain Iceberg REST client cannot do for you: the Generic Tables API — register Lance, Delta, Parquet, images or any other dataset format as first-class catalog tables — and Storage Access Management, mapping Lakekeeper's vended credentials straight into the keys that lance, boto3, fsspec and deltalake already expect. If you have questions, feature requests or just want a chat, we are hanging around in Discord!

Quickstart

pip install pylakekeeper

from pylakekeeper import Client, StaticToken

lk = Client(
    base_url="http://localhost:8181",
    warehouse="my-warehouse",
    auth=StaticToken("my-token"),
)

# Register a Lance dataset as a generic table
lk.generic_tables.create("ai.models", "embeddings", format="lance")

# Load it back with short-lived, vended S3 credentials
resp = lk.generic_tables.load("ai.models", "embeddings", vended=True)

import lance

lance.write_dataset(data, resp.location, storage_options=resp.lance_storage_options)

# ...or open the same location as an fsspec filesystem
import fsspec

fs = fsspec.filesystem("s3", **resp.fsspec_kwargs)
print(fs.ls(resp.location))

Need a running catalog first? The Lakekeeper Getting Started Guide brings one up with docker compose in a couple of minutes.

Scope and Features

pylakekeeper is deliberately small: httpx + pydantic, nothing else. It is not a general-purpose Iceberg REST client — for the standard Iceberg surface use PyIceberg, which pairs with this happily (pip install 'pylakekeeper[iceberg]').

• Generic Tables: Register non-Iceberg datasets (Lance, Delta, Parquet, images, HDF5) as first-class catalog tables and get create / load / list / drop without faking Iceberg metadata.
• Vended Credentials: LoadGenericTableResponse exposes lance_storage_options and fsspec_kwargs, so short-lived credentials flow into your reader with no manual key-mapping.
• OpenID Provider Integration: client_credentials, device_code and authorization_code flows, all with automatic token refresh — or just a StaticToken.
• Fully Typed: Ships py.typed; the package is checked under mypy --strict.
• Well-Tested: Unit tests plus an integration suite that runs a real round-trip against Lakekeeper, Postgres, MinIO and Keycloak via docker compose.

If you are missing something, we would love to hear about it in a GitHub Issue.

Status

Authentication Flows

Flow	Status	Comment
Static Token		StaticToken — bring your own bearer token
OAuth2 Client Credentials		ClientCredentials, refreshes automatically
OAuth2 Device Code		DeviceCodeFlow — for CLIs and notebooks
OAuth2 Authorization Code		AuthorizationCodeFlow — local redirect server

Generic Table Operations

Operation	Status	Comment
Create		Any format identifier — see below
Load		Optional credential vending via vended=True
List		Paginated
Drop
Rename		Not yet exposed by the client

GenericTableFormat provides constants for lance, delta, vortex, paimon and dataset, but Lakekeeper stores the format as a free identifier — any string matching ^[a-z][a-z0-9_-]{0,63}$ is accepted, so pass a plain str for anything not listed.

Storage Backends

Storage	Status	Comment
S3 - AWS		Vended credentials mapped to Lance / boto3 / fsspec
S3 - Custom		MinIO, SeaweedFS and friends; allow_http set automatically
Azure ADLS Gen2		No credential translation yet — raw vended properties are still available on the response
Google Cloud Storage		No credential translation yet — raw vended properties are still available on the response

Optional Extras

Examples

Runnable scripts and notebooks live in python/examples/:

API Reference

All errors derive from LakekeeperError: NotFoundError, ConflictError, AuthError, ConfigError, LakekeeperHTTPError.

Contributing

The client lives in the lakekeeper-clients monorepo alongside the Java and Spark clients.

git clone https://github.com/lakekeeper/lakekeeper-clients.git
cd lakekeeper-clients/python
pip install -e '.[dev]'
pytest tests/                        # unit tests
pytest tests/integration -m integration  # needs Docker

License

Licensed under the Apache License, Version 2.0