Skip to main content

lakeFS High-Level Python SDK

lakeFS High Level SDK for Python, provides developers with the following features:

  1. Simpler programming interface with less configuration
  2. Inferring identity from environment
  3. Better abstractions for common, more complex operations (I/O, transactions, imports)

Requirements

Python 3.10+

Installation & Usage

pip install

pip install lakefs

Import the package

import lakefs

Getting Started

Please follow the installation procedure and afterward refer to the following example snippet for a quick start:

import lakefs
from lakefs.client import Client

# Using default client will attempt to authenticate with lakeFS server using configured credentials
# If environment variables or .lakectl.yaml file exist 
repo = lakefs.repository(repository_id="my-repo")

# (The positional argument is simpler!)
repo = lakefs.repository("my-repo")

# Or explicitly initialize and provide a Client object 
clt = Client(username="<lakefs_access_key_id>", password="<lakefs_secret_access_key>", host="<lakefs_endpoint>")
repo = lakefs.Repository(repository_id="my-repo", client=clt)

# From this point, proceed using the package according to documentation
main_branch = repo.create(storage_namespace="<storage_namespace>").branch(branch_id="main")

# Prefer to use "repo" rather than pass in the same repository_id each time.
...

Examples

Print sizes of all objects in lakefs://repo/main~2

ref = lakefs.Repository("repo").ref("main~2")
for obj in ref.objects():
  print(f"{o.path}: {o.size_bytes}")

Difference between two branches

for i in lakefs.Repository("repo").ref("main").diff("twig"):
   print(i)

You can also use the ref expressions here, for instance .diff("main~2") also works. Ref expressions are the lakeFS analogues of how Git specifies revisions.

Search a stored object for a string

with lakefs.Repository("repo").ref("main").object("path/to/data").reader(mode="r") as f:
   for l in f:
     if "quick" in l:
	   print(l)

Upload and commit some data

with lakefs.Repository("golden").branch("main").object("path/to/new").writer(mode="wb") as f:
   f.write(b"my data")

# Returns a Reference
lakefs.Repository("golden").branch("main").commit("added my data using lakeFS high-level SDK")

# Prints "my data"
with lakefs.Repository("golden").branch("main").object("path/to/new").reader(mode="r") as f:
   for l in f:
     print(l)

Unlike references, branches are readable. This example couldn't work if we used a ref.

Tests

To run the tests using pytest, first clone the lakeFS git repository

git clone https://github.com/treeverse/lakeFS.git
cd lakefs/clients/python-wrapper

Unit Tests

Inside the tests folder, execute pytest utests to run the unit tests.

Integration Tests

See testing documentation for more information

Documentation

lakeFS Python SDK

Author

services@treeverse.io

Metadata

Release files for lakefs 0.17.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for lakefs 0.17.0
File Size Uploaded Provenance Actions
lakefs-0.17.0.tar.gz 36.5 kB Without provenance Details for lakefs-0.17.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for lakefs 0.17.0
File Interpreter ABI Platform Actions
lakefs-0.17.0-py3-none-any.whl Python 3 none any Details for lakefs-0.17.0-py3-none-any.whl

Total release size: 78.7 kB

Release files / lakefs-0.17.0.tar.gz

Download URL lakefs-0.17.0.tar.gz
Size 36.5 kB
Tags Source
SHA-256 checksum
How to use checksums
0c2c1daff9b4e60717abaf0c834b3e9872c2038c77bf0cf891773c7e1cc8f310
BLAKE2b-256 checksum
How to use checksums
0c6e414f2c6d30468418a56276b8c257c05062c739bcc41ffb59dee4209601ad
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / lakefs-0.17.0-py3-none-any.whl

Download URL lakefs-0.17.0-py3-none-any.whl
Size 42.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
aaff11b74c8cbf51980b3bac034ce9035911dcf03f43c86b35c4c5a93f5d5d4b
BLAKE2b-256 checksum
How to use checksums
d17bf0183851c4b82afef38df5eb4f05b79d52ae1bee4dd7a6fcc9941f5a78f5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Security for lakefs 0.17.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.17.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page