lakeFS High-Level Python SDK
lakeFS High Level SDK for Python, provides developers with the following features:
- Simpler programming interface with less configuration
- Inferring identity from environment
- Better abstractions for common, more complex operations (I/O, transactions, imports)
Requirements
Python 3.9+
Installation & Usage
pip install
pip install lakefs
Import the package
import lakefs
Getting Started
Please follow the installation procedure and afterward refer to the following example snippet for a quick start:
import lakefs
from lakefs.client import Client
# Using default client will attempt to authenticate with lakeFS server using configured credentials
# If environment variables or .lakectl.yaml file exist
repo = lakefs.repository(repository_id="my-repo")
# Or explicitly initialize and provide a Client object
clt = Client(username="<lakefs_access_key_id>", password="<lakefs_secret_access_key>", host="<lakefs_endpoint>")
repo = lakefs.Repository(repository_id="my-repo", client=clt)
# From this point, proceed using the package according to documentation
main_branch = repo.create(storage_namespace="<storage_namespace>").branch(branch_id="main")
...
Examples
Print sizes of all objects in lakefs://repo/main~2
ref = lakefs.Repository("repo").ref("main~2")
for obj in ref.objects():
print(f"{o.path}: {o.size_bytes}")
Difference between two branches
for i in lakefs.Repository("repo").ref("main").diff("twig"):
print(i)
You can also use the ref expressions here, for instance
.diff("main~2") also works. Ref expressions are the lakeFS analogues of
how Git specifies revisions.
Search a stored object for a string
with lakefs.Repository("repo").ref("main").object("path/to/data").reader(mode="r") as f:
for l in f:
if "quick" in l:
print(l)
Upload and commit some data
with lakefs.Repository("golden").branch("main").object("path/to/new").writer(mode="wb") as f:
f.write(b"my data")
# Returns a Reference
lakefs.Repository("golden").branch("main").commit("added my data using lakeFS high-level SDK")
# Prints "my data"
with lakefs.Repository("golden").branch("main").object("path/to/new").reader(mode="r") as f:
for l in f:
print(l)
Unlike references, branches are readable. This example couldn't work if we used a ref.
Tests
To run the tests using pytest, first clone the lakeFS git repository
git clone https://github.com/treeverse/lakeFS.git
cd lakefs/clients/python-wrapper
Unit Tests
Inside the tests folder, execute pytest utests to run the unit tests.
Integration Tests
See testing documentation for more information
Documentation
Author
1 maintainer
Metadata
1 maintainer
Release files for lakefs 0.4.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| lakefs-0.4.0.tar.gz | 38.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| lakefs-0.4.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 86.0 kB
Release files / lakefs-0.4.0.tar.gz
| Download URL | lakefs-0.4.0.tar.gz |
|---|---|
| Size | 38.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
65cdc7e436409b61dbe1e180e48f87f2c80226256c6c840e5bd122218168c27e
|
|
BLAKE2b-256 checksum How to use checksums |
350566ac50027ccb7a5a57b5df3339706428d2c41470c6cb6fe953f7c9e6f1b0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.8
|
Release files / lakefs-0.4.0-py3-none-any.whl
| Download URL | lakefs-0.4.0-py3-none-any.whl |
|---|---|
| Size | 47.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ef44e71b2d9dbeeea3c52b56eddc9e2702e7921ff19bc59fcabc2bbc316d5e9c
|
|
BLAKE2b-256 checksum How to use checksums |
2b111a19736c550d9492338fa0e75427d61611ea890e0b908389e5e1e4c40e12
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.8
|
