pyxet is a Python library that provides a lightweight interface for the XetHub platform.
Project description
Pyxet is a Python library that provides a pythonic interface for XetHub. Xethub is simple git-based system capable of storing TBs of ML data and models in a single repository, with block-level data deduplication that enables hundreds of versions of similar data to be stored without requiring much storage.
The pyxet library has 3 components:
-
A fsspec interface that allows compatible libraries such as Pandas, Polars and Duckdb to directly access any version of any file in a Xet repository. See below for some examples.
-
A command line interface inspired by AWSCLI that allows files to be uploaded to and downloaded from Xet repository conveniently and efficiently.
-
A file system mount mechanism that allows any version of any Xet repository to be mounted. This works on Mac, Linux, and Windows 11 Pro.
For API documentation and full examples, please see here.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyxet-0.1.9-cp37-abi3-win_amd64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 7ea043c7dce203b1c0c69823c58c297c6043ecac4a930bca60844b074d326105 |
|
| MD5 | e17722d4db31c2d965ad659e7d61759b |
|
| BLAKE2b-256 | 7f4c4513f94774e97648c1567b3c780b3924aa9c265160af2a8202830aa8d151 |
Hashes for pyxet-0.1.9-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | f4e74f7c1e297b45702fccc8021934c6e85171e8bd2180d2fdbc32a03b1740c2 |
|
| MD5 | 0b158bebca270ebcfd558da4864cbd3d |
|
| BLAKE2b-256 | f1ccecaac7847e5f41cda2e93068d801a70f44e74864734e382225c16b8d67db |
Hashes for pyxet-0.1.9-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 19a3f47ec8f889e2ced50e2a26ec8fc8ba81839eb20744dff03207c40f63f91a |
|
| MD5 | 3dfc851e7015548e0561f0f5b95b987b |
|
| BLAKE2b-256 | 04e95ac58f64520096e456e60b4ce5e252aeb14758656f0bee2a61370465baf4 |
Hashes for pyxet-0.1.9-cp37-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | fab47baef90feeb02ac8e2c278f0063f045a308437358eba21dcfbc58e6dcdf6 |
|
| MD5 | 57d5aeaf4fb8fc989c3ce8d0c51c773e |
|
| BLAKE2b-256 | a8b803d71bcd79e3ef0ac4231c6bf4fdd78f4152b6204d14440a064f56b38d0b |
