Skip to main content

Fast transfer of large files with the Hugging Face Hub.

Project description

License GitHub release Contributor Covenant

🤗 hf-xet - xet client tech, used in huggingface_hub

Welcome

hf-xet enables huggingface_hub to utilize xet storage for uploading and downloading to HF Hub. Xet storage provides chunk-based deduplication, efficient storage/retrieval with local disk caching, and backwards compatibility with Git LFS. This library is not meant to be used directly, and is instead intended to be used from huggingface_hub.

Key features

chunk-based deduplication implementation: avoid transferring and storing chunks that are shared across binary files (models, datasets, etc).

🤗 Python bindings: bindings for huggingface_hub package.

network communications: concurrent communication to HF Hub Xet backend services (CAS).

🔖 local disk caching: chunk-based cache that sits alongside the existing huggingface_hub disk cache.

Installation

Install the hf_xet package with pip:

pip install hf_xet

Quick Start

hf_xet is not intended to be run independently as it is expected to be used from huggingface_hub, so to get started with huggingface_hub check out the documentation here.

Contributions (feature requests, bugs, etc.) are encouraged & appreciated 💙💚💛💜🧡❤️

Please join us in making hf-xet better. We value everyone's contributions. Code is not the only way to help. Answering questions, helping each other, improving documentation, filing issues all help immensely. If you are interested in contributing (please do!), check out the contribution guide for this repository.

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

hf_xet-1.3.2.tar.gz (643.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

hf_xet-1.3.2-cp314-cp314t-win_arm64.whl (3.5 MB view details)

Uploaded CPython 3.14tWindows ARM64

hf_xet-1.3.2-cp314-cp314t-win_amd64.whl (3.6 MB view details)

Uploaded CPython 3.14tWindows x86-64

hf_xet-1.3.2-cp314-cp314t-musllinux_1_2_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.14tmusllinux: musl 1.2+ x86-64

hf_xet-1.3.2-cp314-cp314t-musllinux_1_2_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.14tmusllinux: musl 1.2+ ARM64

hf_xet-1.3.2-cp314-cp314t-manylinux_2_28_aarch64.whl (4.0 MB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.28+ ARM64

hf_xet-1.3.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.2 MB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.17+ x86-64

hf_xet-1.3.2-cp314-cp314t-macosx_11_0_arm64.whl (3.5 MB view details)

Uploaded CPython 3.14tmacOS 11.0+ ARM64

hf_xet-1.3.2-cp314-cp314t-macosx_10_12_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.14tmacOS 10.12+ x86-64

hf_xet-1.3.2-cp313-cp313t-win_arm64.whl (3.5 MB view details)

Uploaded CPython 3.13tWindows ARM64

hf_xet-1.3.2-cp313-cp313t-win_amd64.whl (3.6 MB view details)

Uploaded CPython 3.13tWindows x86-64

hf_xet-1.3.2-cp313-cp313t-musllinux_1_2_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.13tmusllinux: musl 1.2+ x86-64

hf_xet-1.3.2-cp313-cp313t-musllinux_1_2_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.13tmusllinux: musl 1.2+ ARM64

hf_xet-1.3.2-cp313-cp313t-manylinux_2_28_aarch64.whl (4.0 MB view details)

Uploaded CPython 3.13tmanylinux: glibc 2.28+ ARM64

hf_xet-1.3.2-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.2 MB view details)

Uploaded CPython 3.13tmanylinux: glibc 2.17+ x86-64

hf_xet-1.3.2-cp313-cp313t-macosx_11_0_arm64.whl (3.5 MB view details)

Uploaded CPython 3.13tmacOS 11.0+ ARM64

hf_xet-1.3.2-cp313-cp313t-macosx_10_12_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.13tmacOS 10.12+ x86-64

hf_xet-1.3.2-cp37-abi3-win_arm64.whl (3.5 MB view details)

Uploaded CPython 3.7+Windows ARM64

hf_xet-1.3.2-cp37-abi3-win_amd64.whl (3.6 MB view details)

Uploaded CPython 3.7+Windows x86-64

hf_xet-1.3.2-cp37-abi3-musllinux_1_2_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.7+musllinux: musl 1.2+ x86-64

hf_xet-1.3.2-cp37-abi3-musllinux_1_2_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.7+musllinux: musl 1.2+ ARM64

hf_xet-1.3.2-cp37-abi3-manylinux_2_28_aarch64.whl (4.0 MB view details)

Uploaded CPython 3.7+manylinux: glibc 2.28+ ARM64

hf_xet-1.3.2-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.2 MB view details)

Uploaded CPython 3.7+manylinux: glibc 2.17+ x86-64

hf_xet-1.3.2-cp37-abi3-macosx_11_0_arm64.whl (3.5 MB view details)

Uploaded CPython 3.7+macOS 11.0+ ARM64

hf_xet-1.3.2-cp37-abi3-macosx_10_12_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.7+macOS 10.12+ x86-64

File details

Details for the file hf_xet-1.3.2.tar.gz.

File metadata

  • Download URL: hf_xet-1.3.2.tar.gz
  • Upload date:
  • Size: 643.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: maturin/1.12.4

File hashes

Hashes for hf_xet-1.3.2.tar.gz
Algorithm Hash digest
SHA256 e130ee08984783d12717444e538587fa2119385e5bd8fc2bb9f930419b73a7af
MD5 a4adbfefc9727494af806f59ec559720
BLAKE2b-256 8bcb9bb543bd987ffa1ee48202cc96a756951b734b79a542335c566148ade36c

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp314-cp314t-win_arm64.whl.

File metadata

  • Download URL: hf_xet-1.3.2-cp314-cp314t-win_arm64.whl
  • Upload date:
  • Size: 3.5 MB
  • Tags: CPython 3.14t, Windows ARM64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: maturin/1.12.4

File hashes

Hashes for hf_xet-1.3.2-cp314-cp314t-win_arm64.whl
Algorithm Hash digest
SHA256 4a6817c41de7c48ed9270da0b02849347e089c5ece9a0e72ae4f4b3a57617f82
MD5 79975805cb83345fc512cddabbeee34e
BLAKE2b-256 8be8a069edc4570b3f8e123c0b80fadc94530f3d7b01394e1fc1bb223339366c

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp314-cp314t-win_amd64.whl.

File metadata

  • Download URL: hf_xet-1.3.2-cp314-cp314t-win_amd64.whl
  • Upload date:
  • Size: 3.6 MB
  • Tags: CPython 3.14t, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: maturin/1.12.4

File hashes

Hashes for hf_xet-1.3.2-cp314-cp314t-win_amd64.whl
Algorithm Hash digest
SHA256 211f30098512d95e85ad03ae63bd7dd2c4df476558a5095d09f9e38e78cbf674
MD5 8a84911d91f130579c24ac823c505229
BLAKE2b-256 ab966ed472fdce7f8b70f5da6e3f05be76816a610063003bfd6d9cea0bbb58a3

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp314-cp314t-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp314-cp314t-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 91b1dc03c31cbf733d35dc03df7c5353686233d86af045e716f1e0ea4a2673cf
MD5 552faad75bdb049ae069115e7eb829ae
BLAKE2b-256 aee0831f7fa6d90cb47a230bc23284b502c700e1483bbe459437b3844cdc0776

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp314-cp314t-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp314-cp314t-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 3155a02e083aa21fd733a7485c7c36025e49d5975c8d6bda0453d224dd0b0ac4
MD5 3f04750a33b60c6c607f4988b00b4eed
BLAKE2b-256 0c099a3aa7c5f07d3e5cc57bb750d12a124ffa72c273a87164bd848f9ac5cc14

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp314-cp314t-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp314-cp314t-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 cfa760888633b08c01b398d212ce7e8c0d7adac6c86e4b20dfb2397d8acd78ee
MD5 f9149c8e27670966480d1563bee754fb
BLAKE2b-256 01406c4c798ffdd83e740dd3925c4e47793b07442a9efa3bc3866ba141a82365

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 959083c89dee30f7d6f890b36cdadda823386c4de63b1a30384a75bfd2ae995d
MD5 795ad7c9eabb0d3c411bf9607e895f47
BLAKE2b-256 b5dba75cf400dd8a1a8acf226a12955ff6ee999f272dfc0505bafd8079a61267

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp314-cp314t-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp314-cp314t-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 4bc995d6c41992831f762096020dc14a65fdf3963f86ffed580b596d04de32e3
MD5 2b02f5d5a6c877d90d15c4e2ac94cfcb
BLAKE2b-256 f28a08a24b6c6f52b5d26848c16e4b6d790bb810d1bf62c3505bed179f7032d3

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp314-cp314t-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp314-cp314t-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 c34e2c7aefad15792d57067c1c89b2b02c1bbaeabd7f8456ae3d07b4bbaf4094
MD5 7bc87e8e4aa58aa064b27ad6fc85b09f
BLAKE2b-256 ba759d54c1ae1d05fb704f977eca1671747babf1957f19f38ae75c5933bc2dc1

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp313-cp313t-win_arm64.whl.

File metadata

  • Download URL: hf_xet-1.3.2-cp313-cp313t-win_arm64.whl
  • Upload date:
  • Size: 3.5 MB
  • Tags: CPython 3.13t, Windows ARM64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: maturin/1.12.4

File hashes

Hashes for hf_xet-1.3.2-cp313-cp313t-win_arm64.whl
Algorithm Hash digest
SHA256 433c77c9f4e132b562f37d66c9b22c05b5479f243a1f06a120c1c06ce8b1502a
MD5 7f8b3e3b3747434db928990971adb305
BLAKE2b-256 057f724cfbef4da92d577b71f68bf832961c8919f36c60d28d289a9fc9d024d4

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp313-cp313t-win_amd64.whl.

File metadata

  • Download URL: hf_xet-1.3.2-cp313-cp313t-win_amd64.whl
  • Upload date:
  • Size: 3.6 MB
  • Tags: CPython 3.13t, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: maturin/1.12.4

File hashes

Hashes for hf_xet-1.3.2-cp313-cp313t-win_amd64.whl
Algorithm Hash digest
SHA256 31612ba0629046e425ba50375685a2586e11fb9144270ebabd75878c3eaf6378
MD5 aec84cae2872d86af8b922a5729a4cd5
BLAKE2b-256 a1c3859509bade9178e21b8b1db867b8e10e9f817ab9ac1de77cb9f461ced765

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp313-cp313t-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp313-cp313t-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 35b855024ca37f2dd113ac1c08993e997fbe167b9d61f9ef66d3d4f84015e508
MD5 7fbe07e40d202825da469a355fbf165e
BLAKE2b-256 e2e13af961f71a40e09bf5ee909842127b6b00f5ab4ee3817599dc0771b79893

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp313-cp313t-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp313-cp313t-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 1c88fbd90ad0d27c46b77a445f0a436ebaa94e14965c581123b68b1c52f5fd30
MD5 58f563adbd5af5698a63931c75b6aa76
BLAKE2b-256 63d7aecf97b3f0a981600a67ff4db15e2d433389d698a284bb0ea5d8fcdd6f7f

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp313-cp313t-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp313-cp313t-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 c1980abfb68ecf6c1c7983379ed7b1e2b49a1aaf1a5aca9acc7d48e5e2e0a961
MD5 1d34a0030703426f5ea10d7462337a42
BLAKE2b-256 6bb371e856bf9d9a69b3931837e8bf22e095775f268c8edcd4a9e8c355f92484

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 d6dbdf231efac0b9b39adcf12a07f0c030498f9212a18e8c50224d0e84ab803d
MD5 038d0ec08df813136f8f27af8dac194a
BLAKE2b-256 a85c7e4a33a3d689f77761156cc34558047569e54af92e4d15a8f493229f6767

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp313-cp313t-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp313-cp313t-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 c1ae4d3a716afc774e66922f3cac8206bfa707db13f6a7e62dfff74bfc95c9a8
MD5 0947e6dbb89787dc64a091b387a33d2d
BLAKE2b-256 3556987b0537ddaf88e17192ea09afa8eca853e55f39a4721578be436f8409df

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp313-cp313t-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp313-cp313t-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 335a8f36c55fd35a92d0062f4e9201b4015057e62747b7e7001ffb203c0ee1d2
MD5 3296b4d6ac3ca7f1afda89377c346ece
BLAKE2b-256 4975462285971954269432aad2e7938c5c7ff9ec7d60129cec542ab37121e3d6

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp37-abi3-win_arm64.whl.

File metadata

  • Download URL: hf_xet-1.3.2-cp37-abi3-win_arm64.whl
  • Upload date:
  • Size: 3.5 MB
  • Tags: CPython 3.7+, Windows ARM64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: maturin/1.12.4

File hashes

Hashes for hf_xet-1.3.2-cp37-abi3-win_arm64.whl
Algorithm Hash digest
SHA256 83d8ec273136171431833a6957e8f3af496bee227a0fe47c7b8b39c106d1749a
MD5 cfe81e7a3e321c562e96afef9ec5cc79
BLAKE2b-256 cc029a6e4ca1f3f73a164c0cd48e41b3cc56585dcc37e809250de443d673266f

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp37-abi3-win_amd64.whl.

File metadata

  • Download URL: hf_xet-1.3.2-cp37-abi3-win_amd64.whl
  • Upload date:
  • Size: 3.6 MB
  • Tags: CPython 3.7+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: maturin/1.12.4

File hashes

Hashes for hf_xet-1.3.2-cp37-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 9298b47cce6037b7045ae41482e703c471ce36b52e73e49f71226d2e8e5685a1
MD5 455d392a82ee9add9bbbb0646f090a9b
BLAKE2b-256 2b0b99bfd948a3ed3620ab709276df3ad3710dcea61976918cce8706502927af

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp37-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp37-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 06cdbde243c85f39a63b28e9034321399c507bcd5e7befdd17ed2ccc06dfe14e
MD5 9b3d4c9d77e29e4cb5720bc579905a2a
BLAKE2b-256 00b37bc1ff91d1ac18420b7ad1e169b618b27c00001b96310a89f8a9294fe509

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp37-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp37-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 305f5489d7241a47e0458ef49334be02411d1d0f480846363c1c8084ed9916f7
MD5 d4b6c3160a45c8a3d0b1f0c09b8652e8
BLAKE2b-256 7338600f4dda40c4a33133404d9fe644f1d35ff2d9babb4d0435c646c63dd107

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp37-abi3-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp37-abi3-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 06b724a361f670ae557836e57801b82c75b534812e351a87a2c739f77d1e0635
MD5 55d5693e3d66e1daffdddd64bf80f602
BLAKE2b-256 8cadbc7f41f87173d51d0bce497b171c4ee0cbde1eed2d7b4216db5d0ada9f50

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 7c2a054a97c44e136b1f7f5a78f12b3efffdf2eed3abc6746fc5ea4b39511633
MD5 dae93d929ed06d6a0b161e865285afbd
BLAKE2b-256 9dca907890ce6ef5598b5920514f255ed0a65f558f820515b18db75a51b2f878

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp37-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp37-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 a85d3d43743174393afe27835bde0cd146e652b5fcfdbcd624602daef2ef3259
MD5 6f506ee00c14d8b86619d6c08b0d9536
BLAKE2b-256 e471b99aed3823c9d1795e4865cf437d651097356a3f38c7d5877e4ac544b8e4

See more details on using hashes here.

File details

Details for the file hf_xet-1.3.2-cp37-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for hf_xet-1.3.2-cp37-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 f93b7595f1d8fefddfede775c18b5c9256757824f7f6832930b49858483cd56f
MD5 85422106fa6c2c6b80583c68a6028c14
BLAKE2b-256 d828dbb024e2e3907f6f3052847ca7d1a2f7a3972fafcd53ff79018977fcb3e4

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page