Kiwix S3 Cache wrapper to use within Kiwix/OpenZIM projects
Project description
kiwixstorage
helpers for S3 storage, autoconf from URL + Wasabi (wasabisys.com) extras
Goal is mainly to provide a configured s3.client and s3.resource from an URL Users could limit usage to this and use boto3 directly from there.
A few additional wrappers are in place to simplify common actions. Also, non-S3, wasabi-specific features are exposed directly.
Usage
pip install kiwixstorage
Connection
from kiwixstorage import KiwixStorage
url = "https://s3.us-east-1.wasabisys.com/?keyId=x&secretAccessKey=y&bucketName=z"
s3 = KiwixStorage(url)
# test credentials and ensure we can list buckets
if not s3.check_credentials(list_buckets=True, failsafe=True):
return # bad auth
Scraper use-case
online_url = "https://xxx"
fpath = "/local/path.ext"
# retrieve origin etag
etag = requests.head(online_url, allow_redirects=True).headers.get("Etag")
# check if we have that very same version in store
if s3.has_matching_object(key=url, etag=etag)
# lastest version in our store, download from there (using progress output)
s3.download_file(key=url, fpath=fpath, progress=True)
else:
# download the origin file using your regular tools
download_file(url, fpath)
# upload it our storage
s3.upload_file(fpath=fpath, key=url)
# now you have a local file of lastest version and the storage is up to date
Other use cases
# create a bucket
bucket = s3.create_bucket("bucket_name")
# set auto-delete on bucket
s3.set_bucket_autodelete_after(nb_days=7)
# allow public downloads from bucket
s3.allow_public_downloads_on()
# upload a file
s3.upload_file(fpath, "some/path/file.img", meta={"ENCODER_VERSION": "v1"})
# set autodelete on specific file
s3.set_object_autodelete_on(key, datetime.datetime.now())
# download a file
s3.download_file(key, fpath)
# get URL for external download
s3.get_download_url(key)
Resources:
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
kiwixstorage-0.6.tar.gz
(34.5 kB
view hashes)
Built Distribution
kiwixstorage-0.6-py3-none-any.whl
(34.2 kB
view hashes)
Close
Hashes for kiwixstorage-0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 135d3ddad204dd8706cc413675fc792897876239ed7e10ada9d6548096f7bed5 |
|
MD5 | e01f37a0ff704e6693cececc6da0cfe9 |
|
BLAKE2b-256 | 9b0dd5dc417b14f24f531112c48baa6a64efdcb1d7361fd5131b95c3594e468a |