low-level plubming to read/write Quilt packages
Project description
QuiltCore
QuiltCore is a library for building and running Quilt data packages. It is designed to leverage standard open source technology and YAML configuration files so that it can easily be ported to other languages and platforms.
This initial implementation is in Python.
Key Technologies
- Apache Arrow for reading, writing, and representing manifests
- PyArrow for Python bindings to Arrow
- fsspec filesystems for reading and writing files from various sources
- PyYAML for reading and writing YAML configuration files
Example
poetry install
#!/usr/bin/env python
import os
from quiltcore import Domain, UDI
from tempfile import TemporaryDirectory
from upath import UPath
TEST_BKT = "s3://quilt-example"
TEST_PKG = "akarve/amazon-reviews"
TEST_TAG = "1570503102"
TEST_HASH = "ffe323137d0a84a9d1d6f200cecd616f434e121b3f53a8891a5c8d70f82244c2"
TEST_KEY = "camera-reviews"
WRITE_BKT = os.environ.get("WRITE_BUCKET")
SOURCE_URI = f"quilt+{TEST_BKT}#package={TEST_PKG}:{TEST_TAG}"
DEST_URI = f"quilt+{TEST_BKT}#package={TEST_PKG}:{TEST_TAG}"
Get Manifest
remote = UDI.FromUri(SOURCE_URI)
print(f"remote: {remote}")
with TemporaryDirectory() as tmpdir:
local = UPath(tmpdir)
domain = Domain.FromLocalPath(local)
print(f"domain: {domain}")
folder = domain.pull(remote)
print(f"folder: {folder}")
if WRITE_BKT:
tag = domain.push(folder, remote=UDI.FromUri(DEST_URI))
print(f"tag: {tag}")
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
quiltcore-0.6.3.tar.gz
(26.4 kB
view hashes)
Built Distribution
quiltcore-0.6.3-py3-none-any.whl
(34.8 kB
view hashes)
Close
Hashes for quiltcore-0.6.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5a777a5cd94afd293d48682317232712e592bdc2fb081837b4adb342a0e7c172 |
|
MD5 | 3453dd585c7717329f8881ff786548f8 |
|
BLAKE2b-256 | 9fa52a1c246d67e4a879dc57b218a13ed400f573fcd6b7a6024cbd61356599cf |