quiltcore 0.1.1

next-generation package manager for universal data

QuiltCore

QuiltCore is a library for building and running Quilt data packages. It is designed to leverage standard open source technology and YAML configuration files so that it can easily be ported to other languages and platforms.

This initial implementation is in Python.

Key Technologies

• Apache Arrow for reading, writing, and representing manifests
  • PyArrow for Python bindings to Arrow
• fsspec filesystems for reading and writing files from various sources
• PyYAML for reading and writing YAML configuration files

Example

poetry install

from quiltcore import CoreRegistry
from tempfile import TemporaryDirectory
from upath import UPath

TEST_BKT = "s3://quilt-example"
TEST_PKG = "akarve/amazon-reviews"
TEST_TAG = "1570503102"
TEST_HASH = "ffe323137d0a84a9d1d6f200cecd616f434e121b3f53a8891a5c8d70f82244c2"
TEST_KEY = "camera-reviews"

Get Manifest

registry = CoreRegistry(UPath(TEST_BKT))
named_package = registry.get(TEST_PKG)
manifest = named_package.get(TEST_TAG)
blob = manifest.get(TEST_KEY)

Get Object

with TemporaryDirectory() as tmpdirname:
  dest = UPath(tmpdirname) / TEST_KEY
  local = blob.put(dest)
  print(local)
  assert local.exists()
  local_bytes = local.read_bytes()
  assert blob.verify(local_bytes)