Intake single YAML hierarchical catalog.
Project description
Welcome to Intake DAL (data access layer) plugin
This Intake plugin helps abstract a dataset over disparate storage systems (eg: bulk, streaming, serving, …). It also provides an easy way to specialize a hierarchical catalog to a default DAL storage system.
Sample Catalog source entry:
user_events:
driver: dal
args:
default: 'local'
storage:
local: 'csv://{{ CATALOG_DIR }}/data/user_events.csv'
serving: 'in-memory-kv://foo'
batch: 'parquet://{{ CATALOG_DIR }}/data/user_events.parquet'
Example code using sample catalog:
# Specialize the catalog dal default storge mode datasources
# to be "serving".
cat = DalCatalog(path, storage_mode="serving")
# reads from the serving storage system
# using the in-memory-kv Intake plugin
df = cat.user_events.read()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
intake-dal-0.1.5.tar.gz
(16.0 kB
view hashes)
Built Distribution
intake_dal-0.1.5-py3-none-any.whl
(19.2 kB
view hashes)
Close
Hashes for intake_dal-0.1.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 13e6b0ea5a5632d888a79670aaa8b5cd49e363c6ca7e42ea31e4f015be1dc10f |
|
MD5 | 8c1a58985bbeeced7a954d10348a1741 |
|
BLAKE2b-256 | 3316818fbc48effafcbcf6fdbd9794104fc10eb19d191db3675870689f24f7e8 |