MetaStore
Project description
PoC for MetaStore
You need to prepare two things to make ake it running
- a kaffka in the local and a created
dream-1
topic - a s3 credential to access the storage
Setup MetaStore API server
Run the docker-compose to setup meta-store API server
docker-compose up -d
Currently, the API server only contains Kafka, PostgreSQL, and Hasura.
- Kafka: Serve for event subscription
- PostgreSQL: Store all the relationship
- Hasura: Dev tool for PostgreSQL and serve GraphQL endpoint
helloworld.py
It is a sample to write the event handler
@on('dataset', event='updated', filter={'name': 'demo-dataset-1'})
def on_dataset_available():
print(f'on_dataset_available invoked')
dataset = project.get('dataset', 'demo-dataset-1')
model = project.create('model', 'model-1', {'dataset': dataset})
model.update()
return
on
wait for a dataset artifact increated
event and filtering byname=dataset-1
- after training, we create
Model
and send a message to notify other handlers
send.event.py
# in PoC, we don't handle the kafka
# the Project object is a facade to everything {event sender, event listener runner, access to object store}
project = Project('dream-1', profile='local')
dataset = project.get('dataset', 'demo-dataset-1')
if not dataset:
dataset = project.create('dataset', 'demo-dataset-1')
dataset.update()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for infuseai_metastore-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 875048d4b107bfac929d62a57e9ad9a2e8c03ac8eccb67423ac489708ede9033 |
|
MD5 | 1257ec941f6d2a7d6a3699c90e756cb4 |
|
BLAKE2b-256 | 03a75de28a95ec96217f96a00f8ba9405a4ff3a3fad4b6c4f5486091f00d3b39 |