MetaStore
Project description
PoC for MetaStore
You need to prepare two things to make ake it running
- a kaffka in the local and a created
dream-1
topic - a s3 credential to access the storage
Setup MetaStore API server
Run the docker-compose to setup meta-store API server
docker-compose up -d
Currently, the API server only contains Kafka, PostgreSQL, and Hasura.
- Kafka: Serve for event subscription
- PostgreSQL: Store all the relationship
- Hasura: Dev tool for PostgreSQL and serve GraphQL endpoint
helloworld.py
It is a sample to write the event handler
@on('dataset', event='updated', filter={'name': 'demo-dataset-1'})
def on_dataset_available():
print(f'on_dataset_available invoked')
dataset = project.get('dataset', 'demo-dataset-1')
model = project.create('model', 'model-1', {'dataset': dataset})
model.update()
return
on
wait for a dataset artifact increated
event and filtering byname=dataset-1
- after training, we create
Model
and send a message to notify other handlers
send.event.py
# in PoC, we don't handle the kafka
# the Project object is a facade to everything {event sender, event listener runner, access to object store}
project = Project('dream-1', profile='local')
dataset = project.get('dataset', 'demo-dataset-1')
if not dataset:
dataset = project.create('dataset', 'demo-dataset-1')
dataset.update()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for infuseai_metastore-0.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d09a5dd065b16f6769ebf7d31ec11eaa3c7e30ac69acc11d86b8f75d5f59df93 |
|
MD5 | 6eb55d4049106f67c1de885f6ea6c422 |
|
BLAKE2b-256 | 18307f5843e144e5d97ae87905ffaf09b649eeef4ad714139e4efbf6f2608a5c |