Spark support for Feast offline store
Project description
Feast Spark Offline Store plugin
This repo contains a plugin for feast to run an offline store on Spark.
It can be installed from pip and configured in the feature_store.yaml
configuration file to interface with DataSources
using Spark.
Note that this repository has not yet had a major release as it is still work in progress.
Contributing
We strongly encourage you to contribute to our repository. Find out more in our contribution guidelines
Requirements
Currently only supports Feast versions >=0.15.0
.
Installation
pip install feast-spark-offline-store
Or to install from source:
git clone git@github.com:Adyen/feast-spark-offline-store.git
cd feast-spark-offline-store
pip install -e '.[dev]'
Usage
Install feast
and feast_spark_offline_store
and change the Feast configurations in feature_store.yaml
to use feast_spark_offline_store.SparkOfflineStore
:
project: example_feature_repo
registry: data/registry.db
provider: local
online_store:
...
offline_store:
type: feast_spark_offline_store.spark.SparkOfflineStore
spark_conf:
spark.master: "local[*]"
spark.ui.enabled: "false"
spark.eventLog.enabled: "false"
spark.sql.catalogImplementation: "hive"
spark.sql.parser.quotedRegexColumnNames: "true"
spark.sql.session.timeZone: "UTC"
Documentation
See Feast documentation on offline stores and adding custom offline stores.
License
MIT license. For more information, see the LICENSE file.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for feast_spark_offline_store-0.0.5.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | d8cfecbe8667cf61b109076059983853d59e9d286de6e5a098a435ec1891a018 |
|
MD5 | bec78869d8c5c783fdf807aafe13319a |
|
BLAKE2b-256 | e38d1b4d35416ee92e2d07101cd18f3748d4bf5485997cb1e341f2ee9a2c3184 |
Hashes for feast_spark_offline_store-0.0.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ecc1c93bcc66983ccd244e260434d192ef654c05e17329c934e504b3a3d164a4 |
|
MD5 | ea5e5b80c78c12e6a6b06a419be96585 |
|
BLAKE2b-256 | 9a7319029085ca9fc56a65b1c0319359144292a93ccd74aabc56467fd24b9714 |