PackYak 
Packyak AWS CDK
PackYak is a next-generation framework for building and deploying Data Lakehouses in AWS with a Git-like versioned developer workflow that simplifies how Data Scientists and Data Engineers collaborate.
It enables you to deploy your entire Data Lakehouse, ETL and Machine Learning platforms on AWS with no external dependencies, maintain your Data Tables with Git-like versioning semantics and scale data production with Dagster-like Software-defined Asset Graphs.
It combines 5 key technologies into one framework that makes scaling Data Lakehouses and Data Science teams dead simple:
- Git-like versioning of Data Tables with Project Nessie - no more worrying about the version of data, simply use branches, tags and commits to freeze data or roll back mistakes.
- Software-defined Assets (as seen in Dagster) - think of your data pipelines in terms of the data it produces. Greatly simplify how data is produced, modified over time and backfilled in the event of errors.
- Infrastructure-as-Code (AWS CDK and Pulumi) - deploy in minutes and manage it all yourself with minimal effort.
- Apache Spark - write your ETL as simple python processes that are then scaled automatically over a managed AWS EMR Spark Cluster.
- Streamlit - build Streamlit applications that integrate the Data Lakehouse and Apache Spark to provide interactive reports and exploratory tools over the versioned data lake.
Get Started
Install Docker
If you haven't already, install Docker.
Install Python Poetry & Plugins
# Install the Python Poetry CLI
curl -sSL https://install.python-poetry.org | python3 -
# Add the export plugin to generate narrow requirements.txt
poetry self add poetry-plugin-export
Install the packyak CLI:
pip install packyak
Create a new Project
packyak new my-project
cd ./my-project
Deploy to AWS
poetry run cdk deploy
Git-like Data Catalog (Project Nessie)
PackYak comes with a Construct for hosting a Project Nessie catalog that supports Git-like versioning of the tables in a Data Lakehouse.
It deploys with an AWS DynamoDB Versioned store and an API hosted in AWS Lambda or AWS ECS. The Nessie Server is stateless and can be scaled easily with minimal-to-zero operational overhead.
Create a NessieDynamoDBVersionStore
from packyak.aws_cdk import DynamoDBNessieVersionStore
versionStore = DynamoDBNessieVersionStore(
scope=stack,
id="VersionStore",
versionStoreName="my-version-store",
)
Create a Bucket to store Data Tables (e.g. Parquet files). This will store the "Repository"'s data.
myRepoBucket = Bucket(
scope=stack,
id="MyCatalogBucket",
)
Create the Nessie Catalog Service
# hosted on AWS ECS
myCatalog = NessieECSCatalog(
scope=stack,
id="MyCatalog",
vpc=vpc,
warehouseBucket=myRepoBucket,
catalogName=lakeHouseName,
versionStore=versionStore,
)
Create a Branch
Branch off the main branch of data into a dev branch to "freeze" the data as of a particular commit
CREATE BRANCH dev FROM main
Deploy a Spark Cluster
Create an EMR Cluster for processing data
spark = Cluster(
scope=stack,
id="Spark",
clusterName="my-cluster",
vpc=vpc,
catalogs={
# use the Nessie Catalog as the default data catalog for Spark SQL queries
"spark_catalog": myCatalog,
},
installSSMAgent=true,
)
Configure SparkSQL to be served over JDBC
sparkSQL = spark.jdbc(port=10001)
Deploy Streamlit Site
Stand up a Streamlit Site to serve interactive reports and applications over your data.
site = StreamlitSite(
scope=stack,
# Point it at the Streamlit site entrypoint
home="app/home.py",
# Where the Streamlit pages/tabs are, defaults to `dirname(home)/pages/*.py`
# pages="app/pages"
)
Deploy to AWS
packyak deploy
Or via the AWS CDK CLI:
poetry run cdk deploy
1 maintainer
Metadata
1 maintainer
Release files for packyak-aws-cdk 0.4.37
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| packyak-aws-cdk-0.4.37.tar.gz | 500.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| packyak_aws_cdk-0.4.37-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 999.6 kB
Release files / packyak-aws-cdk-0.4.37.tar.gz
| Download URL | packyak-aws-cdk-0.4.37.tar.gz |
|---|---|
| Size | 500.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b5facfaed1ace4d5d4e6aef99879e342eae11e13904516cd0da8091cd7b34d3d
|
|
BLAKE2b-256 checksum How to use checksums |
55bad72fe4f46151640b083e41b4a3d027290000761da77f457c903263668162
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.10.13
|
Release files / packyak_aws_cdk-0.4.37-py3-none-any.whl
| Download URL | packyak_aws_cdk-0.4.37-py3-none-any.whl |
|---|---|
| Size | 499.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9fdcdf5e3720491eeac0cfd02b55d66c8e6218949683f6d507d3834c04a24908
|
|
BLAKE2b-256 checksum How to use checksums |
f077029f0408fb233f265d0465fd09f6381f07d357a616d0db6f97b6d4b76d7c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.10.13
|
