No project description provided
Project description
Datashack
What is this?
Create AWS big data streaming pipeline with Datashack sdk. For example:
Users = StreamingTable("users")
Users['id'] = Column('string')
Users['age'] = Column('int')
Users['name'] = Column('string')
run datashack plan/apply
to see the actual changes or actually applying them to your AWS account.
creates this pipeline:
- Provision Kinesis+Spark+Glue+Iam
- Automate schema evolution
- Tests
Pre-requisites
To work with this project, you will need to have the following software installed on your machine:
-
Node.js: You can download and install Node.js from the official website: https://nodejs.org/en/download/
-
Python: You can download and install Python from the official website: https://www.python.org/downloads/
-
Terraform and CDKTF: These tools are used for defining infrastructure as code. To install them, follow the instructions on the HashiCorp website: https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli. Additionally, you can install the CDKTF CLI by running the following command:
npm install --global cdktf-cli@latest
Make sure to run this command in your terminal or command prompt.
Getting Started
run in your terminal
pip install datashack-sdk
git clone https://github.com/datashack-dev/datashack-sdk-examples
datashack plan ./datashack-sdk-examples/my_app/models
Roadmap
- Additional AWS services
- Additional data sources and sinks, such as Apache Kafka or Elasticsearch
- 3rd Party sources - Google Analytics, Salesforce ...
- Spark based transformation and processing capabilities
- Integration with CICD workflows
- Integration with runtime applications
- Testing
- Automatic documentation generation
- More languages support for Datashack SDK: Go, Java, Yaml
Stay tuned
We are working on a fully funcional beta with many more features. Join here so we can ping you
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for datashack_sdk-0.0.8-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 19e4f273e1bf6a463b853c1417370037b18050f31d0d6f1a41b9ddabea7615f9 |
|
MD5 | 64642bbcbbcbb07d0c00fecae672efeb |
|
BLAKE2b-256 | 37c4fa49828a524a8dd30b1df16a62192d9d8c2ad9864671ec9eadccf7de6980 |