The shift-left CLI to manage your shift-left project
Project description
Shift Left Utils
A comprehensive toolkit for migrating SQL batch processing to real-time Apache Flink on Confluent Cloud, with AI-powered code translation and advanced pipeline management. It also addresses KsqlDB to Apache Flink SQL migration.
Installation
pip install shift_left
Features
SQL Migration & Translation: Automatically migrate KSQL, Spark SQL, and DBT code to Flink SQL using LLM-based agents with validation and refinement capabilities.
Pipeline Management: Build, validate, and deploy Flink SQL pipelines with dependency management, execution planning, and blue-green deployment strategies.
Project Structure: Scaffold and manage Flink projects following medallion architecture (sources, intermediates, dimensions, facts, views) with comprehensive metadata and testing frameworks.
Test Harness: Develop unit test SQL templates with synthetic data to unit test Flink SQL statements.
Quick Start
# Initialize a new Flink project
shift_left project init my_project ./my_project --project-type kimball
# Build table inventory
shift_left table build-inventory ./pipelines
# Deploy a pipeline
shift_left pipeline deploy ./pipelines --table-name my_table
CLI Commands
Project Management
shift_left project init- Initialize a new Flink projectshift_left project validate-config- Validate configuration fileshift_left project list-topics- List Kafka topicsshift_left project list-compute-pools- List Flink compute pools
Table Management
shift_left table init- Create table structureshift_left table build-inventory- Build table inventoryshift_left table migrate- Migrate SQL with AI assistance
Pipeline Deployment
shift_left pipeline deploy- Deploy Flink pipelinesshift_left pipeline build-metadata- Build pipeline metadata
Testing
shift_left table init-unit-tests- Create test file for unit testsshift_left table run-unit-tests- Run the test suiteshift_left table delete-unit-tests- Remove unit test artifacts
Documentation
Requirements
- Python 3.12+
- Confluent Cloud account (for deployment features)
License
Apache License 2.0
Links
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file shift_left-0.1.48.tar.gz.
File metadata
- Download URL: shift_left-0.1.48.tar.gz
- Upload date:
- Size: 748.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.9.22 {"installer":{"name":"uv","version":"0.9.22","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
66ef540bf7d89b5fdd031ebf57818d11c281b3e2136ce8621506c1b37f7fa26d
|
|
| MD5 |
ef031ff63b5e3c4ed2dcbddf3f007007
|
|
| BLAKE2b-256 |
68c268a33e8397e91b0a31bbec720003f6f990e05c0a01bd34636903759f1d12
|
File details
Details for the file shift_left-0.1.48-py3-none-any.whl.
File metadata
- Download URL: shift_left-0.1.48-py3-none-any.whl
- Upload date:
- Size: 199.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.9.22 {"installer":{"name":"uv","version":"0.9.22","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
547b9c3278e40f93e4b7aa047de8f7f7168670339b9dc86a8869bee6e4972dfb
|
|
| MD5 |
84d16bd0aa84fbf14a23c8f81a9ed861
|
|
| BLAKE2b-256 |
64667cdd9f7bfffe048c0f7798272c2d11bf31b5285e95a33d6e16914d390222
|
