A scripting language to simply manage a very large amount of i/o heavy workloads. Such as API calls for your ETL, ELT or any program needing Python and/or SQL
Project description
Buelon
A scripting language to simply manage a very large amount of i/o heavy workloads. Such as API calls for your ETL, ELT or any program needing Python and/or SQL
Table of Contents
Installation
pip install buelon
That's it!
This will install the cli command bue
. Check install by running bue --version
or bue -v
Note:
This package uses Cython and you may need to install python3-dev
using
sudo apt-get install python3-dev
[more commands and information].
If you would like to use this repository without Cython,
you may git clone
since it is not technically dependent on
these scripts, but they do provide a significant performance boost.
Quick Start
- Get example template:
bue example
(warning: this command will over-write.env
) - Start Bucket server, Hub and 3 workers:
bue demo
- Upload script and wait for results:
python3 example.py
Production Start
Security: Make sure bucket, hub and workers are under
a private network only
(you will need a web server or something similar
under the same private network
to access this tool using bue upload
)
- Run bucket server:
bue bucket -b 0.0.0.0:61535
- Run hub:
bue hub -b 0.0.0.0:65432 -k localhost:61535
- Run worker(s):
bue worker -b localhost:65432 -k localhost:61535
- Upload code:
bue upload -b localhost:65432 -f ./example.bue
Supported Languages
- Python
- SQLite3
- PostgreSQL
Learn by Example
# IMPORTANT: tabs are 4 spaces. white_space == " "
# setting scopes is how you make new jobs with errors
# not slow down your servers by setting them to a lower scope.
# And/or how you handle running heavy processes on large machine
# and small process on small machines
$ production-small
# define a job called `accounts`
accounts:
python # <-- select the language to be run. currently only python, sqlite3 and postgres are currently available
accounts # select the function(for python) or table(for sql) name that will be used
example.py # either provide a file or write code directly using the "`" char (see below example)
request:
python
request_report
example.py
status:
python
$ testing-small # <-- "scope" for a single step. A lower scope will be given less priority over higher scopes. See PIPE_WORKER_SCOPES in `.env` file generated by `bue example`
get_status
example.py
download:
python
!9 # <-- "priority" higher numbers are more important and run first within their scope.
get_report
example.py
manipulate_data:
sqlite3
some_table # *vvvv* see below for writing code directly *vvvv*
`
SELECT
*,
CASE
WHEN sales = 0
THEN 0.0
ELSE spend / sales
END AS acos
FROM some_table
`
## this one's just to show postgres as well
#manipulate_data_again:
# postgres
# another_table
# `
#select
# *,
# case
# when spend = 0
# then 0.0
# else sales / spend
# end AS roas
#from another_table
#`
upload:
python
upload_to_db
example.py
# these are pipes and what will tell the server what order to run the steps
# and also transfer the returned data between steps
# each step will be run individually and could be run on a different computer each time
accounts_pipe = | accounts # single pipes currently need a `|` before or behind the value
api_pipe = request | status | download | manipulate_data | upload
# currently there are only two syntax's for "running" pipes.
# either by itself:
# pipe()
#
# or in a loop:
# for value in pipe1():
# pipe2(value)
# # Another Example:
# v = pipe(accounts_pipe) # <-- single call
# pipe2(v)
# right not you cannot pass arguments within the pipe being used for the for loop.
# in this case `accounts_pipe()` cannot be `accounts_pipe(some_value)`
for account in accounts_pipe():
api_pipe(account)
Plans
If this projects sees some love, or I just find more free time, I'd like to support more languages. Even compiled languages such as rust
, go
and c++
. Allowing teams that write different languages to work on the same program.
Better bue script errors
License
- MIT License
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for buelon-1.0.11-cp311-cp311-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | deaf0ff27423427f18be896a6ffc9c4bc4b412b66947e1c0e447dbc0af420fe4 |
|
MD5 | f2d6263aa422e1cfaa00276cdf85b8f8 |
|
BLAKE2b-256 | 4eca5b79a3fd355890d51c378403b4219a69a74588b20665e18d77cc1bb80c9f |