No project description provided
Project description
Harmonizer
Harmonizer is a library we have developed at Dune to translate Dune queries from PostgreSQL and Spark SQL to DuneSQL. We currently use this library in our migration service in the app.
A query is translated through two steps:
- We use SQLGlot to transpile the query. This is an excellent tool which parses a SQL query into an Abstract Syntax Tree (AST), and then translates it to a different dialect. We use it to translate from Spark SQL to DuneSQL, and from PostgreSQL to DuneSQL.
- We pass the query through custom rules to make additional changes to the query. Examples of such rules are
- mapping known changes in table names from the legacy Postgres datasets to corresponding table names in DuneSQL
- translating string literals '0x...' to 0x... in DuneSQL, since we support native hex literals.
Getting started
Install with
pip install dune-harmonizer
Now import the translate_
functions in your code:
from dune.harmonizer import translate_spark, translate_postgres
with function signatures
def translate_spark(query: str) -> str:
...
def translate_postgres(query: str, dataset: str) -> str:
...
Contributing
Contributions are very welcome!
Please open an issue or PR, and we will get back to you as soon as we can.
If you've found a table that doesn't get mapped to one that exists on Dune SQL, then you can open an issue or just add the table mapping to this line here in a PR.
If there is a function that doesn't get mapped correctly, then you can open an issue or try and add one here using sqlglot and open a PR.
Development
Install with
poetry install
If the Ruff linter complains, running the following and committing the changes should suffice
poetry run ruff . --fix
poetry run black .
Run tests with
poetry run pytest
We test on examples in the test_cases
directory.
To force an update of the expected outputs, run the update_expected_outputs
script like below
poetry run python tests/update_expected_outputs.py
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dune_harmonizer-0.3.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 807414958b92f6d0762b9c46a6214f2ad7dbe9e78b094d4e6c49c93ce21e6fef |
|
MD5 | 2eda392d4afe8dd3738d0b6f8a7fd9b9 |
|
BLAKE2b-256 | 86fcd242cf4a8dfc046f3e4bc3fbe41a11b889f3d1af6a30e634836d5dd75192 |