Databricks optimize utility
Library to generate more easily optimize statement based on existing data.
Build from source
Build the wheel file is enough to run
python -m build --wheel
Table optimizer
This object can be initialized with a table object name (with three level namespace "catalog.schema.table"), the method pre_optimization will run some statistics to identify the best column to run z-order (column with highest cardinality) and to re-partition (this depends on the size of the table and the size of the generated files partition). This is used to define and print the optimization query. Then to run the optimization itself we have two additional method, run_optimize and run_partition, these two can be executed even on their own without the need to run the pre_optimization, in this case the method will run the statistics and the generation separately.
from dbks_optimize.optimizer import TableOptimizer
opt = TableOptimizer(spark,"testing.bakehouse.sales_customers",force_partition_on_col='customerID')
opt.pre_optimization() #print out the optimization statement
opt.run_optimizer() #execute optimize on the table
opt.run_partition() #execute partitioning query (it clone i a new table and overwrite the existing one)
Schema optimizer
This object leverage the existing class for single table object and iterate over list of table available computing table statistics. pre_optimize is used to generate the optimize statement, while to run real optimization we can use run_db_optimization.
from dbks_optimize.optimizer import SchemaOptimizer
opt = SchemaOptimizer(spark,'testing.bakehouse')
opt.pre_optimization()
opt.run_db_optimization()
Catalog optimizer
Using the same logic iterate over all schemas and for each on on all tables to generate the statement.
from dbks_optimize.optimizer import CatalogOptimizer
opt = CatalogOptimizer(spark,'testing')
opt.pre_optimization()
opt.run_catalog_optimization()
Metadata
Release files for databricks-optimize 0.2.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| databricks_optimize-0.2.2.tar.gz | 5.7 kB | Without provenance | Details for databricks_optimize-0.2.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| databricks_optimize-0.2.2-py3-none-any.whl | Python 3 | none | any | Details for databricks_optimize-0.2.2-py3-none-any.whl |
Total release size: 12.1 kB
Release files / databricks_optimize-0.2.2.tar.gz
| Download URL | databricks_optimize-0.2.2.tar.gz |
|---|---|
| Size | 5.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1fe4faf391322d10297639959bc7a83908986501637b9a63513fe12394012e12
|
|
BLAKE2b-256 checksum How to use checksums |
c2ad55ea77db051e3d2726fc3c150f148b21924cc0a9eca4de2b5f72afbbaa41
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Release files / databricks_optimize-0.2.2-py3-none-any.whl
| Download URL | databricks_optimize-0.2.2-py3-none-any.whl |
|---|---|
| Size | 6.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
06965b532d22a0ebd8fdaa7ea58b9f8652244bac27f19da8687f726c46f87c63
|
|
BLAKE2b-256 checksum How to use checksums |
d5ae9d2a69ee81258fb6310653bd727bf1639b4e30029b7bbf895798aded8eae
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Security for databricks-optimize 0.2.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.