Skip to main content

Databricks optimize utility

Library to generate more easily optimize statement based on existing data.

Build from source

Build the wheel file is enough to run

python -m build --wheel

Table optimizer

This object can be initialized with a table object name (with three level namespace "catalog.schema.table"), the method pre_optimization will run some statistics to identify the best column to run z-order (column with highest cardinality) and to re-partition (this depends on the size of the table and the size of the generated files partition). This is used to define and print the optimization query. Then to run the optimization itself we have two additional method, run_optimize and run_partition, these two can be executed even on their own without the need to run the pre_optimization, in this case the method will run the statistics and the generation separately.

from dbks_optimize.optimizer import TableOptimizer

opt = TableOptimizer(spark,"testing.bakehouse.sales_customers",force_partition_on_col='customerID')

opt.pre_optimization() #print out the optimization statement

opt.run_optimizer() #execute optimize on the table

opt.run_partition() #execute partitioning query (it clone i a new table and overwrite the existing one)

Schema optimizer

This object leverage the existing class for single table object and iterate over list of table available computing table statistics. pre_optimize is used to generate the optimize statement, while to run real optimization we can use run_db_optimization.

from dbks_optimize.optimizer import SchemaOptimizer

opt = SchemaOptimizer(spark,'testing.bakehouse')

opt.pre_optimization()

opt.run_db_optimization()

Catalog optimizer

Using the same logic iterate over all schemas and for each on on all tables to generate the statement.

from dbks_optimize.optimizer import CatalogOptimizer

opt = CatalogOptimizer(spark,'testing')

opt.pre_optimization()

opt.run_catalog_optimization()

Metadata

Release files for databricks-optimize 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for databricks-optimize 0.2.2
File Size Uploaded Provenance Actions
databricks_optimize-0.2.2.tar.gz 5.7 kB Without provenance Details for databricks_optimize-0.2.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for databricks-optimize 0.2.2
File Interpreter ABI Platform Actions
databricks_optimize-0.2.2-py3-none-any.whl Python 3 none any Details for databricks_optimize-0.2.2-py3-none-any.whl

Total release size: 12.1 kB

Release files / databricks_optimize-0.2.2.tar.gz

Download URL databricks_optimize-0.2.2.tar.gz
Size 5.7 kB
Tags Source
SHA-256 checksum
How to use checksums
1fe4faf391322d10297639959bc7a83908986501637b9a63513fe12394012e12
BLAKE2b-256 checksum
How to use checksums
c2ad55ea77db051e3d2726fc3c150f148b21924cc0a9eca4de2b5f72afbbaa41
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.7

Release files / databricks_optimize-0.2.2-py3-none-any.whl

Download URL databricks_optimize-0.2.2-py3-none-any.whl
Size 6.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
06965b532d22a0ebd8fdaa7ea58b9f8652244bac27f19da8687f726c46f87c63
BLAKE2b-256 checksum
How to use checksums
d5ae9d2a69ee81258fb6310653bd727bf1639b4e30029b7bbf895798aded8eae
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.7

Security for databricks-optimize 0.2.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.2 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page