unitycatalog-migrate 0.1.0

Migrate catalogs from Databricks to Unity Catalog

Unity Catalog Migrate

Migrate catalogs, schemas, and tables from Databricks to Unity Catalog.

Requirements

• Python 3.9 or higher
• Databricks configuration file: https://docs.databricks.com/en/dev-tools/auth/config-profiles.html

Installation

To install the Unity Catalog Migrator, you can use the following commands:

pip install unitycatalog-migrate

Usage

Migrate Catalogs

ucm migrate-catalog NAMES... --profile <databricks-profile> 

Migrate Schemas

ucm migrate-schema FULL_NAMES... --profile <databricks-profile> 

where FULL_NAMES are in the format catalog.schema

Migrate Tables

ucm migrate-table FULL_NAMES..  --profile <databricks-profile>  

where FULL_NAMES are in the format catalog.schema.table

Configuration

The Unity Catalog Migrator uses the following environment variables:

• UC_HOST_URL: The URL of the Unity Catalog server. Default is http://localhost:8080/api/2.1/unity-catalog.
• UC_TOKEN: The token to authenticate with the Unity Catalog server. Default is None. You can also pass the token as an argument to the command.

Example

# Use Databricks CLI to get tables 
table_names=$(databricks tables list catalog1 schema1 --profile DATABRICKS_TEST | awk 'NR>1 {print $1}' | paste -sd ' ' -)

# Migrate the tables to Unity Catalog using ucm
echo $table_names | xargs ucm migrate-table --profile DATABRICKS_TEST 

Remarks

• Catalog and Schema need to exist in Unity Catalog before migrating tables

Not supported:

• system tables
• Variant datatype

Contributing

• Contributions are welcome. Please fork and make a PR and I'll take a look asap.
• Star the repo if you like it.