databricks-bundle 0.5.10b2

Databricks bundle for the Pyfony framework

Databricks bundle

This bundle allows you to create simple functional Databricks pipelines, which can be easily auto-documented and unit-tested. It is part of the Bricksflow framework.

Installation

Install the bundle via Poetry:

$ poetry add databricks-bundle && poetry add databricks-connect --dev && poetry add py4j --dev

Add the DatabricksBundle.autodetect() to your application's Kernel.py to activate it:

from pyfony.kernel.BaseKernel import BaseKernel
from databricksbundle.DatabricksBundle import DatabricksBundle

class Kernel(BaseKernel):
    
    def _registerBundles(self):
        return [
            # ...
            DatabricksBundle.autodetect(),
            # ...
        ]

Usage

1. Writing functional pipelines
2. Recommended pipelines structure
3. Configuring pipelines
4. Using dependencies
5. Databricks Connect setup