dataform-checklist 1.0.0

Generate Excel deployment checklists from Dataform repositories

Dataform Deployment Checklist Exporter

A Python package for generating Excel deployment checklists from Dataform repositories with automatic status tracking.

Features

• 🔍 Automatically scans all .sqlx files in Dataform repositories
• 📊 Exports to formatted Excel with color-coded status
• 🔄 Tracks changes: New, Existed, Deleted files
• 🎨 Professional Excel formatting with conditional colors
• 📦 Easy installation and distribution
• 🚀 Command-line interface

Installation

From Source

cd scripts
pip install -e .

For Distribution (Create wheel)

cd scripts
python setup.py sdist bdist_wheel
pip install dist/dataform-checklist-1.0.0-py3-none-any.whl

Install on Another Machine

# Copy the wheel file to target machine, then:
pip install dataform-checklist-1.0.0-py3-none-any.whl

Usage

Command Line

After installation, use the dataform-checklist command:

# Basic usage
dataform-checklist /path/to/dataform/repo

# With version
dataform-checklist /path/to/dataform/repo -v v1.01.00

# With custom output path
dataform-checklist /path/to/dataform/repo -o /output/path -v v1.02.00

# With custom repository name
dataform-checklist /path/to/dataform/repo -n MyRepo -v v2.00.00

# Full options
dataform-checklist /path/to/repo -o /output -n WWIM -v v1.00.00

As Python Library

from dataform_checklist import DataformInventoryExporter

# Create exporter
exporter = DataformInventoryExporter(
    repository_path="/path/to/dataform/repo",
    output_path="/output/path",
    repository_name="WWIM",
    version="v1.00.00"
)

# Run export
exporter.run()

# Or use individual methods
files = exporter.scan_sqlx_files()
inventory = exporter.create_new_inventory(files)
exporter.export_to_excel(inventory)

Output

Filename Format

PTTEP_Deployment_Checklist_{repository_name}_{version}.xlsx

Excel Structure

Index	Path/Folder/Directory	file_name	status
1	dashboard_wwim	ins_pbi_well_info.sqlx	New
2	datamart_wwim	ins_well_allocation.sqlx	Existed
3	refined_wwim	p2_refined_a_ann_barrier.sqlx	Deleted

Status Colors

• New 🟢 - LightGreen (#90EE90)
• Existed 🔵 - LightBlue (#ADD8E6)
• Deleted 🔴 - LightCoral (#F08080)

Distribution

Build Distributable Package

cd scripts
python setup.py sdist bdist_wheel

This creates:

• dist/dataform-checklist-1.0.0.tar.gz (source)
• dist/dataform-checklist-1.0.0-py3-none-any.whl (wheel)

Share with Team

1. Copy wheel file to shared location or network drive
2. Team members install with:

   pip install dataform-checklist-1.0.0-py3-none-any.whl
   

3. Run from anywhere:

   dataform-checklist /path/to/repo
   

Upload to Internal PyPI (Optional)

If you have an internal PyPI server:

twine upload --repository internal dist/*

Then team members can install with:

pip install dataform-checklist --index-url http://your-pypi-server

Requirements

• Python 3.7+
• pandas >= 2.0.0
• openpyxl >= 3.1.0

Examples

Local Development

# Install in editable mode for development
pip install -e .

# Run from anywhere
dataform-checklist ~/projects/dataform-repo

Production Use

# Install from wheel
pip install dataform-checklist-1.0.0-py3-none-any.whl

# Schedule daily exports
dataform-checklist /prod/dataform/repo -v "v1.$(date +%Y%m%d).0"

CI/CD Integration

# .github/workflows/checklist.yml
- name: Generate Deployment Checklist
  run: |
    pip install dataform-checklist-1.0.0-py3-none-any.whl
    dataform-checklist . -v "v${{ github.run_number }}.0.0"
    
- name: Upload Checklist
  uses: actions/upload-artifact@v3
  with:
    name: deployment-checklist
    path: PTTEP_Deployment_Checklist_*.xlsx

Uninstallation

pip uninstall dataform-checklist

License

Internal use only - PTTEP Data Engineering Team

Support

For issues or questions, contact the data engineering team.