dataform-checklist 1.3.1

Generate Excel deployment checklists from Dataform repositories

Dataform Deployment Checklist Exporter

A Python package for generating Excel deployment checklists from Dataform repositories with automatic status tracking.

Features

• 🔍 Automatically scans all .sqlx and dependencies.js files in Dataform repositories
• 📊 Exports to formatted Excel with color-coded status
• 🔄 Tracks changes: New, Existed, Deleted files
• 🎨 Professional Excel formatting with conditional colors and top alignment
• 📝 Remark column with sources from dependencies.js (formatted with newlines)
• 🔔 Automatic version checking and update notifications
• 📦 Easy installation via PyPI
• 🚀 Command-line interface

Installation

From PyPI (Recommended)

pip install dataform-checklist

From Source

git clone https://github.com/OshigeAkito/dataform-checklist-package.git
cd dataform-checklist-package
pip install -e .

Upgrade to Latest Version

pip install --upgrade dataform-checklist

Usage

Command Line

After installation, use the dataform-checklist command:

# Basic usage
dataform-checklist /path/to/dataform/repo

# With version
dataform-checklist /path/to/dataform/repo -v v1.01.00

# With custom output path
dataform-checklist /path/to/dataform/repo -o /output/path -v v1.02.00

# With custom repository name
dataform-checklist /path/to/dataform/repo -n MyRepo -v v2.00.00

# Full options
dataform-checklist /path/to/repo -o /output -n WWIM -v v1.00.00

As Python Library

from dataform_checklist import DataformInventoryExporter

# Create exporter
exporter = DataformInventoryExporter(
    repository_path="/path/to/dataform/repo",
    output_path="/output/path",
    repository_name="WWIM",
    version="v1.00.00"
)

# Run export
exporter.run()

# Or use individual methods
files = exporter.scan_sqlx_files()
inventory = exporter.create_new_inventory(files)
exporter.export_to_excel(inventory)

Output

Filename Format

PTTEP_Dataform_Deployment_Checklist_{repository_name}_{version}.xlsx

Excel Structure

#	Path/Folder/Directory	Job/Script Name	Remark	status
1	dashboard_wwim	ins_pbi_well_info.sqlx		New
2	datamart_wwim	ins_well_allocation.sqlx		Existed
3	trusted	dependencies.js	wild_asset, production_data, external_api	Existed
4	refined_wwim	p2_refined_a_ann_barrier.sqlx		Deleted

Status Colors

• New 🟢 - LightGreen (#90EE90)
• Existed 🔵 - LightBlue (#ADD8E6)
• Deleted 🔴 - LightCoral (#F08080)

Version History

v1.2.3 (Latest)

• Refactored row alignment to apply vertical top alignment to entire rows
• Improved code maintainability

v1.2.2

• Added top vertical alignment to all columns for better readability
• Enabled text wrapping in remark column

v1.2.1

• Fixed header row reading issue
• Updated filename prefix format

v1.2.0

• Added remark column for dependencies.js sources
• Sources displayed with newline formatting
• Automatic scanning of dependencies.js files
• Automatic version checking on CLI startup

v1.0.0

• Initial release with basic .sqlx scanning
• Status tracking (New/Existed/Deleted)
• Excel export with color coding

Requirements

• Python 3.7+
• pandas >= 2.0.0
• openpyxl >= 3.1.0

Examples

Local Development

# Install in editable mode for development
pip install -e .

# Run from anywhere
dataform-checklist ~/projects/dataform-repo

Production Use

# Install from PyPI
pip install dataform-checklist

# Schedule daily exports
dataform-checklist /prod/dataform/repo -v "v1.$(date +%Y%m%d).0"

CI/CD Integration

# .github/workflows/checklist.yml
- name: Generate Deployment Checklist
  run: |
    pip install dataform-checklist
    dataform-checklist . -v "v${{ github.run_number }}.0.0"
    
- name: Upload Checklist
  uses: actions/upload-artifact@v3
  with:
    name: deployment-checklist
    path: PTTEP_Dataform_Deployment_Checklist_*.xlsx

Uninstallation

pip uninstall dataform-checklist

Dependencies.js Support

The tool automatically detects dependencies.js files and extracts source declarations:

// dependencies.js
var sources = ["wild_asset", "production_data", "external_api"];

These sources will be listed in the Remark column with newline formatting for readability.

License

MIT License - See LICENSE file for details

Repository

GitHub: OshigeAkito/dataform-checklist-package

PyPI: dataform-checklist

Support

For issues or questions, please open an issue on GitHub.