foundry-duck-transforms 0.1.4

Run transforms quickly on local machine and analyse using DuckDb and pyspark

Transforms package

A package for transforming data using transform syntax similar to Palantir Foundry. It enables you to write and test transforms locally before running them on foundry. Also they can be run faster using duckdb as underlying data engine.

Getting Started

Installation

Install the package using pip:

pip install foundry-duck-transforms

Basic Usage

1. Create a transform file (e.g. my_transform.py)
2. Run it using the transforms CLI:

python -m transforms.run my_transform.py dev,master

Above command will run the transform while downloading data from dev, fallbacking to master if dataset has no data on dev.

CLI Options

The transforms runner supports several options to customize execution:

python -m transforms.run [OPTIONS] TRANSFORM_TO_RUN FALLBACK_BRANCHES

Available options:

• --engine [spark|duckdb|spark-sail]: Engine to use for the transformation (default: spark)
• --omit-checks: Disables checks running
• --sail-server-url TEXT: Sail server url (required when using spark-sail engine)
• --dry-run: Dry run the transformation without writing results
• --local-dev-branch-name TEXT: Branch name for local development (default: "duck-fndry-dev")

Example with options:

python -m transforms.run my_transform.py dev,master --engine duckdb --dry-run

Development Setup

Prerequisites

• Python 3.7+
• pip
• Access to Palantir Foundry environment

Local Development

1. Clone the repository
2. Install development dependencies:

pip install -e ".[dev]"

Foundry Dev Tools Configuration

See here for detailed configuration instructions.

VSCode Setup

Add this to your .vscode/launch.json for debugging support:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python Debugger: Current File",
      "type": "debugpy",
      "request": "launch",
      "module": "transforms.run",
      "args": ["${file}", "dev,master"],
      "console": "integratedTerminal"
    }
  ]
}

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Submit a pull request

License

This project is licensed under the MIT License - see the LICENSE file for details.