prophecy-orchestrate 1.0.19.dev50

Prophecy Orchestrate - High-performance data processing library with Go backend

Project Setup Guide

Prerequisites

1. Install Go

brew install go

2. Configure Go Environment

Add these lines to .bash_profile or .zshrc:

export GOPATH=$HOME/go
export PATH=$GOPATH/bin:$PATH

Note: After adding these lines, restart your terminal or run source .zshrc

3. Create Go Directory Structure

mkdir -p $GOPATH/{bin,pkg,src}

4. Install Dependencies

1. Install Protobuf:

   brew install protobuf
   

2. Install cmake

   brew install cmake
   

3. Install gRPC:

   go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.5.1
   

4. Install Project Dependencies:

   make install-dependencies 
   

5. init/update all submodules

   git submodule update --init --recursive
   

Development

Building and Testing

• Build and run the project:

  make all
  

• Run unit tests:

  make test
  

Running the Server Locally

1. Run Orchestrator

export RESOURCEMANAGER_TYPE=local
export LOG_FORMAT=console
go run ./pkg/cmd

2. Run Task Coordinator

export RESOURCEMANAGER_TYPE=local
export LOG_FORMAT=console
export MODE=task_coordinator
export DUMPTRACEONEXIT=true
export PPROFADDR=":6061"
go run ./pkg/cmd

Logging Options

• For JSON logging: unset LOG_FORMAT
• For trace logging: Add -trace flag
• For debug logging: Add -debug flag

Running with Docker

1. Build Docker images:

   make docker-build
   

2. Run with docker-compose:

   ./run_docker_compose_test.sh
   

Monitoring and Administration

Admin UI

• View Pipelines, Runs & Details: http://localhost:9089/engine/view
• Debug Profiler: http://localhost:6060/debug/pprof

Debugging

go tool pprof profile
(pprof)> tree 
(pprof)> web 

Runtime Configuration

Set log level dynamically:

curl --location 'http://localhost:9089/log/level' \
  --header 'Content-Type: application/json' \
  --data '{
    "level": "debug"
  }' 

Gem Debugging

Snowflake

CLI

• export table from snowflake

COPY INTO '@%TUSHAR/prophecy/data1row' from tushar file_format = (type = parquet) OVERWRITE = TRUE;

• download the parquet file in local

pip install snowflake-cli-labs
snow sql --query 'GET @%TUSHAR/prophecy/data file:///Users/tushargarg/;'

Prophecy DBT

A lightweight, stateful DBT parser and template resolver that leverages dbt-core libraries while providing a simpler API.

Overview

Prophecy DBT is designed to create a lighter, stateful version of DBT that can be used for template resolution, parsing, and executing SQL commands with materialization. It leverages dbt-core's libraries when available, providing the best of both worlds:

1. Compatibility with core DBT functionality
2. Lightweight processing when full DBT functionality isn't needed
3. Stateful handling of models, macros, and sources
4. Simple API for integration with web services and applications

Features

• Optimized Jinja Resolution: Uses dbt-core's built-in Jinja rendering when available, with fallback to a lightweight implementation
• Stateful Entity Management: Tracks models, macros, and sources in memory for efficient updates
• Multiple Access Methods: CLI, REST API, and GRPC services for integration
• Smart Materialization: Generates proper SQL for table/view creation based on model configuration
• High Performance: Designed for speed and efficiency when processing templates
• Automatic DBT Context: Injects appropriate context variables based on DBT project structure

Installation

# Basic installation
pip install prophecy_dbt

Start Server

export LOG_FORMAT=console && export ENABLE_PRODUCTION_DEBUGGING=true && export DBT_DBT_EAGER_LOAD_PROFILE=false && python -m server --log-level DEBUG

Usage

Command Line

# Resolve a simple template
prophecy-dbt resolve-template "SELECT * FROM {{ ref('model_name') }}" --project-dir ./my_dbt_project

# Resolve a file with materialization
prophecy-dbt resolve-file ./models/my_model.sql --materialization --project-dir ./my_dbt_project

Python API

from parser.jinja_resolve import resolve_template, resolve_with_materialization

# Simple template resolution
sql = resolve_template(
    template="SELECT * FROM {{ ref('model_name') }}",
    project_dir="./my_dbt_project"
)

REST API

Make a request for Materiazliation:

curl --location 'http://localhost:50054/resolve_jinja_templates' \
--header 'Content-Type: application/json' \
--data '{
    "entity":"orders",
    "mode": "RUN",
    "entity_type": "MODEL",
    "parameters": {
        "test_amount": "200"
    }
}'

Make a request for compiled DBT Code:

curl --location 'http://localhost:50054/resolve_jinja_templates' \
--header 'Content-Type: application/json' \
--data '{
    "entity":"orders",
    "mode": "COMPILE",
    "entity_type": "MODEL",
    "parameters": {
        "test_amount": "200"
    }
}'

Architecture

The package is designed with a layered architecture:

1. Core Layer: Basic Jinja resolution without DBT dependencies
2. DBT Integration Layer: Enhanced resolution using dbt-core libraries
3. Stateful Layer: Adds in-memory tracking of entities and dependencies
4. API Layer: Provides various interfaces (CLI, REST, GRPC)

This design allows the package to work with or without dbt-core installed, adapting to the available dependencies.

How It Works

The package includes a smart integration with dbt-core that leverages the following components:

1. dbt.clients.jinja: For optimal Jinja rendering
2. dbt.context: For providing the proper variables and functions
3. dbt.parser: For analyzing SQL and extracting metadata

When dbt-core is not available, it falls back to a custom implementation that covers the essential functionality.