dremioframe 0.23.0

A dataframe-like library for Dremio Cloud & Dremio Software

DremioFrame (currently in alpha)

DremioFrame is a Python library that provides a dataframe builder interface for interacting with Dremio Cloud & Dremio Software. It allows you to list data, perform CRUD operations, and administer Dremio resources using a familiar API.

Documentation

🚀 Getting Started

• Installation
• Optional Dependencies
• S3 Integration
• Quick Start
• Configuration
• Connecting to Dremio
• Tutorial: ETL Pipeline
• Cookbook / Recipes
• Troubleshooting
• API Compatibility Guide

🛠️ Data Engineering

• Dataframe Builder API
• Querying Data
• Joins & Transformations
• Aggregation
• Sorting & Filtering
• Creating Tables
• Ingestion of API's
• Ingestion Patterns
• dlt Integration
• Database Ingestion
• File System Ingestion
• File Upload
• Exporting Data
• Working with Files
• Caching
• Pydantic Integration
• Iceberg Tables
• Iceberg Lakehouse Management
• Schema Evolution
• Incremental Processing
• Query Templates
• Export Formats
• SQL Linting

📊 Analysis & Visualization

• Charting & Plotting
• Interactive Plotting
• Query Profiling

🧠 AI Capabilities and AI Agent

note: this libraries embdedded agent is primarily meant as a code generation assist tool, not meant as an alternative to the integrated Dremio agent for deeper administration and natural language analytics. Login to your Dremio instance's UI to leverage integrated agent.

• Overview
• DremioAgent Class
• MCP Server
• MCP Client Integration
• Document Extraction
• Script Generation
• SQL Generation
• API Call Generation
• Observability
• Reflections
• Governance
• Data Quality
• SQL Optimization
• CLI Chat

📐 Data Modeling

• Medallion Architecture
• Dimensional Modeling
• Slowly Changing Dimensions
• Semantic Views
• Documenting Datasets

⚙️ Orchestration

• Overview
• Tasks & Sensors
• Extensions
• Scheduling
• Dremio Jobs
• Iceberg Tasks
• Reflection Tasks
• Data Quality Task
• Distributed Execution
• Deployment
• CLI & UI
• Web UI
• Backends
• Best Practices

✅ Data Quality

• DQ Framework
• YAML Syntax
• Recipes

🔧 Administration & Governance

• Administration
• Catalog Management
• Reflections Management
• User Defined Functions (UDFs)
• Security Best Practices
• Security Patterns
• Governance: Masking & Row Access
• Governance: Tags
• Governance: Lineage
• Governance: Privileges
• Space & Folder Management
• Batch Operations
• Lineage Tracking

🔗 Integrations

• Airflow Integration
• Notebook Integration

🚀 Performance & Deployment

• Performance Tuning
• Bulk Loading Optimization
• Connection Pooling
• Query Cost Estimation
• CI/CD & Deployment

📚 Reference

• Function Reference
• SQL Functions Guide
• CLI Reference
• API Reference
• Async Client

🧪 Testing

• Mock/Testing Framework
• Advanced Usage
• Architecture
• Testing Guide
• Contributing

Installation

[!NOTE] DremioFrame has many optional dependencies for advanced features like AI, Chart Exporting, and Distributed Orchestration. See Optional Dependencies for a full list.

pip install dremioframe

To install with optional dependencies (e.g., for static image export):

pip install "dremioframe[image_export]"

Quick Start

Dremio Cloud

from dremioframe.client import DremioClient

# Assumes DREMIO_PAT and DREMIO_PROJECT_ID are set in env
client = DremioClient()

# Query a table
df = client.table('finance.bronze.transactions').select("transaction_id", "amount", "customer_id").limit(5).collect()
print(df)

Dremio Software v26+

# Assumes DREMIO_SOFTWARE_HOST and DREMIO_SOFTWARE_PAT are set in env
client = DremioClient(mode="v26")

# Or with explicit parameters
client = DremioClient(
    hostname="v26.dremio.org",
    pat="your_pat_here",
    tls=True,
    mode="v26"
)

Dremio Software v25

client = DremioClient(
    hostname="localhost",
    username="admin",
    password="password123",
    tls=False,
    mode="v25"
)

Features

from dremioframe.client import DremioClient

# Assumes DREMIO_PAT and DREMIO_PROJECT_ID are set in env
client = DremioClient()

# List catalog
print(client.catalog.list_catalog())

# Query data
df = client.table('finance.bronze.transactions').select("transaction_id", "amount", "customer_id").filter("amount > 1000").collect()
print(df)

# Calculated Columns
df.mutate(amount_with_tax="amount * 1.08").show()

# Aggregation
df.group_by("customer_id").agg(total_spent="SUM(amount)").show()

# Joins
customers = client.table('finance.silver.customers')
df.join(customers, on="transactions.customer_id = customers.customer_id").show()

# Iceberg Time Travel
df.at_snapshot("123456789").show()

# API Ingestion
client.ingest_api(
    url="https://api.example.com/users",
    table_name="finance.bronze.users",
    mode="merge",
    pk="id"
)

# Charting
sales_summary = client.table('finance.gold.sales_summary')
sales_summary.chart(kind="bar", x="category", y="total_sales", save_to="sales.png")

# Export
df.to_csv("transactions.csv")
df.to_parquet("transactions.parquet")

# Insert Data (Batched)
import pandas as pd
data = pd.DataFrame({"id": [1, 2], "name": ["A", "B"]})
client.table("finance.bronze.raw_data").insert("finance.bronze.raw_data", data=data, batch_size=1000)

# SQL Functions
from dremioframe import F

client.table("finance.silver.sales") \
    .select(
        F.col("dept"),
        F.sum("amount").alias("total_sales"),
        F.rank().over(F.Window.order_by("amount")).alias("rank")
    ) \
    .show()

# Merge (Upsert)
client.table("finance.silver.customers").merge(
    target_table="finance.silver.customers",
    on="customer_id",
    matched_update={"name": "source.name", "updated_at": "source.updated_at"},
    not_matched_insert={"customer_id": "source.customer_id", "name": "source.name"},
    data=data
)

# Data Quality
df.quality.expect_not_null("customer_id")
df.quality.expect_row_count("amount > 10000", 5, "ge") # Expect at least 5 large transactions

# Query Explanation
print(df.explain())

# Reflection Management
client.admin.create_reflection(dataset_id="...", name="my_ref", type="RAW", display_fields=["col1"])

# Async Client
# async with AsyncDremioClient() as client: ...

# CLI
# dremio-cli query "SELECT * FROM finance.gold.sales_summary LIMIT 10"

# Local Caching
# client.table("finance.bronze.transactions").cache("my_cache", ttl_seconds=300).sql("SELECT * FROM my_cache").show()

# Interactive Plotting
# df.chart(kind="scatter", backend="plotly").show()

# UDF Manager
# client.udf.create("add_one", {"x": "INT"}, "INT", "x + 1")

# Raw SQL
# df = client.query("SELECT * FROM finance.silver.customers")

# Source Management
# client.admin.create_source_s3("my_datalake", "bucket")

# Query Profiling
# client.admin.get_job_profile("job_123").visualize().show()

# Iceberg Client
# client.iceberg.list_tables("finance")

# Orchestration CLI
# dremio-cli pipeline list
# dremio-cli pipeline ui --port 8080

# Data Quality Framework
# dremio-cli dq run tests/dq

Sample .env

# Dremio Cloud Configuration
# Required for Dremio Cloud connections
# DREMIO_PAT: Personal Access Token for Dremio Cloud
# DREMIO_PROJECT_ID: The Project ID of your Dremio Cloud project
DREMIO_PAT=your_dremio_cloud_pat_here
DREMIO_PROJECT_ID=your_dremio_project_id_here

# Optional Dremio Cloud Configuration
# DREMIO_URL: Custom base URL for Dremio Cloud (defaults to data.dremio.cloud)
# Note: DREMIO_URL is REQUIRED for running integration tests (tests/test_integration.py)
# DREMIO_URL=data.dremio.cloud

# Dremio Software Configuration
# Required for connecting to Dremio Software (v26+ recommended)
# DREMIO_SOFTWARE_PAT: Personal Access Token for Dremio Software
# DREMIO_SOFTWARE_HOST: Hostname/URL of your Dremio Software instance (e.g. dremio.example.com)
#
# The following variables are specifically used by the test suite (tests/test_integration_software.py)
# and for legacy authentication (v25 or v26 without PAT):
# DREMIO_SOFTWARE_PORT: Port for the Dremio Flight/REST service (default: 32010 for Flight)
# DREMIO_SOFTWARE_USER: Username for Dremio Software
# DREMIO_SOFTWARE_PASSWORD: Password for Dremio Software
# DREMIO_SOFTWARE_TLS: Enable TLS/SSL (true/false, default: false)
#
# Note: If using Software, comment out Cloud variables above to avoid confusion, though the client 'mode' determines which are used.
# DREMIO_SOFTWARE_PAT=your_software_pat_here
# DREMIO_SOFTWARE_HOST=dremio.example.com
# DREMIO_SOFTWARE_PORT=32010
# DREMIO_SOFTWARE_USER=your_username
# DREMIO_SOFTWARE_PASSWORD=your_password
# DREMIO_SOFTWARE_TLS=false
# DREMIO_SOFTWARE_TESTING_FOLDER=Space.Folder

# Test Suite Configuration
# TEST_FOLDER: The namespace (Space) to use for creating temporary test folders and tables.
# Defaults to "testing" if not set. Ensure this Space exists in Dremio.
# TEST_FOLDER=testing

# AI Provider Configuration
# Required for AI features (DremioAgent, SQL generation, etc.)
# Uncomment the one you wish to use.

# OpenAI (Default)
# OPENAI_API_KEY=sk-...

# Anthropic (Claude)
# ANTHROPIC_API_KEY=sk-ant-...

# Google (Gemini)
# GOOGLE_API_KEY=AIza...