dremioframe 0.18.0

A dataframe-like library for Dremio Cloud & Dremio Software

DremioFrame (currently in alpha)

DremioFrame is a Python library that provides a dataframe builder interface for interacting with Dremio Cloud & Dremio Software. It allows you to list data, perform CRUD operations, and administer Dremio resources using a familiar API.

Documentation

🚀 Getting Started

• Installation
• Optional Dependencies
• S3 Integration
• Quick Start
• Configuration
• Connecting to Dremio
• Tutorial: ETL Pipeline
• Cookbook / Recipes
• Troubleshooting

🛠️ Data Engineering

• Dataframe Builder API
• Querying Data
• Joins & Transformations
• Aggregation
• Sorting & Filtering
• Ingestion API
• Ingestion Patterns
• dlt Integration
• Database Ingestion
• File System Ingestion
• File Upload
• Exporting Data
• Working with Files
• Caching
• Pydantic Integration
• Iceberg Tables
• Iceberg Lakehouse Management
• Schema Evolution
• Incremental Processing
• Query Templates
• Export Formats
• SQL Linting

📊 Analysis & Visualization

• Charting & Plotting
• Interactive Plotting
• Query Profiling

🧠 AI Capabilities and AI Agent

note: this libraries embdedded agent is primarily meant as a code generation assist tool, not meant as an alternative to the integrated Dremio agent for deeper administration and natural language analytics. Login to your Dremio instance's UI to leverage integrated agent.

• Overview
• DremioAgent Class
• MCP Server Integration
• Document Extraction
• Script Generation
• SQL Generation
• API Call Generation
• Observability
• Reflections
• Governance
• Data Quality
• SQL Optimization
• CLI Chat

📐 Data Modeling

• Medallion Architecture
• Dimensional Modeling
• Slowly Changing Dimensions
• Semantic Views
• Documenting Datasets

⚙️ Orchestration

• Overview
• Tasks & Sensors
• Extensions
• Scheduling
• Dremio Jobs
• Iceberg Tasks
• Reflection Tasks
• Data Quality Task
• Distributed Execution
• Deployment
• CLI & UI
• Web UI
• Backends
• Best Practices

✅ Data Quality

• DQ Framework
• YAML Syntax
• Recipes

🔧 Administration & Governance

• Administration
• Catalog Management
• Reflections Management
• User Defined Functions (UDFs)
• Security Best Practices
• Security Patterns
• Governance: Masking & Row Access
• Governance: Tags
• Governance: Lineage
• Governance: Privileges
• Space & Folder Management
• Batch Operations
• Lineage Tracking

🔗 Integrations

• Airflow Integration
• Notebook Integration

🚀 Performance & Deployment

• Performance Tuning
• Bulk Loading Optimization
• Connection Pooling
• Query Cost Estimation
• CI/CD & Deployment

📚 Reference

• Function Reference
• SQL Functions Guide
• CLI Reference
• API Reference
• Async Client

🧪 Testing

• Mock/Testing Framework
• Advanced Usage
• Architecture
• Testing Guide
• Contributing

Installation

[!NOTE] DremioFrame has many optional dependencies for advanced features like AI, Chart Exporting, and Distributed Orchestration. See Optional Dependencies for a full list.

pip install dremioframe

To install with optional dependencies (e.g., for static image export):

pip install "dremioframe[image_export]"

Quick Start

Dremio Cloud

from dremioframe.client import DremioClient

# Assumes DREMIO_PAT and DREMIO_PROJECT_ID are set in env
client = DremioClient()

# Query a table
df = client.table("Samples.samples.dremio.com.zips.json").select("city", "state").limit(5).collect()
print(df)

Dremio Software

client = DremioClient(
    hostname="localhost",
    port=32010,
    username="admin",
    password="password123",
    tls=False
)

Features

from dremioframe.client import DremioClient

client = DremioClient(pat="YOUR_PAT", project_id="YOUR_PROJECT_ID")

# List catalog
print(client.catalog.list_catalog())

# Query data
df = client.table("Samples.samples.dremio.com.zips.json").select("city", "state").filter("state = 'MA'").collect()
print(df)

# Calculated Columns
df.mutate(total_pop="pop * 2").show()

# Aggregation
df.group_by("state").agg(avg_pop="AVG(pop)").show()

# Joins
df.join("other_table", on="left_tbl.id = right_tbl.id").show()

# Iceberg Time Travel
df.at_snapshot("123456789").show()



# API Ingestion
client.ingest_api(
    url="https://api.example.com/users",
    table_name="users",
    mode="merge",
    pk="id"
)

# Charting
df.chart(kind="bar", x="category", y="sales", save_to="sales.png")

# Export
df.to_csv("data.csv")
df.to_parquet("data.parquet")

# Insert Data (Batched)
import pandas as pd
data = pd.DataFrame({"id": [1, 2], "name": ["A", "B"]})
client.table("my_table").insert("my_table", data=data, batch_size=1000)

# SQL Functions
from dremioframe import F

client.table("sales") \
    .select(
        F.col("dept"),
        F.sum("amount").alias("total_sales"),
        F.rank().over(F.Window.order_by("amount")).alias("rank")
    ) \
    .show()

# Merge (Upsert)
client.table("target").merge(
    target_table="target",
    on="id",
    matched_update={"name": "source.name"},
    not_matched_insert={"id": "source.id", "val": "source.val"},
    data=data
)

# Data Quality
df.quality.expect_not_null("city")
df.quality.expect_row_count("pop > 1000000", 5, "ge") # Expect at least 5 cities with pop > 1M

# Query Explanation
print(df.explain())

# Reflection Management
client.admin.create_reflection(dataset_id="...", name="my_ref", type="RAW", display_fields=["col1"])

# Async Client
# async with AsyncDremioClient(pat="...") as client: ...

# CLI
# dremio-cli query "SELECT 1"

# Local Caching
# client.table("source").cache("my_cache", ttl_seconds=300).sql("SELECT * FROM my_cache").show()

# Interactive Plotting
# df.chart(kind="scatter", backend="plotly").show()

# UDF Manager
# client.udf.create("add_one", {"x": "INT"}, "INT", "x + 1")

# Raw SQL
# df = client.query("SELECT * FROM my_table")

# Source Management
# client.admin.create_source_s3("my_datalake", "bucket")

# Query Profiling
# client.admin.get_job_profile("job_123").visualize().show()

# Iceberg Client
# client.iceberg.list_tables("my_namespace")

# Orchestration CLI
# dremio-cli pipeline list
# dremio-cli pipeline ui --port 8080

# Data Quality Framework
# dremio-cli dq run tests/dq