soigia 0.1.11

Python SDK for Soi Gia

soigia

soigia is a Python SDK built around three practical workspaces:

• DataTable for dataframe-style work with a friendlier API
• PipeLine for repeatable business workflows
• UI for lightweight internal screens on top of Streamlit

This README focuses on those three areas only.

Install

pip install soigia

For local development:

pip install -e .

1. DataTable

DataTable is the main table abstraction in soigia. It keeps the pandas mental model, but adds a more opinionated, workflow-friendly API for cleaning, filtering, joining, validating, and exporting data.

Use DataTable when you want:

• a table object that behaves like a dataframe
• chainable query-style operations
• simple data loading helpers
• schema checks before a dataset moves to the next step
• snapshot and version helpers for auditability

Typical workflow

1. Load rows from records, CSV, JSON, Excel, Parquet, Feather, pickle, SQL, or Google Sheets.
2. Clean and normalize the data.
3. Filter, sort, deduplicate, or join with another table.
4. Validate the final shape and types.
5. Export or snapshot the result.

Example

from soigia.datatable import DataTable

orders = DataTable.from_records(
    [
        {"order_id": 1001, "customer": "Alice", "amount": 120.5, "city": "Hanoi"},
        {"order_id": 1002, "customer": "Bob", "amount": 88.0, "city": "Saigon"},
        {"order_id": 1003, "customer": "Carol", "amount": 240.0, "city": "Danang"},
    ]
)

adults = orders.objects.filter(amount__gte=100).order_by("amount")
print(adults.to_rows())

Practical things you can do

• DataTable.from_records(records) for raw Python data
• DataTable.from_csv(path) for flat files
• DataTable.from_json(path) for payloads and exports
• DataTable.from_excel(path) for business spreadsheets
• DataTable.from_parquet(path) for analytics datasets
• DataTable.from_sql(...) for database-backed loading
• df.objects.filter(...) for query-like row selection
• df.objects.exclude(...) for inverse filtering
• df.objects.order_by(...) for sorting
• df.objects.distinct(...) for deduplication
• df.objects.group_by(...) for grouped summaries
• df.join(...) and related helpers for relational work
• df.validate_schema(...) for checks before release
• df.snapshot(...) and df.auto_version(...) for versioned outputs

When to use it

DataTable is a good fit when:

• you need dataframe behavior, but want a stricter workflow layer
• you are passing data between cleaning steps and business logic
• you want tests to assert shape and content more clearly
• you need small, readable transformations instead of scattered pandas calls

DataTable reference

For the full API list and examples, see:

• Module Reference

2. PipeLine

PipeLine is the workflow layer for business processing. It is designed for repeatable jobs where data comes in, gets normalized, goes through a few stages, and then writes out artifacts and summary files.

Use PipeLine when you want:

• a predictable execution order
• stage-based processing
• automatic output files
• rollback hooks for side effects
• config-driven behavior
• a shared model namespace for computed results

Mental model

1. load_data() brings data into memory.
2. Each stage mutates or enriches the working dataset.
3. self.model stores computed results.
4. self.config exposes YAML values.
5. save_outputs() writes final artifacts.
6. on_rollback(...) protects side effects when a stage fails.

Example

from soigia.base_pipeline import BasePipeline


class SalesPipeline(BasePipeline):
    stages = ["clean", "enrich", "summarize"]

    def load_data(self):
        return build_sales_rows()

    def clean(self):
        self.data_df = self.data_df.dropna().reset_index(drop=True)

    def enrich(self):
        self.data_df["net_amount"] = self.data_df["amount"] * 0.98

    def summarize(self):
        self.model.total_rows = len(self.data_df)
        self.model.total_amount = float(self.data_df["amount"].sum())


pipeline = SalesPipeline(name="sales")
result = pipeline.run()

print(result.success)
print(result.summary_path)
print(result.csv_path)

What a pipeline usually includes

• input loading from files, databases, or APIs
• normalization and cleaning
• business-specific enrichment
• summary or scoring logic
• CSV, SQLite, and Parquet outputs
• a Markdown summary for traceability
• log files for debugging and audit

Recommended structure

• keep the pipeline class small and stage-oriented
• put reusable config in config.yaml
• use config.example.yaml as the checked-in template
• use self.model for computed values that later stages need
• keep rollback handlers focused on external effects, not dataframe-only work

Pipeline reference

For the end-to-end guide, see:

• Pipeline End-to-End Guide

3. UI

soigia.ui is a declarative Streamlit layer for internal tools. It is meant for dashboards, admin screens, review pages, and lightweight operational views.

Use UI when you want:

• a single-file screen definition
• simple page composition
• forms, metrics, tables, and filters
• a fast way to expose operational workflows

Mental model

1. Define a page with ui.page().
2. Compose layout blocks with ui.sidebar(), ui.columns(), ui.tabs(), and ui.form().
3. Render data with ui.table() and ui.metric().
4. Add inputs with ui.text_input(), ui.select(), and ui.number_input().
5. Run the app with ui.run().

Example

from soigia.ui import ui

orders = [
    {"id": 1, "customer": "Alice", "amount": 120.0, "status": "paid"},
    {"id": 2, "customer": "Bob", "amount": 88.5, "status": "pending"},
    {"id": 3, "customer": "Carol", "amount": 220.0, "status": "paid"},
]


@ui.page("Orders Dashboard")
def orders_page(ctx):
    ui.markdown("Track orders, filter data, and prepare quick actions.")

    with ui.sidebar():
        keyword = ui.text_input("Search customer")
        status = ui.select("Status", ["all", "paid", "pending"])
        min_amount = ui.number_input("Minimum amount", value=0)

    filtered = [
        row
        for row in orders
        if keyword.value.lower() in row["customer"].lower()
        and (status.value == "all" or row["status"] == status.value)
        and row["amount"] >= min_amount.value
    ]

    ui.metric("Orders", len(filtered))
    ui.metric("Total amount", sum(row["amount"] for row in filtered))
    ui.table(filtered)


ui.run()

Common building blocks

• ui.page() for page registration
• ui.sidebar() for filters and navigation
• ui.columns() for split layouts
• ui.tabs() for grouped views
• ui.form() for update flows
• ui.table() for records and results
• ui.metric() for KPI cards
• ui.text(), ui.markdown(), and input widgets for interaction

When to use it

UI works best for:

• internal admin dashboards
• review screens for ops or business teams
• quick forms and compact workflows
• tools that should stay simple enough to maintain in one file

UI reference

For the end-to-end UI guide, see:

• UI End-to-End Guide

Quick Summary

• Use DataTable when the problem is mostly table manipulation.
• Use PipeLine when the problem is a staged business workflow.
• Use UI when the problem is an internal screen or dashboard.

Need More Detail?

The focused docs live here:

• Module Reference
• Pipeline End-to-End Guide
• UI End-to-End Guide