xfmr-zem 0.2.4

Zem: Unified Data Pipeline Framework (ZenML + NeMo Curator + DataJuicer) for multi-domain processing

🚀 Zem

Zem is a high-performance, unified data pipeline framework designed for the modern AI era. It seamlessly bridges ZenML's production-grade orchestration with specialized curation powerhouses like NVIDIA NeMo Curator and Alibaba Data-Juicer using the Model Context Protocol (MCP).

---

✨ Key Features

• 🏗️ Config-Driven Power: Define complex, production-ready pipelines in single YAML files.
• ⚡ True Parallel DAGs: Execute independent processing branches concurrently using a custom ParallelLocalOrchestrator.
• 🧠 Frontier LLM Integration: Smart data masking, classification, and summarization via Ollama or OpenAI.
• 📊 Deep Observability: Real-time profiling, per-tool performance metrics, and a beautiful integrated dashboard.
• 🔄 Adaptive Caching: Fine-grained, step-level cache control to optimize your development cycles.
• 🔌 Cloud Native: Native support for S3, GCS, and Parquet with seamless export to Hugging Face Hub and Vector DBs.

---

🏗️ Architecture

graph TD
    YAML["📄 pipeline.yaml"] --> Client["🛠️ Zem CLI / Client"]
    Client --> ZenML["🌀 ZenML Orchestrator"]
    ZenML --> Parallel["⚡ Parallel Local Orchestrator"]
    Parallel --> MCP_Bridge["🔗 MCP Bridge"]
    
    subgraph "Specialized Servers (MCP)"
        MCP_Bridge --> Nemo["🦁 NeMo Curator (GPU)"]
        MCP_Bridge --> DJ["🧃 Data-Juicer"]
        MCP_Bridge --> LLM["🤖 Frontier LLMs"]
        MCP_Bridge --> Prof["📈 Profiler"]
    end
    
    subgraph "Storage & Sinks"
        Nemo --> S3["☁️ Cloud / Parquet"]
        DJ --> HF["🤗 Hugging Face"]
        LLM --> VDB["🌐 Vector DB"]
    end

---

🚀 Quick Start

1. Installation

git clone https://github.com/OAI-Labs/xfmr-zem.git
cd xfmr-zem
uv sync

2. Initialize a New Project

# Bootstrap a standalone project with a sample agent
uv run zem init my_project
cd my_project

3. Run Your First Pipeline

uv run zem run pipeline.yaml

4. Visualize & Inspect

# Open ZenML Dashboard
uv run zem dashboard

# Preview results with sampling
uv run zem preview <artifact_id> --sample --limit 5

---

📖 Guided Documentation

Topic	Description	Link
Core Concepts	Understand the Zem architecture and MCP model.	AGENTS.md
Pipeline YAML	How to write and validate your pipeline configs.	Standard Example
Advanced Parallelism	Setup true local concurrency.	Parallel Guide
LLM & Sinks	Connecting to external AI stacks.	Phase 4 Demo

---

🤝 Contributing

We welcome contributions! Whether it's a new MCP server, a performance fix, or a typo in the docs, feel free to open a Pull Request.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

---

⚖️ License

Distributed under the Apache-2.0 License. See LICENSE for more information.

---

Built with ❤️ by the OAI-Labs Team