opendqv 2.10.0

"Trust is easier to build than to repair." That is why OpenDQV exists. A 422 at the point of write is cheaper than a data incident three weeks later.

Beta (v2.x). Public API surface (REST, contract YAML, MCP tools, Python SDK) is stable. Breaking changes follow a one-release deprecation cycle. Security fixes backported to the latest 2.x line. See API Stability for commitments.

OpenDQV is a write-time data validation service. Source systems call it before writing data. Bad records return a 422 with per-field errors. Good records pass through. No payload is stored.

flowchart LR
    subgraph Callers
        direction TB
        SF[Salesforce]
        SAP[SAP]
        DYN[Dynamics]
        ORA[Oracle]
        WEB[Web forms]
        ETL1[ETL pipelines]

        DJ[Django clean]
        PY[Python scripts]
        PD[Pandas / ETL]

        CD[Claude Desktop]
        CUR[Cursor]
        LLM[LLM agents]
    end

    subgraph OpenDQV
        direction TB
        API[Validation API\nREST / batch]
        SDK[LocalValidator\nin-process SDK]
        MCP[MCP Server\nAI-native]
        API & SDK & MCP --> CON[Contracts · YAML\nGovernance · RBAC\nAudit trail]
        API & SDK & MCP --> GEN[Code Generator\nApex · JS · SQL]
    end

    subgraph Results
        direction TB
        R1[valid: true / false]
        R2[per-field errors]
        R3[severity levels]
        R4[webhooks on events]
    end

    SF & SAP & DYN & ORA & WEB & ETL1 --> API
    DJ & PY & PD --> SDK
    CD & CUR & LLM --> MCP

    API & SDK & MCP --> R1

    subgraph Importers
        IMP[dbt schema · GX suites\nSoda checks · ODCS · CSV]
    end
    IMP --> CON

    style API fill:#0d3b5e,stroke:#092a44,color:#fff
    style SDK fill:#0d3b5e,stroke:#092a44,color:#fff
    style MCP fill:#0d3b5e,stroke:#092a44,color:#fff
    style CON fill:#1a8aad,stroke:#14708d,color:#fff
    style GEN fill:#1a8aad,stroke:#14708d,color:#fff
    style R1 fill:#2ec4e6,stroke:#1a8aad,color:#0d3b5e
    style R2 fill:#2ec4e6,stroke:#1a8aad,color:#0d3b5e
    style R3 fill:#2ec4e6,stroke:#1a8aad,color:#0d3b5e
    style R4 fill:#2ec4e6,stroke:#1a8aad,color:#0d3b5e
    style IMP fill:#1a8aad,stroke:#14708d,color:#fff

A 422 at the point of write closes the feedback loop — producers see failures immediately and fix them at source. Rejection rates drop over time because the tool changes the incentive, not just the outcome.

For post-landing monitoring use Great Expectations, Soda, or dbt tests — they're complementary, not competing. OpenDQV owns layer one (write-time enforcement); those tools own layer three (post-ingestion observability).

AI Agents — first-class via MCP

OpenDQV ships a built-in Model Context Protocol server, so Claude Desktop, Cursor, and any other MCP-compatible agent can discover contracts, validate records, and explain failures through tool calls the agent explicitly declares — no hallucinated compliance, no invented rules.

4-minute demo: Claude Desktop uses two MCP servers — OpenDQV for validation, Marmot for catalog lineage — to check a menu item against ppds_menu_item for Natasha's Law allergen compliance, stating which tool calls it makes and why. (Backup: download the MP4 from the repo)

For tool reference, write guardrails, remote/enterprise mode, and the Marmot composition pattern, see docs/mcp.md.

Reserved agent_id prefix. The prefix OpenDQV_SA_ is reserved for OpenDQV-owned system traffic — smoke probes, demos, MCP self-tests, perf harnesses. The pattern is OpenDQV_SA_[Category]_[Scope] (e.g. OpenDQV_SA_smoke_v240, OpenDQV_SA_probe_persona_b). Customer-facing metrics endpoints (/api/v1/stats, /api/v1/agents, MCP get_quality_metrics, MCP list_agents) suppress these by default so tenant views stay clean of dev/test traffic. Pass include_system=true to surface them for diagnostics — each row carries an is_system_agent flag.

Install

install.sh creates a virtual environment, installs dependencies, and launches the onboarding wizard. Docker pulls ghcr.io/opendqv/opendqv:latest — no build step required.

⚠️ AUTH_MODE=open (the default) has no authentication. Set AUTH_MODE=token and a strong SECRET_KEY in .env before any non-local deployment. See SECURITY.md.

Your First Validation

1. Write a contract — drop a YAML file in your contracts directory (run opendqv init --all to copy every bundled contract, or opendqv init for a single starter):

contract:
  name: order
  version: "1.0"
  owner: "Data Governance"
  status: active
  rules:
    - name: valid_email
      type: regex
      field: email
      pattern: "^[^@\\s]+@[^@\\s]+\\.[^@\\s]+$"
      severity: error
      error_message: "Invalid email format"
    - name: amount_positive
      type: min
      field: amount
      min: 0.01
      severity: error
      error_message: "Order amount must be positive"
    - name: status_valid
      type: allowed_values
      field: status
      allowed_values: [pending, confirmed, shipped, cancelled]
      severity: error
      error_message: "Invalid order status"

2. Reload contracts:

curl -X POST http://localhost:8000/api/v1/contracts/reload

3. Send a bad record — OpenDQV rejects it:

curl -s -X POST http://localhost:8000/api/v1/validate \
  -H "Content-Type: application/json" \
  -d '{"contract": "order", "record": {"email": "not-an-email", "amount": -5, "status": "unknown"}}'

{
  "valid": false,
  "errors": [
    {"field": "email",  "rule": "valid_email",    "message": "Invalid email format",        "severity": "error", "error_code": "OPENDQV_REGEX_VALID_EMAIL"},
    {"field": "amount", "rule": "amount_positive", "message": "Order amount must be positive", "severity": "error", "error_code": "OPENDQV_MIN_AMOUNT_POSITIVE"},
    {"field": "status", "rule": "status_valid",    "message": "Invalid order status",        "severity": "error", "error_code": "OPENDQV_ALLOWED_VALUES_STATUS_VALID"}
  ],
  "contract": "order",
  "version": "1.0"
}

(Abridged — the real response also carries contract_hash, engine_version, validated_at, and per-error suggested_fix.)

4. Fix the record — it passes:

curl -s -X POST http://localhost:8000/api/v1/validate \
  -H "Content-Type: application/json" \
  -d '{"contract": "order", "record": {"email": "alice@example.com", "amount": 49.99, "status": "pending"}}'

{"valid": true, "errors": [], "warnings": [], "contract": "order", "version": "1.0"}

The customer contract ships pre-seeded if you want to skip step 1. The quickstart guide walks through authoring, lifecycle, and batch validation.

Rules

Rules have severity: error (blocks the record) or severity: warning (flags but allows). Any rule can include a condition block so it applies only when a condition on another field holds — value, not_value, or present: true|false.

Full reference: docs/rules/

How it compares

A mature data governance programme operates across three layers, each with a distinct job:

OpenDQV Core owns layer one. Your catalog handles layer two, your pipeline tools handle layer three.

They're complementary. Use Great Expectations to monitor your warehouse. Use OpenDQV to stop bad data from getting there in the first place.

Contracts

41 production-ready contracts ship inside the opendqv package covering GDPR, HIPAA, SOX, MiFID II, UK Building Safety Act, Martyn's Law, Natasha's Law, Ofcom Online Safety Act, EU DORA, and 20+ other regulatory frameworks across UK, EU, and US. pip install opendqv gives you all of them — opendqv list works with zero configuration.

See docs/compliance-contracts.md for the full list with regulatory context, or browse opendqv/contracts/ directly. 17 minimal starter templates are in examples/starter_contracts/.

Performance

EC2 c6i.large, 2 workers, 12-rule contract, mixed 50/50 workload: ~482 req/s, p99 ~182 ms. Sizing rule: WEB_CONCURRENCY = number of vCPUs.

See docs/benchmark_throughput.md for full platform comparison, methodology, and monthly volume extrapolation.

Documentation

API Stability

OpenDQV is in Beta as of 2.0.0. The following stability commitments apply to the v2.x series:

• REST API endpoints — paths, request bodies, and response shapes are stable within v2.x. Backwards-incompatible changes require a major version bump and follow a deprecation cycle (one minor release of warnings before removal).
• YAML contract format — the contract schema (rules, fields, types) is stable within v2.x. New rule types may be added; existing rules will not change semantics without a deprecation cycle.
• Python SDK — OpenDQVClient, AsyncOpenDQVClient, and LocalValidator public method signatures are stable within v2.x. Internal helpers (prefixed _) are not covered.
• MCP tools — tool names and parameters are stable within v2.x.
• Security fixes — backported to the latest 2.x line on a best-effort basis.

Presence and null handling (v2.5+)

• Presence is explicit. A format rule (regex, min, max_length, date_format, …) never implies that a field must exist — add not_empty / not_empty_string when it must. Every non-presence rule treats a missing, null, or blank value as absent and passes. A cross-field rule (compare, field_sum, ratio_check, date_diff, …) whose counterpart is absent or blank fails, and the error entry carries counterpart_missing: true. See docs/contract_conformance.md.
• Unknown rule types are refused at load (v2.8+). A typo in type: (e.g. min_lenght) is a load error naming the known types — on the REST rule endpoints (422), the MCP draft tool, every importer, and a stored YAML file (the contract does not load; the refusal is logged; opendqv lint catches it before deploy). Until 2.7 such a rule loaded as a disabled rule that passed everything. Both engines refuse the same day.

Contributing

See CONTRIBUTING.md for setup instructions, coding guidelines, and how to submit changes.

License

MIT — see LICENSE.

Acknowledgements

Led by Sunny Sharma, BGMS Consultants Ltd. The vision, the architecture, every contract, and every design decision in this repository are directed by a human who believes data quality is a write-time responsibility.

OpenDQV is built with a hybrid team. Sunny leads — carbon and silicon. Three AI collaborators execute: Claude Sonnet 4.6 (primary developer), Claude Opus 4.6 (strategic auditor), and Grok (market intelligence). All answer to the same ethos: trust is easier to build than to repair.