metapod-probes 0.1.1

Lightweight metadata extraction probes — 135 probes across 35 data sources

metapod

Lightweight metadata and quality probes for the target platform. Runs inside your network (VPN, Azure, on-prem) and extracts only metadata — no raw data leaves the perimeter.

Quick start (Python)

# Install
pip install -e ".[oracle]"

# Create config
python cli.py init
# Edit config.yaml with your Oracle credentials

# List available probes
python cli.py list-probes

# Run all Oracle metadata probes
python cli.py run "oracle.metadata.*" --schema RISK_MGMT

# Run all probes (metadata + quality)
python cli.py run "oracle.*" --schema RISK_MGMT

# Run a single probe
python cli.py run oracle.metadata.tables --schema RISK_MGMT

# Run and push results to target platform
python cli.py run "oracle.*" --schema RISK_MGMT --push

Quick start (Docker)

docker build -t metapod .
docker run -v $(pwd)/config.yaml:/app/config.yaml -v $(pwd)/output:/app/output \
  metapod "oracle.metadata.*"

Available probes

Oracle metadata (9 probes)

Probe	Source	Extracts
oracle.metadata.tables	ALL_TABLES	Table names, row counts, tablespace
oracle.metadata.columns	ALL_TAB_COLUMNS	Column names, types, nullable, position
oracle.metadata.views	ALL_VIEWS	View SQL definitions
oracle.metadata.source_code	ALL_SOURCE	PL/SQL: procedures, functions, packages, triggers
oracle.metadata.synonyms	ALL_SYNONYMS	Private + public synonym mappings
oracle.metadata.db_links	ALL_DB_LINKS	Database link definitions
oracle.metadata.constraints	ALL_CONSTRAINTS	PK, FK, unique constraints with columns
oracle.metadata.indexes	ALL_INDEXES	Index types and column compositions
oracle.metadata.comments	ALL_TAB/COL_COMMENTS	Business descriptions

Oracle quality (5 probes)

Probe	Method	Measures
oracle.quality.null_ratio	Sampled COUNT	NULL % per column
oracle.quality.distinct_count	Oracle stats	Cardinality per column
oracle.quality.min_max	Oracle stats	Value ranges for numeric/date
oracle.quality.freshness	LAST_ANALYZED	Data staleness in days
oracle.quality.pattern_check	REGEXP_LIKE	PII patterns (email, phone, SSN, IBAN)

Architecture

Customer network (VPN / Azure / on-prem)
┌─────────────────────────────────────────┐
│  metapod                          │
│  ├── oracle.metadata.* → local JSON     │
│  ├── oracle.quality.*  → local JSON     │
│  └── --push → HTTPS to target platform API     │
└────────────────┬────────────────────────┘
                 │ outbound HTTPS only
                 ▼
┌─────────────────────────────────────────┐
│  the target platform Platform                       │
│  └── Metastore (MinIO)                  │
│      └── probes/oracle/metadata/*.json  │
└─────────────────────────────────────────┘

• Zero inbound access — only outbound HTTPS to target platform API
• No raw data — only metadata (schema structure, statistics, descriptions)
• Local-first — results written to ./output/ before optional push
• Config-driven — YAML file, no code changes per deployment

Configuration

source:
  type: oracle
  host: prod-db.example.com
  port: 1521
  service_name: ORCLPDB1
  username: metadata_reader
  password: "${ORACLE_PASSWORD}"
  schema_name: RISK_MGMT

target:
  url: https://your-platform.example.com
  api_key: "${METAPOD_API_KEY}"
  project: default

output_dir: ./output

Output format

Every probe produces a standard JSON envelope:

{
  "probe": "oracle.metadata.tables",
  "source": {"type": "oracle", "id": "prod-db:1521/ORCLPDB1/RISK_MGMT"},
  "timestamp": "2026-03-27T23:00:00Z",
  "records": [...],
  "record_count": 42,
  "duration_ms": 350
}

Adding probes

Create a new file in probes/oracle/metadata/ or probes/oracle/quality/:

from core.base_probe import BaseProbe
from core.registry import register

@register
class MyProbe(BaseProbe):
    @staticmethod
    def probe_name() -> str:
        return "oracle.metadata.my_probe"

    @staticmethod
    def description() -> str:
        return "What this probe extracts"

    def execute(self, connection, schema: str, **kwargs) -> list[dict]:
        with connection.cursor() as cur:
            cur.execute("SELECT ... FROM all_... WHERE owner = :schema", schema=schema.upper())
            return [{"col": row[0]} for row in cur.fetchall()]

License

Apache 2.0