metapod-probes 0.3.0

Lightweight metadata extraction probes — 185 probes across 42 data sources

metapod

Lightweight metadata extraction probes. 185 probes across 42 data sources in 9 categories — metadata, governance, quality, profiling, operations, compliance, cost, schema history, and lineage.

Runs inside your network. Extracts only metadata — no raw data leaves the perimeter.

Install

pip install metapod-probes                     # Core (file scanners, BI tools)
pip install 'metapod-probes[oracle]'           # + Oracle
pip install 'metapod-probes[snowflake]'        # + Snowflake
pip install 'metapod-probes[azure]'            # + Azure (ADLS, Blob, ADF)
pip install 'metapod-probes[all]'              # Everything

Quick start

metapod init                                    # Generate config.yaml
metapod list-probes                             # Show all 155 probes
metapod run "oracle.*" --schema RISK_MGMT       # Extract Oracle metadata
metapod push --target https://your-platform.com --project myproject  # Send results
metapod serve --port 9090                       # Pull-mode daemon

Supported sources (40)

Databases (15)

Source	Probes	Driver
Oracle	18	oracledb
PostgreSQL	10	psycopg2
Snowflake	5	snowflake-connector
BigQuery	5	google-cloud-bigquery
SQL Server / Azure SQL	5	pyodbc
MySQL	4	pymysql
IBM DB2	4	ibm_db
MariaDB	2	pymysql
SAP HANA	2	hdbcli
Teradata	2	teradatasql
DuckDB	3	duckdb
Trino	3	trino
SQLite	2	stdlib
MongoDB	1	pymongo
Elasticsearch	1	elasticsearch

Cloud platforms (7)

Source	Probes	API
Databricks Unity Catalog	6	REST API
Azure Data Factory	5	Azure Management API
ADLS Gen2	4	azure-storage-file-datalake
Azure Blob Storage	3	azure-storage-blob
Azure File Share	3	azure-storage-file-share
AWS S3	3	boto3
Oracle Cloud (OCI)	3	oci-python-sdk

BI tools (4)

Source	Probes	Method
Power BI	6	.pbit/.pbix ZIP parsing
Excel	6	.xlsx OOXML (table detector + formula lineage)
Tableau	5	.twb/.twbx XML parsing
Qlik Sense/View	3	.qvf SQLite / .qvw binary

ETL / Orchestration (6)

Source	Probes	Method
Informatica PowerCenter	5	Repository XML export
Informatica Cloud (IICS)	4	REST API v2/v3
Oracle Data Integrator	3	Smart Export XML
Airflow	3	REST API
Alteryx	2	.yxmd XML parsing
dbt	3	manifest.json parsing

Governance platforms (4)

Source	Probes	Direction
Collibra	3	Bidirectional
Microsoft Purview	3	Bidirectional
Blindata	3	Bidirectional
Witboost	2	Bidirectional

Files / Other (4)

Source	Probes	Method
File Scanner	5	Local filesystem (auto-dispatch + CSV/JSON/Parquet schema)
SharePoint / OneDrive	3	Local sync / REST / Graph API
Python Transforms	1	AST-based pandas/PySpark detection
Avro	1	Header-only schema reading

Architecture

Customer network (VPN / Azure / on-prem)
┌─────────────────────────────────────────────┐
│  metapod                                     │
│  ├── 155 probes across 40 sources           │
│  ├── Push mode: metapod push --target URL    │
│  ├── Pull mode: metapod serve --port 9090    │
│  └── File mode: metapod run → ./output/      │
└──────────────────┬──────────────────────────┘
                   │ outbound HTTPS only
                   ▼
         Any metadata consumer

• Zero inbound access — only outbound HTTPS
• No raw data — only metadata (schema, statistics, descriptions)
• Local-first — results written to ./output/ before push
• Config-driven — YAML with ${ENV_VAR} substitution

Commands

metapod run "oracle.*"                          # Extract
metapod push --target URL --project ID          # Push to platform
metapod serve --port 9090                       # Pull-mode daemon
metapod list-probes                             # Show all probes
metapod diff                                    # Compare extractions
metapod schedule "oracle.*" --cron "0 6 * * *"  # Cron schedule
metapod init                                    # Generate config

Adding a probe

from core.base_probe import BaseProbe
from core.registry import register

@register
class MyProbe(BaseProbe):
    @staticmethod
    def probe_name() -> str:
        return "mydb.metadata.tables"

    @staticmethod
    def description() -> str:
        return "Extract table metadata"

    def execute(self, connection, schema: str, **kwargs) -> list[dict]:
        # Your extraction logic here
        return [{"table_name": "example"}]

License

Apache 2.0