metapod-probes 0.4.1

Lightweight metadata extraction probes — 191 probes across 47 data sources

metapod

Lightweight metadata extraction and data governance probes. 197 probes across 49 data sources in 11 categories - metadata, governance, quality, profiling, operations, compliance, cost, schema history, lineage, PII detection, and data contracts.

Runs inside your network. Extracts only metadata - no raw data leaves the perimeter.

Install

pip install metapod-probes                     # Core (file scanners, BI tools)
pip install 'metapod-probes[oracle]'           # + Oracle
pip install 'metapod-probes[azure]'            # + Azure (ADLS, Blob, ADF, File Share)
pip install 'metapod-probes[soda]'             # + Soda Core (data quality checks)
pip install 'metapod-probes[ydata]'            # + YData Profiling
pip install 'metapod-probes[presidio]'         # + PII Detection (~75 entity types)
pip install 'metapod-probes[all]'              # Everything

Air-gapped environments (no internet):

# On machine with internet:
bash scripts/build_airgapped.sh --extras azure
# Transfer tarball, then on air-gapped machine:
tar xzf metapod-probes-airgapped-*.tar.gz && bash install.sh

Quick start

metapod init                                    # Generate config.yaml
metapod list-probes                             # Show all 197 probes
metapod run "oracle.*" --schema RISK_MGMT       # Extract Oracle metadata
metapod run "adls_gen2.*" --since 2026-03-29    # Incremental (only recent)
metapod run "azure_data_factory.*" --parallel   # Parallel execution
metapod push --target https://your-platform.com # Send results
metapod serve --port 9090                       # Pull-mode daemon

Supported sources (49)

Databases (15)

Source	Probes	Driver
Oracle	18	oracledb
PostgreSQL	17	psycopg2
Azure SQL / SQL Server	15	pyodbc
Snowflake	5	snowflake-connector
BigQuery	5	google-cloud-bigquery
MySQL	6	pymysql
IBM DB2	4	ibm_db
MariaDB	2	pymysql
SAP HANA	2	hdbcli
Teradata	2	teradatasql
DuckDB	3	duckdb
Trino	3	trino
SQLite	2	stdlib
MongoDB	1	pymongo
Elasticsearch	1	elasticsearch

Cloud platforms (8)

Source	Probes	API
Databricks Unity Catalog	11	REST API
Azure Data Factory	8	Management API (lineage + dataset paths)
ADLS Gen2	4	azure-storage-file-datalake
Azure Blob Storage	4	azure-storage-blob
Azure File Share	3	azure-storage-file-share
Azure RBAC	2	Azure Management API
AWS S3	3	boto3
Oracle Cloud (OCI)	3	oci-python-sdk

BI tools (4)

Source	Probes	Method
Power BI	6	.pbit/.pbix ZIP parsing
Excel	6	.xlsx OOXML (table detector + formula lineage)
Tableau	5	.twb/.twbx XML parsing
Qlik Sense/View	3	.qvf SQLite / .qvw binary

ETL / Orchestration (6)

Source	Probes	Method
Informatica PowerCenter	5	Repository XML export
Informatica Cloud (IICS)	4	REST API v2/v3
Oracle Data Integrator	3	Smart Export XML
Airflow	3	REST API
Alteryx	2	.yxmd XML parsing
dbt	3	manifest.json parsing

Governance platforms (4)

Source	Probes	Direction
Collibra	3	Bidirectional
Microsoft Purview	3	Bidirectional
Blindata	3	Bidirectional
Witboost	2	Bidirectional

Data Quality & Profiling (3)

Source	Probes	Method
Soda Core	2	SodaCL checks + scan result import
YData Profiling	2	Dataset profiling + anomaly detection
Schema Contract	2	YAML contract validation + drift detection

Compliance & Governance (3)

Source	Probes	Method
Presidio PII	2	~75 PII entity types (PAN, IBAN, CF, GDPR) + custom YAML
Access Audit	1	Cross-reference PII findings with RBAC roles
Data Retention	1	Check dataset age against retention policies

Cost & Operations (1)

Source	Probes	Method
Storage Cost	1	Estimate Azure storage costs per directory

Files / Other (5)

Source	Probes	Method
File Scanner	5	Local filesystem (auto-dispatch + CSV/JSON/Parquet schema)
SharePoint / OneDrive	3	Local sync / REST / Graph API
Python Transforms	1	AST-based pandas/PySpark detection
Avro	1	Header-only schema reading
Generic	1	User-defined path

Architecture

Customer network (VPN / Azure / on-prem)
+---------------------------------------------+
|  metapod                                     |
|  +-- 197 probes across 49 sources           |
|  +-- Push mode: metapod push --target URL    |
|  +-- Pull mode: metapod serve --port 9090    |
|  +-- File mode: metapod run -> ./output/     |
|  +-- --parallel for concurrent execution     |
|  +-- --since for incremental extraction      |
|  +-- --delta for change detection            |
+---------------------+------------------------+
                      | outbound HTTPS only
                      v
            Any metadata consumer

• Zero inbound access - only outbound HTTPS
• No raw data - only metadata (schema, statistics, descriptions)
• Local-first - results written to ./output/ before push
• Config-driven - YAML with ${ENV_VAR} substitution
• Air-gapped - full offline install bundle available
• Token cache - single Azure AD authentication per session
• Probe cache - shared API results between related probes

Commands

metapod run "oracle.*"                          # Extract
metapod run "adls_gen2.*" --parallel -w 8       # Parallel (8 workers)
metapod run "adls_gen2.*" --since 2026-03-29    # Incremental
metapod run "oracle.*" --delta                  # Delta (only changes)
metapod push --target URL --project ID          # Push to platform
metapod serve --port 9090                       # Pull-mode daemon
metapod list-probes                             # Show all probes
metapod diff                                    # Compare extractions
metapod schedule "oracle.*" --cron "0 6 * * *"  # Cron schedule
metapod init                                    # Generate config
metapod new-probe mydb.metadata.tables          # Scaffold new probe

PII Detection (Presidio)

~75 built-in entity types for Financial Services, Healthcare, Telco, Public Admin:

pip install 'metapod-probes[presidio]'
metapod run "presidio.compliance.pii_scan" -c config_presidio.yaml

• 13 PAN issuers with Luhn validation (Visa, MC, Amex, PagoBancomat, Nexi, Postepay...)
• 25 Financial Services patterns (IBAN, SWIFT, ABI/CAB, ISIN, LEI, NDG, AML SAR...)
• 23 domain patterns (ICD-10, IMEI, POD/PDR energy, targa IT, VIN, PagoPA IUV...)
• Custom YAML patterns - define your own without modifying code

Data Contracts

# contract.yml
contracts:
  - dataset: TRANSACTIONS
    columns:
      - name: transaction_id
        type: integer
        nullable: false
      - name: amount
        type: decimal
    rules:
      min_columns: 3

metapod run "schema_contract.quality.validate" -s contract.yml

Adding a probe

from core.base_probe import BaseProbe
from core.registry import register

@register
class MyProbe(BaseProbe):
    @staticmethod
    def probe_name() -> str:
        return "mydb.metadata.tables"

    @staticmethod
    def description() -> str:
        return "Extract table metadata"

    def execute(self, connection, schema: str, **kwargs) -> list[dict]:
        return [{"table_name": "example"}]

Or use the scaffold: metapod new-probe mydb.metadata.tables

License

Apache 2.0