Berg

Berg is a safe management layer for Apache Iceberg catalogs and table metadata. It helps teams inspect tables, identify health problems, describe desired state, review deterministic changes, and execute approved operations inside their own infrastructure.

Berg does not replace Spark, Flink, Trino, Athena, or another query engine. It does not move or transform table records. Its focus is controlled metadata and catalog operations with explicit approval, backend capability checks, durable execution state, idempotency, audit history, and failure reconciliation.

Install

With pip:

python -m pip install berg-iceberg

With uv:

uv tool install berg-iceberg

Then:

berg --version
berg --help

What you can do

berg connect
berg capabilities
berg scan
berg inspect analytics.events
berg health analytics.events
berg maintenance-report analytics.events --output json
berg diff table.yaml --fail-on-drift --output json

Use --output json for automation and CI/CD integrations.

For a metadata change, Berg uses a reviewable plan:

berg plan analytics.events \
  --set-property write.target-file-size-bytes=536870912 \
  --save events-plan.json

berg apply events-plan.json --dry-run
berg apply events-plan.json --approve --audit-file audit.jsonl
berg execution-status ID --output json

Plans are checked against current table state and the selected backend. Berg rejects unsupported operations, stale plans, unauthorized changes, and unsafe duplicate execution before mutation.

For CI/CD policy gates, use berg validate-policy and berg authorize on the exact saved plan before any execution. See the CI/CD integration guide and the provider-neutral examples in examples/ci.

Customer-managed policy bundles are versioned berg.dev/v1 YAML or JSON documents. They can scope allow, warning, approval, and denial rules to operations, namespaces, and tables. Use docs/policy-bundles.md and the example in examples/policies/production.yaml.

Declarative table management

Desired state can be written as a portable berg.dev/v1 YAML or JSON document:

berg validate-spec table.yaml
berg diff table.yaml
berg plan --spec table.yaml --save table-plan.json

Python users can author the same document without connecting to a catalog:

from berg.sdk import TableDefinition

events = (
    TableDefinition.for_table("analytics.events")
    .with_column("id", "long", required=True)
    .with_column("created_at", "timestamp", required=True)
    .partition_by("created_at", transform="day")
    .with_property("write.target-file-size-bytes", "536870912")
)

events.write("events.yaml")

Policies can be authored with the same typed SDK:

from berg.sdk import PolicyDefinition

PolicyDefinition.for_policy("production") \\
    .require_approval("review-properties", operations=("SET_PROPERTY",)) \\
    .write("production-policy.yaml")

Supported catalog paths

Berg has compatibility contracts for:

• Iceberg REST Catalog with S3-compatible storage
• Project Nessie
• Apache Polaris
• JDBC/PostgreSQL catalogs
• Hive Metastore
• AWS Glue Catalog with S3

Support is operation- and backend-specific. Run berg capabilities and read the compatibility matrix before enabling an operation in automation.

Customer-local Agent

The Agent runs the same guarded execution kernel inside customer infrastructure. It receives an approved plan, uses customer credentials, and writes state and audit records locally. It requires no inbound service and does not upload raw table data to Berg Cloud.

berg agent apply plan.json --approve
berg agent status --output json

berg maintenance-report and berg agent status are read-only operational commands. Maintenance reports explain recommendations and backend blocks; Agent status includes the versioned status/heartbeat envelope and local inbox, processing, and quarantine counts.

The reference container deployment is in docker-compose.agent.yml.

Local development

The private conformance repository contains disposable Docker environments and live compatibility contracts. The core repository contains the installable package and its offline test suite:

git clone https://github.com/Berg-Cloud/berg.git
cd berg
uv sync --group dev
uv run pytest

See the conformance repository for local catalog stacks. See local development, the capability matrix, and the Python API guide for details.

The live backend matrix and disposable acceptance environments are maintained in the private Berg Conformance repository. See the repository topology for the ownership and migration rules.

Safety boundaries

Berg currently supports approved metadata operations such as properties, compatible schema additions, additive partition fields, certified snapshot expiration, and guarded orphan-file cleanup. Compaction, arbitrary destructive changes, and unsupported backend operations remain disabled or capability-gated.

The product roadmap and release boundaries are documented in V0_SCOPE.md, V1_SCOPE.md, V1.2_SCOPE.md, V1.3_SCOPE.md, V1.4_SCOPE.md, and docs/roadmap.md.

License and security

See SECURITY.md for vulnerability reporting and docs/security.md for the operational security model.