Skip to main content

Goose

A columnar storage engine for analytical workloads — 10 specialized column encodings, roaring-bitmap predicate pushdown, bloom-filter partition skipping, zone-map pruning, probabilistic sketches, and an SQL query advisor. Ships with a PostgreSQL profiler/exporter CLI.

CI License: MIT Python 3.10+

Goose is a universal booster: a column-encoding and predicate-pushdown layer you can use three ways —

  1. As a Python library — point it at data, let it pick the best encoding per column, write a compressed columnar store, and query it with predicate pushdown.
  2. As a PostgreSQL cold tier — profile a Postgres table, export it to Goose, and query the compressed copy (CDC keeps them in sync).
  3. As an embedded encoding library — link the C (libgoose.a) or Rust (goose-encoding) reference implementation into another database to get encoding intelligence and predicate pushdown without changing your storage engine.

The binary format, encoding selection algorithm, and predicate evaluation logic are specified language-agnostically in reference/SPEC.md, so any database can implement a compatible reader/writer.


Why

PostgreSQL is great at OLTP and terrible at analytical scans over years of history. Goose takes the analytical workload off Postgres: a verb_id column with 6 distinct values compresses 58×; a monotonic timestamp column compresses 10.8× with FOR+ZSTD; a sparse boolean flag becomes a roaring bitmap you can filter without decoding the column. Predicate pushdown then skips whole partitions using bloom filters and zone maps before touching row data.

Features

  • 10 column encodings — raw, bitpacked, dictionary, offset_blob, delta_zstd, zstd, for (frame-of-reference), for_zstd, roaring (roaring-bitmap booleans), zstd_dict (shared trained dictionary).
  • Predicate pushdown — eq, neq, gt, gte, lt, lte, in, compound and/or. Each predicate produces a roaring bitmap of matching rows; predicates are intersected, never materializing the full column.
  • Partition skipping — per-partition and per-block bloom filters + zone maps let the reader rule out entire partitions before decode.
  • Adaptive selectivity feedback — observed predicate selectivities reorder filters so the most selective (cheapest) predicates run first.
  • Cross-column correlation statistics — propagate partition pruning across correlated columns.
  • Probabilistic sketches — approximate answers (count-distinct, etc.) for fast exploration.
  • Atomic writes + CRC32 checksums — every file is written to a .tmp then renamed; partition manifests carry checksums, verified on read.
  • SQL query advisor — goose.advisor.optimize() takes a SQL WHERE clause, reorders predicates by selectivity, and returns optimized SQL + a structured predicate tree for GooseTable.query().
  • PostgreSQL integration — goose-pg CLI to profile, export, benchmark, and query.

Install

pip install weji-goose                 # core engine (numpy, zstandard, sqlglot)
pip install "weji-goose[pg]"           # + psycopg2-binary for the goose-pg CLI
pip install "weji-goose[dev]"          # + pytest for running the test suite

Note: the PyPI distribution is weji-goose; the import name is import weji_goose.

Requires Python ≥ 3.10.

Quick start — Python library

import numpy as np
from weji_goose.schema import TableSchema
from weji_goose.column import ColumnSpec, ColumnType, Encoding
from weji_goose.table import GooseTable
from weji_goose.query import Predicate, CompoundPredicate

schema = TableSchema("demo", columns=[
    ColumnSpec("id",    ColumnType.INT64,   Encoding.DELTA_ZSTD),  # monotonic → delta
    ColumnSpec("flag",  ColumnType.BOOLEAN, Encoding.ROARING),     # sparse bool → bitmap
    ColumnSpec("city",  ColumnType.TEXT,    Encoding.DICTIONARY),  # low-card text → dict
])

# Create and write
table = GooseTable.create(schema, "./demo_table")
table.insert({
    "id":   np.arange(1_000_000, dtype=np.int64),
    "flag": np.array([i % 5 == 0 for i in range(1_000_000)], dtype=bool),
    "city": np.array(["stjohns", "corner", "bay"] * 333_333 + ["stjohns"], dtype=object),
})

# Predicate pushdown: filter without decoding the columns
table = GooseTable.open("./demo_table")
result = table.query(
    ["city", "id"],
    where=CompoundPredicate("and", [
        Predicate("flag", "eq", True),
        Predicate("id",   "gte", 999_000),
    ]),
)
print(result["city"][:5], result["id"][:5])
print("on-disk size:", table.total_size_bytes(), "bytes")

Supported types

int64, int32, int16, float64, float32, boolean, uuid, text, interval.

Quick start — PostgreSQL cold tier (goose-pg)

# 1. Profile a table → schema + per-column compression estimates
goose-pg profile \
    --db-url "postgresql://user:pass@host/db" \
    --table sensor_readings \
    --output ./sensor_readings_schema.json -v

# 2. Export the table to Goose format (auto-partitioned, resumable)
goose-pg export \
    --db-url "postgresql://user:pass@host/db" \
    --table sensor_readings \
    --output-dir ./goose_data \
    --partition-column recorded_at --partition-interval month --resume -v

# 3. Benchmark: profile + export + measured compression ratio
goose-pg benchmark \
    --db-url "postgresql://user:pass@host/db" \
    --table sensor_readings \
    --output-dir ./bench

# 4. Query the compressed Goose data with predicate pushdown
goose-pg query \
    --goose-dir ./goose_data/sensor_readings \
    --columns recorded_at,pm25 \
    --where "pm25 >= 35 AND region = 'bay_st_george'" \
    --format csv

goose-pg query --where parses the SQL WHERE clause through the Goose advisor (using the opened table's schema) and pushes it down. Output formats: table (default), csv, json.

A docker-compose.yml is included for integration testing against a throwaway Postgres. The goose service is profile-gated under cli, so pass --profile cli to invoke it:

docker compose up -d postgres
docker compose --profile cli run --rm goose --help
docker compose --profile cli run --rm goose query --goose-dir ./goose_data/mytable --columns id,name

The query advisor

goose.advisor.optimize(sql, schema) reorders WHERE predicates by selectivity — roaring-bitmap lookups first, then bloom-filtered ID lookups, then range predicates — and returns optimized SQL plus a predicate tree:

from weji_goose.advisor import optimize, SCHEMAS

oq = optimize(
    "SELECT id, verb_id FROM xapi_events "
    "WHERE verb_id IN ('completed', 'passed') AND actor_id = 42",
    schema=SCHEMAS["xapi_events"],
)
print(oq.optimized_sql)
# SELECT id, verb_id
# FROM xapi_events
# WHERE actor_id = 42 AND verb_id IN ('completed', 'passed')

print(oq.predicate_json)
# {'op': 'and', 'predicates': [
#     {'column': 'actor_id', 'op': 'eq',  'value': 42},
#     {'column': 'verb_id',  'op': 'in',  'value': ['completed', 'passed']}]}

OptimizedQuery exposes: original_sql, optimized_sql, predicate_json, table_name, selected_columns, warnings.

Encodings

Encoding Best for
raw High-cardinality, incompressible numeric data
bitpacked Dense booleans (8 values/byte)
dictionary Low-cardinality text/categorical (≈ < 50k distinct)
offset_blob Variable-length blobs with a dictionary offset index
delta_zstd Monotonic / near-sequential int64 (timestamps, IDs)
zstd Generic high-entropy compressible data
for Frame-of-reference: clustered int ranges
for_zstd FOR + ZSTD for clustered ints that still compress
roaring Sparse booleans — queryable without column decode
zstd_dict Repeated text patterns via a shared trained dictionary

Benchmarks

On a 10,000-row synthetic sample (weji_goose/benchmarks/benchmark_results.json, reproducible via python -m weji_goose.benchmarks.demo):

Metric Value
Overall compression vs PG 3.83×
Write throughput ~10,000 rows/s
Full-scan throughput ~3.96M rows/s
Best per-column compression verb_id → 58.5× (dictionary, 6 distinct)
Column Encoding Compression
verb_id dictionary 58.5×
context_org dictionary 14.8×
stored delta_zstd 10.8×
actor_id for 8.0×
object_id dictionary 7.4×

Project layout

weji_goose/            core engine (import as `import weji_goose`)
  schema.py            TableSchema, ColumnType, Encoding enums
  table.py             GooseTable: create / open / insert / query / scan
  query.py             predicate pushdown engine, Predicate / CompoundPredicate
  advisor.py           SQL WHERE → optimized predicates
  reader.py writer.py  checksummed, atomic, mmap-backed I/O
  bloom.py roaring.py  partition skipping + row bitmaps
  sketch.py            probabilistic sketches
  correlation.py       cross-column pruning stats
goose_pg/              PostgreSQL integration
  cli.py               goose-pg CLI: profile / export / benchmark / query
  profiler.py          schema introspection + data profiling
  exporter.py          bulk export to Goose partitions
  type_map.py          PostgreSQL → Goose type/encoding mapping
reference/
  SPEC.md              language-agnostic binary format + algorithm spec
  c/                   C reference (libgoose.a / libgoose.so + tests)
  rust/                Rust reference crate (goose-encoding)
tests/                 272 tests

Reference implementations & spec

Goose's on-disk format and algorithms are defined in reference/SPEC.md so any database can implement a compatible reader/writer. Reference implementations live under reference/:

  • C — reference/c/ builds libgoose.a / libgoose.so (make, make test).
  • Rust — reference/rust/ is the goose-encoding crate (cargo build).

These are maintained alongside the Python engine as the canonical cross-language contract.

Development

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,pg]"
pytest -q          # 272 tests

The Python reference encoders/decoders are in weji_goose/; the C and Rust references are in reference/. Benchmarks: python -m weji_goose.benchmarks.demo.

License

MIT © 2026 WEJI Northern Technologies Inc.

Metadata

Release files for weji-goose 0.3.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for weji-goose 0.3.1
File Size Uploaded Provenance Actions
weji_goose-0.3.1.tar.gz 112.8 kB With provenance Details for weji_goose-0.3.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for weji-goose 0.3.1
File Interpreter ABI Platform Actions
weji_goose-0.3.1-py3-none-any.whl Python 3 none any Details for weji_goose-0.3.1-py3-none-any.whl

Total release size: 207.6 kB

Release files / weji_goose-0.3.1.tar.gz

Download URL weji_goose-0.3.1.tar.gz
Size 112.8 kB
Tags Source
SHA-256 checksum
How to use checksums
cd52a81e43d3d9e1a769a8929deadc932f951b3402371c58a700fa92aabdf627
BLAKE2b-256 checksum
How to use checksums
cd94c644f5a1ad1dbf17cdbd9ac88fb5b95757773e32d3f769f36d2c444cf4b9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 11, 2026.

Transparency log

Release files / weji_goose-0.3.1-py3-none-any.whl

Download URL weji_goose-0.3.1-py3-none-any.whl
Size 94.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e394f0d5fc8b01f3266bed6bf04d105ab485eeda4d9a775c9b850083cf620819
BLAKE2b-256 checksum
How to use checksums
b32004a3aab28446d79b4d74df683311b4a8c4e5c3e698daff9eaefb449fadda
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 11, 2026.

Transparency log

Security for weji-goose 0.3.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for weji_goose-0.3.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 11, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.3.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page