geneva 0.10.0

Geneva - Multimodal Data Lake for AI

Geneva - Multimodal Data Platform

Geneva is a petabyte-scale multimodal feature engineering and data management platform built on LanceDB.

What lives in this repo

• src/ - Geneva client library and core runtime
• src/tests/ - unit test suites
• src/integ_tests/ - integration test suites
• src/stress_tests/ - stress and load tests
• docs/ - mkdocs configuration and API docs (source for autogenerated public facing api documentation page)
• e2e/ - end-to-end test suites and UDF manifests
• notebook/ - quickstart notebooks (TODO replace with link to colab demo notebook)
• internal_docs/ - internal design and operational notes
• tools/ - helper scripts for local clusters and cleanup

User facing documentation should be submitted here.

Quickstart (local development)

uv sync --all-groups --all-extras --locked

import geneva
import pyarrow as pa

@geneva.udf(data_type=pa.int32())
def double(x: int) -> int:
    return x * 2

conn = geneva.connect("./db")
table = conn.create_table("numbers", [{"x": i} for i in range(10)])
table.add_columns({"doubled": double})
with conn.local_ray_context():
    table.backfill("doubled")
result = table.search().select(["doubled"]).to_arrow()

Development

See Development for details.

Configuration

Geneva supports specifying configuration in a few different ways. Refer to CONFIGURATION.md for more details.