airbyte-destination-ailake

Airbyte destination connector for AI-Lake Format — writes Airbyte records to AI-Lake vector tables (Apache Iceberg + HNSW/IVF-PQ).

What it does

Each Airbyte stream maps to one AI-Lake table at {table_base_path}/{stream_name}/. For every record batch the connector:

1. Extracts the text field from each record (text_field, dot-notation supported for nested fields).
2. Embeds texts via the configured backend (cmd, openai, or cohere).
3. Writes the batch to the AI-Lake table (Parquet + HNSW index in the footer).
4. Commits an Iceberg snapshot on each Airbyte state message and at the end of the sync.

Tables created this way are fully compatible with Apache Iceberg — Spark, Trino, DuckDB, and PyIceberg can read tabular data without the AI-Lake SDK. The AI-Lake SDK activates vector-search.

Installation

pip install airbyte-destination-ailake          # core (cmd/http embed modes only)
pip install "airbyte-destination-ailake[openai]"  # + OpenAI Embeddings API
pip install "airbyte-destination-ailake[cohere]"  # + Cohere Embed API
pip install "airbyte-destination-ailake[fastembed]"              # + local ONNX embeddings, no API key
pip install "airbyte-destination-ailake[sentence-transformers]"  # + local PyTorch embeddings, no API key

Configuration

A record whose embedding comes back with a NaN/Infinity value (a malformed embed_cmd response, or an API edge case) is rejected at write time with a clear error instead of being silently written — the sync fails visibly rather than producing a table with a corrupted vector.

Embedding modes

cmd — custom command

{
  "embed_mode": "cmd",
  "embed_cmd": "python my_embedder.py"
}

The command must read a JSON string array from stdin and write a JSON float-array-of-arrays to stdout:

# stdin:  ["text one", "text two"]
# stdout: [[0.01, -0.02, ...], [0.03, 0.04, ...]]

openai — OpenAI Embeddings API

{
  "embed_mode": "openai",
  "openai_api_key": "sk-...",
  "openai_model": "text-embedding-3-small",
  "embedding_dim": 1536
}

For Azure OpenAI set openai_base_url to your deployment endpoint.

cohere — Cohere Embed API

{
  "embed_mode": "cohere",
  "cohere_api_key": "...",
  "cohere_model": "embed-english-v3.0",
  "embedding_dim": 1024
}

fastembed — local ONNX embeddings (no API key)

{
  "embed_mode": "fastembed",
  "fastembed_model": "BAAI/bge-small-en-v1.5",
  "embedding_dim": 384
}

Runs entirely in-process via ONNX Runtime — no PyTorch dependency, smallest install footprint of the local options. Good default for CI/demos and low-memory environments. Requires pip install "airbyte-destination-ailake[fastembed]".

sentence-transformers — local embeddings (no API key)

{
  "embed_mode": "sentence_transformers",
  "sentence_transformers_model": "BAAI/bge-small-en-v1.5",
  "sentence_transformers_device": "",
  "embedding_dim": 384
}

PyTorch-based — widest model selection (any Hugging Face sentence-embedding model), supports GPU via sentence_transformers_device: "cuda". Heavier install than fastembed. Requires pip install "airbyte-destination-ailake[sentence-transformers]".

Both local modes default to BAAI/bge-small-en-v1.5 (dim=384) — a small, fast, well-regarded open model. Neither requires an API key, an internet connection after the first model download, or any external service.

Running locally (CLI)

# Check connection
airbyte-destination-ailake check --config config.json

# Write records from catalog + stdin messages
airbyte-destination-ailake write --config config.json --catalog catalog.json

Docker

docker build -t airbyte-destination-ailake:latest .
docker run --rm -v $(pwd):/data airbyte-destination-ailake:latest check --config /data/config.json

Reading the tables after sync

import ailake

results = ailake.search(
    table="s3://my-lake/airbyte/my_stream/",
    query=my_query_embedding,
    top_k=10,
)

Or with any Iceberg-compatible engine (no AI-Lake SDK needed for tabular access):

import duckdb
duckdb.sql("INSTALL iceberg; LOAD iceberg;")
duckdb.sql("SELECT content FROM iceberg_scan('s3://my-lake/airbyte/my_stream/')")

Development

pip install -e ".[dev,openai,cohere]"
pytest tests/

License

MIT OR Apache-2.0