Enterprise corpus — codebase, Slack, meeting notes, and documentation — flowing continuously through the CocoIndex incremental sync engine into a production AI agent with always-fresh context. Only the Δ (delta) is reprocessed on every change. Keywords: RAG pipeline, agent memory, enterprise retrieval, AI agent context, live indexing, retrieval-augmented generation, production LLM apps, streaming ETL, incremental ingestion.

Your agents deserve fresh context.

Star us ❤️ → Star CocoIndex on GitHub — open-source Python framework for RAG, vector search, and live agent context  ·  cocoindex.io — the CocoIndex homepage: incremental data pipelines for AI agents  ·  CocoIndex documentation — quickstart, connectors, ops, transformations, target stores, RAG and knowledge graph recipes  ·  Join the CocoIndex Discord community — help, showcase, release notes, and live chat with maintainers

CocoIndex turns codebases, meeting notes, inboxes, Slack, PDFs, and videos into live, continuously fresh context for your AI agents and LLM apps to reason over effectively — with minimal incremental processing. Get your production AI agent ready in 10 minutes with reliable, continuously fresh data — no stale batches, no context gap

Incremental · only the delta  ·  Any scale · parallel by default  ·  Declarative · Python, 5 min

stars downloads pypi python rust license discord

CI release links

cocoindex-io/cocoindex | Trendshift




Built with CocoIndex ❤️

CocoIndex-code — flagship MCP server for AI coding agents. AST-aware incremental semantic code index that keeps live call graphs, symbols, vectors, and chunks fresh on every commit. 70% fewer tokens per turn, 80-90% cache hits on re-index, sub-second freshness. Supports Python, TypeScript, Rust, and Go. Features: Δ-only incremental processing, semantic search by meaning (not grep), call graphs and blast-radius analysis, global repo view for duplicates and architecture. Build coding agents (generate, refactor) and code-review agents (catch, approve). One install — Claude Code, Cursor, and other MCP-aware agents see your whole repository instantly. Keywords: MCP server, coding agent, code intelligence, AST chunking, semantic code search, call graph, vector embedding, repository context, Claude Code, Cursor, incremental indexing, blast radius.

See all 20+ examples · updated every week →


Get started

pip install -U cocoindex

Declare what should be in your target — CocoIndex keeps it in sync forever, recomputing only the Δ.

import cocoindex as coco
from cocoindex.connectors import localfs, postgres
from cocoindex.ops.text import RecursiveSplitter

@coco.fn(memo=True)                          # ← cached by hash(input) + hash(code)
async def index_file(file, table):
    for chunk in RecursiveSplitter().split(await file.read_text()):
        table.declare_row(text=chunk.text, embedding=embed(chunk.text))

@coco.fn
async def main(src):
    table = await postgres.mount_table_target(PG, table_name="docs")
    table.declare_vector_index(column="embedding")
    await coco.mount_each(index_file, localfs.walk_dir(src).items(), table)

coco.App(coco.AppConfig(name="docs"), main, src="./docs").update_blocking()

Run once to backfill. Re-run anytime — only the changed files re-embed.

Building with an AI coding agent?
Drop in our CocoIndex skill so your agent writes correct v1 code — concepts, APIs, patterns, all in one file.
See Use with AI coding agents for install steps.

Full quickstart — open-book icon linking to the CocoIndex documentation quickstart: pip install, declare sources and targets, run the incremental engine    Learn the concept — lightbulb icon linking to the CocoIndex core-concepts guide: sources, targets, flows, incremental engine, and data lineage

Animated GitHub Star button for the cocoindex-io/cocoindex repository: a cursor clicks the star, it fills yellow, confetti bursts, the star count ticks up, and an 'Appreciate a star if you like it!' caption with a beating heart shows below the button



React — for data engineering

React — for data engineering. The CocoIndex mental model: Target = F(Source). A persistent-state-driven dataflow where you declare the desired target state and the engine keeps it in sync with the latest source data and code, forever, at low latency and low cost. Source files (.py, .md, .pdf, .ts) flow through your Python transformation F into a live target dots-matrix index; only the Δ is reprocessed on every change, and every target dot traces back to its exact source byte. Four core properties: Python not a DAG (sky), declare target state (yellow bullseye), lineage end-to-end (coral connected dots), and incremental at any scale (mint Δ+1). Your code is as simple as the one-off version — the engine does the rest. Keywords: React for data engineering, declarative ETL, persistent state, data lineage, dataflow, Δ only, incremental indexing, CocoIndex.

What happens when either side changes — CocoIndex tracks per-row provenance so the Δ propagates at minimum cost. Two scenarios shown in one illustration: (top) Source change — one file (b.md) is edited and only one target dot re-syncs (coral pulse). (bottom) Code change — the transformation function F is rewritten from v1 to v2 and only the dots whose outputs depend on the changed code re-run (amber/yellow pulses). Source on the left, F in the center (Python code block), target dots-matrix on the right. Keywords: incremental indexing, change data capture, delta processing, fine-grained invalidation, code-aware caching, hash-of-code invalidation, memoization, reproducible pipelines, incremental recomputation.

See the React ↔ CocoIndex mental model →



Incremental engine for long-horizon agents

Data transformation for any engineer, designed for AI workloads —
with a smart incremental engine for always-fresh, explainable data.

Learn the concept — purple button with a lightbulb icon linking to the CocoIndex core-concepts guide: sources, targets, flows, incremental engine, and data lineage

CocoIndex's Python-native transformation flows connect 8 source categories (Codebases, Meeting Notes, Web · APIs, File System · Blob Stores, Databases, Message Queues, Images · Video, Voice · Transcripts) through the incremental engine out to 6 target stores (Relational DB, Data Warehouse, Vector DB, Graph DB, Message Queue, Feature Store). A flow.py code block (@coco.fn · def f(src): · chunks = split(src) · target.row(embed(chunks))) shows the shared pipeline; only the Δ is reprocessed — unchanged src hits the cache, changed src re-runs split() and Δ → re-embed. The persistent data-pipeline control plane runs eight always-on subsystems: live caching, pipeline catalog, version tracking, continuously learning, lineage, task scheduling, metrics collection, and failure management. Keywords: data pipeline, ETL, source connectors, vector database, graph database, incremental engine, streaming ingestion, caching, lineage, versioning, scheduling, metrics, retries.



Why incremental?

Your agents are only as good as the data they see.
Batch pipelines drift stale. CocoIndex stays live — and only runs the Δ.

Why incremental? — one illustration combining the four core benefits of CocoIndex's incremental engine. Sub-second fresh (mint): a stopwatch ticking under a second, source changes propagate to the target in under a second so agents see the world as it is, not as it was yesterday. 10× cheaper at scale (yellow): a 10,000-row corpus block where only a thin Δ 0.1% column re-runs and 99.9% stays cached — you skip the other 99.9% of your corpus and pay a fraction of the compute, embedding, and LLM bill. Explainable by default (coral): a lineage thread links a source byte (handbook.md L42) to a target vector — every vector, row, or graph node in the target traces back to its exact source byte for debuggable, auditable, regulator-friendly AI pipelines. Production-grade (purple): a shield stamped with the Rust crab surrounded by retry loops, back-off dots, a DLQ tray, and a no-data-loss check — Rust core with retries, exponential back-off, dead-letter queues, and no-data-loss guarantees, production-ready for long-horizon AI agents. Keywords: incremental indexing, Δ-only reprocessing, sub-second freshness, low-latency RAG, cost-efficient embeddings, data lineage, retrieval-augmented generation, Rust core, retries, back-off, dead letters, no data loss, long-horizon agents.



What can you build?

See all 20+ examples · updated every week →

Working starters from the examples tree — clone, plug your source, ship.

Real-time code index — walk a git repo, AST-chunk source files, embed with sentence-transformers, upsert to pgvector / LanceDB, incremental on every commit. Keywords: code search, code embedding, semantic code retrieval, Python.

PDF → RAG index — ingest PDFs from local, S3, or GDrive, extract + chunk text, embed chunks, upsert to pgvector / LanceDB. Classic retrieval-augmented-generation stack, incremental. Keywords: RAG, document Q&A, PDF search, vector database.

HN trending topics — pull Hacker News threads via Algolia, recursively parse comments, LLM-extract topics with Gemini 2.5 Flash, rank by weighted hit count (thread=5, comment=1), store in Postgres. Incremental. Keywords: Hacker News, trending topics, LLM extraction, Gemini, Postgres, news intelligence, topic ranking.

Conversation → knowledge graph — LLM extracts people, topics, decisions, action items from transcripts and upserts into Neo4j / Kuzu. Live graph, incremental. Keywords: knowledge graph, entity extraction, meeting intelligence, agent memory.

Multi-repo summarization — walk N git repos, extract structure, LLM-summarize per-repo + a rolled-up org summary, refresh on every push. Keywords: internal platform, developer experience, monorepo, SDK docs.

Structured extraction — BAML / DSPy typed schema extraction from forms, PDFs, intakes, invoices into Postgres / warehouse. Incremental. Keywords: ETL, LLM extraction, schema-first, patient intake, invoice processing, KYC, contracts.

Podcast → knowledge graph — transcribe YouTube / Spotify audio with speaker diarization, LLM-extract speakers and statements, resolve entities across episodes, store in SurrealDB / Neo4j. Keywords: podcast, diarization, YouTube, Whisper, SurrealDB, knowledge graph, entity resolution.

CSV → Kafka live — watch a folder of CSV files, publish each row as a JSON message to a Kafka topic via CocoIndex's Kafka target connector. Incremental, sub-second, no producer loop. Keywords: Kafka, CDC, streaming, StreamNative, Confluent, CSV ingestion, event streaming.


Share what you build — a banner with a trail of tiny hearts rising from the bottom behind the text, inviting the CocoIndex community to share projects built with the framework

Building something with CocoIndex? We want to see it.
Tag @cocoindex_io on X or drop a link in #showcase on Discord. We'll boost it. 🥥



Community

Join the CocoIndex Discord community — live chat with maintainers and users, showcase your projects, get help building RAG pipelines and knowledge graphs Subscribe to the CocoIndex YouTube channel — video tutorials, live demos, architecture deep dives, and AI agent recipes Read the CocoIndex blog — engineering deep dives, release notes, RAG and knowledge graph tutorials, and case studies Follow @cocoindex_io on X (formerly Twitter) for release notes, demos, launches, and AI data pipeline updates



We love Contributors — section title banner with a pulsing coral heart badge and cream twinkle sparkles. Every typo fix, new connector, and doc tweak makes CocoIndex better. Keywords: open-source contribution, pull request, typo fix, new connector, good first issue, Hacktoberfest, community, coconut heart.

We are so excited to meet you.
Every typo fix, new connector, doc tweak, or full-on rewrite makes CocoIndex better.
Come hang out — big PRs and small ones, both welcome.

📝 Read the contributing guide  ·  🐛 good first issues  ·  💬 Say hi on Discord



CocoIndex Enterprise

CocoIndex Enterprise — built for enterprise scale. Four headline stats for PB-scale incremental indexing: PB corpus scale incrementally indexed (coral), 10× fewer LLM embedding calls vs. full recompute (yellow), 100% lineage coverage with every byte traceable (mint), Δ only the delta always (sky). Below, a wide 50×8 corpus matrix of 400 dim tiles represents a petabyte-scale store where a single coral Δ slice of 8 tiles re-runs while the other 99.9% stays cached. Keywords: enterprise RAG, petabyte-scale indexing, incremental compute, delta-only, lineage, parallel chunking, zero-copy, failure isolation.

Large corpus — built for enterprise scale.

Incremental compute is the only way to keep large corpora fresh without re-embedding them every cycle.
CocoIndex scales from a single repo to petabyte-scale stores — parallel by default, delta-only by design.


Process once. Reconcile forever.

When a source changes, CocoIndex identifies the affected records, propagates the change
across joins and lookups, updates the target, and retires stale rows —
without touching anything that didn't change.


Built on a Rust engine.

The core is Rust — production-grade from day zero.
Parallel chunking, zero-copy transforms where possible, and failure isolation
so one bad record doesn't stall the flow.



Explore CocoIndex Enterprise — bright blue pill button linking to cocoindex.io/enterprise, the PB-scale incremental data pipeline for AI agents



Apache 2.0 · © CocoIndex contributors 🥥

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.0.20)

2 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for cocoindex from gravatar.com cocoindex Avatar for georgehe from gravatar.com georgehe

Credits

Author: CocoIndex

License

Apache Software License

Requires

Python >=3.11

Tags

indexing real-time incremental pipeline search ai etl rag dataflow context-engineering

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cocoindex-1.0.18.tar.gz (741.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

cocoindex-1.0.18-cp314-cp314t-win_amd64.whl (9.8 MB view details)

Uploaded CPython 3.14tWindows x86-64

cocoindex-1.0.18-cp314-cp314t-manylinux_2_28_x86_64.whl (9.6 MB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.28+ x86-64

cocoindex-1.0.18-cp314-cp314t-manylinux_2_28_aarch64.whl (9.4 MB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.28+ ARM64

cocoindex-1.0.18-cp314-cp314t-macosx_11_0_arm64.whl (9.5 MB view details)

Uploaded CPython 3.14tmacOS 11.0+ ARM64

cocoindex-1.0.18-cp311-abi3-win_amd64.whl (9.8 MB view details)

Uploaded CPython 3.11+Windows x86-64

cocoindex-1.0.18-cp311-abi3-manylinux_2_28_x86_64.whl (9.6 MB view details)

Uploaded CPython 3.11+manylinux: glibc 2.28+ x86-64

cocoindex-1.0.18-cp311-abi3-manylinux_2_28_aarch64.whl (9.4 MB view details)

Uploaded CPython 3.11+manylinux: glibc 2.28+ ARM64

cocoindex-1.0.18-cp311-abi3-macosx_11_0_arm64.whl (9.5 MB view details)

Uploaded CPython 3.11+macOS 11.0+ ARM64

cocoindex-1.0.18-cp311-abi3-macosx_10_12_x86_64.whl (9.4 MB view details)

Uploaded CPython 3.11+macOS 10.12+ x86-64

File details

Details for the file cocoindex-1.0.18.tar.gz.

File metadata

  • Download URL: cocoindex-1.0.18.tar.gz
  • Upload date:
  • Size: 741.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: maturin/1.14.1

File hashes

Hashes for cocoindex-1.0.18.tar.gz
Algorithm Hash digest
SHA256 f366a14438f3c581abf94a31b6cb96897e69f3d0776fd52da3ae20a9cc10a98f
MD5 124275728042b42efb762780648630fb
BLAKE2b-256 c3999e055aaa2ecf803d3bff3295ddc7613baba8073b23b4528776655758cf5a

See more details on using hashes here.

File details

Details for the file cocoindex-1.0.18-cp314-cp314t-win_amd64.whl.

File metadata

File hashes

Hashes for cocoindex-1.0.18-cp314-cp314t-win_amd64.whl
Algorithm Hash digest
SHA256 70c0448cc6281ac77188f37995b49a1a94c91cbd1fd9e2b1b6ae8d4ec1e72096
MD5 45f6f6faf00bdd7ef8ae71a892f3b9d1
BLAKE2b-256 a73e5754546e3fe77dc7ced2a5dca7c3e6732a4fa495cc2eed7d3d949b88d0f3

See more details on using hashes here.

File details

Details for the file cocoindex-1.0.18-cp314-cp314t-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for cocoindex-1.0.18-cp314-cp314t-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 28be1d33bb70592e1b989afd60346ac587e62f50d7b1ed3846c6855d885c93b7
MD5 d3df56619fb307d3c3457d8e4b08aa8b
BLAKE2b-256 c11ce2700f861d0ab2be62aa08e3c08cde193967afafec8bf8c1f7a938e5b92e

See more details on using hashes here.

File details

Details for the file cocoindex-1.0.18-cp314-cp314t-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for cocoindex-1.0.18-cp314-cp314t-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 49150befb5878fb7b99e325cbab0b35c46cacf3184c92ec6a44a7bf3725d4a58
MD5 092a345dd9ba8b78c90f05666752862c
BLAKE2b-256 57c4f2e1d9ea6561e254187a62f9021119d2ffedc88c48311813981e4f14c805

See more details on using hashes here.

File details

Details for the file cocoindex-1.0.18-cp314-cp314t-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for cocoindex-1.0.18-cp314-cp314t-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 74e137d1496db81ceee2ef600cb91d79d22bec68b5a1ca19fdc410c03e4d8f57
MD5 06fde6b3b7cd84fc5b2a8b313e541b66
BLAKE2b-256 2e9596876ea3f3d832df6cb70196b6de6f96c3c1f1cf8ca1b0c36dd8c6be60d3

See more details on using hashes here.

File details

Details for the file cocoindex-1.0.18-cp311-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for cocoindex-1.0.18-cp311-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 9546e2a5cf050692eb14accbcccfa9d235367ecd7efb071f43a8de46f7b981db
MD5 b1e4c0dfe4d24c559e399d270cb2709c
BLAKE2b-256 9452bfc4fd2dd838067d3b78ed30c7a8b8c2b4b5c17470737ab14e1588496cf5

See more details on using hashes here.

File details

Details for the file cocoindex-1.0.18-cp311-abi3-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for cocoindex-1.0.18-cp311-abi3-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 3830c748675672b55cd6edaa9c222d28ccd1a782ebc29ab08c3a187ee4b82059
MD5 1e24ca5c1c974a6648500322e956d5e6
BLAKE2b-256 ded083ddd620431c31c3224d8efa01475d7e92fb982aa2eaf453ab64f8d697e9

See more details on using hashes here.

File details

Details for the file cocoindex-1.0.18-cp311-abi3-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for cocoindex-1.0.18-cp311-abi3-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 e6f20760676a39ea069e7659c8e8fc8d9c296165097839e5806a80c4e376ebc0
MD5 a34f77a22277289da174d9a568a4e3ca
BLAKE2b-256 527cd44c4c62da545d3eaa97691100dcd164c793c2bc0486a5082bbd40a9aa0b

See more details on using hashes here.

File details

Details for the file cocoindex-1.0.18-cp311-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for cocoindex-1.0.18-cp311-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 b9102afba6e6f66d7958bb4fc01bd04db7b4807ef25b1101f39be879efcae600
MD5 34f71d091c848e8c33479374bb2eaf9e
BLAKE2b-256 0dfd51e464e9705cd4ede622bd610efe4ec86fea6a1bec9e28ed697e86425732

See more details on using hashes here.

File details

Details for the file cocoindex-1.0.18-cp311-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for cocoindex-1.0.18-cp311-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 30e669eb179781a08c300a0d868ad0e624d8383fee8f009df1adb3325e6d79f5
MD5 76ccdff96c452452ac462520193bf488
BLAKE2b-256 cac31c2f230578bbe3b207eed5b2c32b6baa86dec8be6f76165d4ff88d00dbb4

See more details on using hashes here.

Release history Release notifications | RSS feed

1.0.20

1.0.19

This release

1.0.18

1.0.17

1.0.16

1.0.15

1.0.14

1.0.13

1.0.11

1.0.10

1.0.9

1.0.8 yanked

1.0.7

1.0.6

1.0.5

1.0.4

1.0.3

1.0.2

1.0.1

1.0.0

1.0.0a50 pre-release

1.0.0a49 pre-release

1.0.0a48 pre-release

1.0.0a47 pre-release

1.0.0a46 pre-release

1.0.0a45 pre-release

1.0.0a44 pre-release

1.0.0a43 pre-release

1.0.0a42 pre-release

1.0.0a41 pre-release

1.0.0a40 pre-release

1.0.0a39 pre-release

1.0.0a38 pre-release

1.0.0a37 pre-release

1.0.0a36 pre-release

1.0.0a35 pre-release

1.0.0a34 pre-release

1.0.0a33 pre-release

1.0.0a32 pre-release

1.0.0a31 pre-release

1.0.0a30 pre-release

1.0.0a29 pre-release

1.0.0a28 pre-release

1.0.0a27 pre-release

1.0.0a26 pre-release

1.0.0a25 pre-release

1.0.0a24 pre-release

1.0.0a23 pre-release

1.0.0a22 pre-release

1.0.0a21 pre-release

1.0.0a20 pre-release

1.0.0a19 pre-release

1.0.0a18 pre-release

1.0.0a17 pre-release

1.0.0a16 pre-release

1.0.0a15 pre-release

1.0.0a14 pre-release

1.0.0a13 pre-release

1.0.0a12 pre-release

1.0.0a11 pre-release

1.0.0a10 pre-release

1.0.0a9 pre-release

1.0.0a8 pre-release

1.0.0a7 pre-release

1.0.0a6 pre-release

1.0.0a5 pre-release

1.0.0a4 pre-release

1.0.0a2 pre-release

1.0.0a1 pre-release

0.3.39

0.3.38

0.3.37

0.3.36

0.3.35

0.3.34

0.3.33

0.3.32

0.3.31

0.3.30

0.3.29

0.3.28

0.3.27

0.3.26

0.3.25

0.3.24

0.3.23

0.3.22

0.3.21

0.3.20

0.3.19

0.3.18 yanked

Reason this release was yanked:

bug

0.3.17

0.3.16

0.3.15

0.3.14

0.3.13

0.3.12

0.3.11

0.3.10

0.3.9

0.3.8

0.3.7

0.3.6

0.3.5

0.3.4

0.3.3

0.3.2 yanked

Reason this release was yanked:

0.3.3 makes batching more stable on Gemini

0.3.1 yanked

Reason this release was yanked:

0.3.3 makes batching more stable on Gemini

0.3.0

0.2.23

0.2.22

0.2.21

0.2.20

0.2.19

0.2.18

0.2.17

0.2.16

0.2.15

0.2.14

0.2.13

0.2.12

0.2.11

0.2.10

0.2.9

0.2.8

0.2.7

0.2.6

0.2.5

0.2.4

0.2.3

0.2.2

0.2.1

0.2.0

0.1.83

0.1.82

0.1.81

0.1.80

0.1.79

0.1.78

0.1.77

0.1.76

0.1.75

0.1.74

0.1.73

0.1.72

0.1.71

0.1.70

0.1.69

0.1.68

0.1.67

0.1.66

0.1.65

0.1.64

0.1.63

0.1.62

0.1.61

0.1.60

0.1.59

0.1.58

0.1.57

0.1.56

0.1.55

0.1.54

0.1.53

0.1.52

0.1.51

0.1.50

0.1.49

0.1.48

0.1.47

0.1.46

0.1.45

0.1.44

0.1.43 yanked

Reason this release was yanked:

bug

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page