refresh-engine

Keep anything in sync with its source by refreshing only what changed.

refresh-engine is an asyncio-first, domain-neutral Python library for keeping derived data current: search indexes, caches, embeddings, configuration, registries, or anything else built from a source. You describe how to list and read resources and how to apply a change; the engine detects what changed, plans the work in dependency order, runs it safely, and remembers the outcome.

Features

• Precise change detection. Deterministic fingerprints detect added, modified, and deleted resources. Reliable cheap indicators, such as ETags or version tokens, skip reading unchanged resources entirely.
• Four refresh modes. Incremental, full, dependency-aware, and targeted refreshes, with targets chosen by ID or by composable selectors.
• Dependency ordering. Dependencies are refreshed before their dependents, changes propagate to everything that depends on them, and cycles are isolated.
• Safe by default. Bounded concurrency, retries with jittered exponential backoff, timeouts, and fail-fast or best-effort policies. A partial discovery never deletes anything, and failed work is retried by the next refresh.
• Durable state. State advances only for successful resources, in one atomic commit per refresh. Use the in-memory store or plug in your own database.
• Scheduling and overlap control. A drift-free interval scheduler and policies that coalesce, queue, skip, or supersede overlapping requests.
• Observability. Typed lifecycle events with correlation IDs, structured logging through structlog with secret redaction, metrics, and optional OpenTelemetry spans and metrics using application-owned providers.
• Lightweight and typed. Fully type-annotated, with two runtime dependencies: structlog and tenacity.

Installation

refresh-engine requires Python 3.12 or newer.

pip install refresh-engine

For the optional OpenTelemetry integration, install refresh-engine[otel].

Quick start

Implement a source that lists and reads resources, and an operation that applies each change. The engine does the rest:

import asyncio

from refresh_engine import (
    DiscoveryResult,
    PlanAction,
    RefreshEngine,
    ResourceSource,
    RefreshRequest,
    Resource,
    ResourceSnapshot,
)

CATALOG = {"apples": 120, "pears": 80}
SEARCH_INDEX: dict[str, int] = {}


class InventorySource(ResourceSource):
    async def discover(self) -> DiscoveryResult:
        return DiscoveryResult(Resource(item) for item in sorted(CATALOG))

    async def snapshot(self, resource: Resource, /) -> ResourceSnapshot:
        return ResourceSnapshot(
            resource.resource_id, content=CATALOG[resource.resource_id]
        )


async def update_index(
    resource: Resource,
    snapshot: ResourceSnapshot | None,
    action: PlanAction,
    request: RefreshRequest,
    /,
) -> None:
    if action is PlanAction.DELETE:
        SEARCH_INDEX.pop(resource.resource_id, None)
    elif snapshot is not None:
        SEARCH_INDEX[resource.resource_id] = snapshot.content


async def main() -> None:
    async with RefreshEngine(InventorySource(), update_index) as engine:
        first = await engine.refresh()
        print(f"{first.status}: refreshed {first.refreshed_count}", SEARCH_INDEX)

        CATALOG["pears"] = 75
        del CATALOG["apples"]
        second = await engine.refresh()
        print(
            f"{second.status}: modified {second.modified_count},"
            f" deleted {second.deleted_count}",
            SEARCH_INDEX,
        )


asyncio.run(main())

success: refreshed 2 {'apples': 120, 'pears': 80}
success: modified 1, deleted 1 {'pears': 75}

The second refresh touches only the modified and the deleted resource. Continue with the getting started guide to add durable state, dependencies, scheduling, and logging.

Performance

The engine's overhead is small next to real I/O: with in-memory sources, a full refresh of 100,000 resources takes about 3 seconds, and an incremental refresh that finds nothing changed takes about half a second. See the benchmarks for the method and details.

Documentation

The documentation covers:

• Getting started and concepts
• Guides for every feature, including a production checklist
• Runnable examples, including SQLite and PostgreSQL state stores
• The API reference
• The architecture and its guarantees

Agent Skills

Coding agents such as Claude Code, Codex, Cursor, and GitHub Copilot can use the refresh-engine Agent Skill to integrate the library correctly:

npx skills add https://github.com/smuniharish/refresh-engine/tree/master/refresh-engine-skills/skills/refresh-engine

Contributing

Contributions are welcome. Read the contributing guide, and report vulnerabilities as described in the security policy. Notable changes are listed in the changelog.

License

refresh-engine is licensed under the Apache License 2.0.