rusty-git

High-speed pure Rust Git engine (gix + git2) compiled via PyO3 with both native async (Tokio) and GIL-released sync APIs.

rusty-git provides in-process Git operations with zero OS subprocess forks and statically-compiled native wheels (no dynamic libgit2.so dependencies required).

Installation

pip install rusty-git
# or with uv:
uv add rusty-git

Quickstart: 100% In-Process Demo (Zero Subprocesses)

Copy-paste this into example.py to test immediately in any environment:

import asyncio
import os
import tempfile
import rusty_git


async def main():
    with tempfile.TemporaryDirectory() as tmp_dir:
        repo_path = os.path.join(tmp_dir, "demo.git")

        # 1. Initialize a bare repository in-process (ZERO subprocesses)
        await rusty_git.init_bare_repository(repo_path)

        # 2. Create the initial root commit in-process
        res = await rusty_git.create_commit(
            repo_path,
            branch="main",
            files={
                "README.md": b"# Hello from rusty-git!\n",
                "src/main.py": b"print('Hello World')\n",
            },
            message="Initial commit",
            author_name="Alice",
            author_email="alice@example.com",
        )
        print("Root Commit SHA:", res["commit_sha"])

        # 3. Inspect commit in-process
        commit = await rusty_git.get_commit(repo_path, "HEAD")
        print(f"Author:  {commit['author_name']} <{commit['author_email']}>")
        print(f"Message: {commit['message']}")

        # 4. Read tree blob without disk checkout
        blob = await rusty_git.get_tree_blob(repo_path, "HEAD", "README.md")
        print("Blob Content:", blob.decode("utf-8").strip())

        # 5. List branches
        branches = await rusty_git.list_branches(repo_path)
        print("Branches:", branches)


asyncio.run(main())

Dual-Color Usage: Async & Sync

1. Asynchronous API (Default - for Litestar, FastAPI, Granian, Uvicorn)

All blocking Git I/O is offloaded to a Tokio multi-threaded worker pool, releasing the Python GIL:

import asyncio
import rusty_git


async def main():
    repo_path = "/path/to/your/repo"

    # 1. Inspect commits
    commit = await rusty_git.get_commit(repo_path, "HEAD")
    print(commit["sha"], commit["message"], commit["author_name"])

    # 2. Read tree blobs without disk checkout
    readme = await rusty_git.get_tree_blob(repo_path, "main", "README.md")
    print(readme.decode("utf-8"))

    # 3. Fast unified diffs
    hunks = await rusty_git.compare_refs(repo_path, "main~1", "main")
    for hunk in hunks:
        print(hunk["new_path"], hunk["content"])

    # 4. Dry run: show conflicts before anyone clicks "Merge"
    preview = await rusty_git.preview_merge(repo_path, base_ref="main", head_ref="feature")
    if preview["status"] == "conflict":
        print("Conflicts:", preview["conflicts"])  # nothing was written
    else:
        print("Would produce tree:", preview["tree_sha"])

    # 5. 3-Way Merge (update_ref=True moves `main` to the result atomically).
    #    strategy: "merge" (default) | "squash" | "fast-forward"
    result = await rusty_git.execute_merge(
        repo_path,
        base_ref="main",
        head_ref="feature",
        author_name="Automated Merge",
        author_email="bot@example.com",
        update_ref=True,
        strategy="squash",
    )
    if result["status"] == "merged":
        print("Merged Commit SHA:", result["commit_sha"], "ref moved:", result["ref_updated"])
    elif result["status"] == "up-to-date":
        print("Already up to date at", result["commit_sha"])


asyncio.run(main())

2. Synchronous API (rusty_git.sync - for CLI tools, Celery, scripts)

Direct execution on the calling thread with the Python GIL cleanly released (py.detach):

import rusty_git

# Read commit synchronously
commit = rusty_git.sync.get_commit("/path/to/repo.git", "HEAD")
print(commit["sha"], commit["author_name"])

# List branches synchronously
branches = rusty_git.sync.list_branches("/path/to/repo.git")
print("Branches:", branches)

# Read file directly from commit tree
blob = rusty_git.sync.get_tree_blob("/path/to/repo.git", "HEAD", "pyproject.toml")

# Create a commit directly on any branch (new or existing)
res = rusty_git.sync.create_commit(
    "/path/to/repo.git",
    branch="feature-x",
    files={"hello.txt": b"Hello world!\n"},
    message="feat: add hello.txt",
)

# 3-Way Merge synchronously (strategy: "merge" | "squash" | "fast-forward")
result = rusty_git.sync.execute_merge(
    "/path/to/repo.git",
    "main",
    "feature",
    author_name="Merge Bot",
    author_email="bot@example.com",
    update_ref=True,  # move `main` to the merge result under the ref lock
    strategy="fast-forward",  # raise GitRepositoryError instead of merging
)

# Dry run first - writes nothing (no tree, commit, or ref)
preview = rusty_git.sync.preview_merge("/path/to/repo.git", "main", "feature")
print(preview["status"], preview["conflicts"], preview["tree_sha"])

Architectural Note: Forges & Branch Management

rusty-git is designed for high-performance server-side forges and tools operating on bare repositories:

• Creating branches with commits: Passing any branch name to create_commit(...) will create or advance that branch atomically.
• Creating a branch without a commit: update_ref(repo_path, "feature", sha) moves refs/heads/feature through a gix ref transaction — it takes the same ref locks as git, appends to the reflog, and keeps packed-refs consistent. Never write refs/heads/* by hand: that bypasses locking, races with concurrent pushes, and resurrects deleted refs once the name is also in packed-refs. Pass expected_old="" to create-only, a 40-hex SHA for compare-and-swap, and target=None to delete. Non-fast-forward branch moves are refused unless force=True.
• Switching active branch (HEAD): set_head(...) moves the symbolic HEAD reference under the same transaction. Editing the HEAD file directly has the same hazards as writing a branch by hand.
• Merge strategies: execute_merge(..., strategy=...) offers the three forge buttons. "merge" (default) fast-forwards when it can and otherwise writes a two-parent merge commit; "squash" always writes exactly one commit on the base, defaulting to the concatenated feature commit summaries; "fast-forward" refuses anything that is not a fast-forward with GitRepositoryError (cannot fast-forward). A head that is already contained in the base reports "up-to-date" — a success, not a conflict.
• Dry-run merges: preview_merge(repo_path, base_ref, head_ref) runs the same ORT tree merge and writes nothing — no tree, no commit, no ref — returning status, conflicts, and the tree_sha the merge would produce. Use it to show conflicts before the user clicks "Merge".

Engine Architecture: Pure gix vs. git2 Wire Streaming

rusty-git intentionally separates user-facing operations from network pack streaming:

• 100% Pure gix Core (WebAssembly / Pyodide Ready): commit (atomic in-memory trees & commits), inspect (refs, tree blobs, histogram unified diffs), merge (3-way ORT tree merge, dry-run previews, and the merge/squash/fast-forward strategies) and ops (has_object, repository_stats, fsck, pack_refs) run entirely on pure-Rust gix. They require zero C headers and run in browsers and Pyodide.
• Wire Protocol Pack Streaming (git2): Ingesting thin packs (git push) and streaming sliding-window delta packs (git clone/git fetch) use libgit2's battle-tested pack engine. Once gix-pack high-level streaming pack generation reaches full maturity, these remaining wire endpoints will transition to gix, completing the journey to 100% pure Rust.
• Native-only maintenance (git2): repack and gc reuse that same pack engine — folding a whole object database into one delta-compressed pack is exactly the code not to hand-roll — so the gc module is compiled only for non-wasm32 targets (#[cfg(not(target_arch = "wasm32"))]), like the wire modules. fork_repository stays portable: it copies object and ref files (gix + std::fs) and never rewrites a pack.

This keeps rusty-git focused entirely on performance-critical operations without unnecessary file-I/O wrapper overhead.

Scope and Non-Goals

rusty-git is a Git engine, not a forge and not a web framework. That boundary is deliberate: the library does Git work in-process and the application owns everything around it.

In scope

• In-process Git object, ref and merge operations on bare repositories — create_commit, update_ref, set_head, execute_merge, preview_merge, the inspect read surface, and the ops maintenance primitives (has_object, repository_stats, fsck, pack_refs, fork_repository, repack, gc — the last two native-only, see the architecture note above).
• Smart-HTTP wire serving as bytes in / bytes out (advertise, upload_pack, receive_pack) with zero OS subprocesses.
• Engine paths that stay pure gix and therefore compile for wasm32 (Pyodide) as well as native.

Out of scope - the application's job

If a downstream project asks for one of these, the answer is the table above plus the recommended application-side approach - not a new export. The repo-root TODO-missing.md is the backlog and outcome log for the gaps that are in scope, and a boundary change has to be recorded there before it becomes a task.

Smart-HTTP Protocol Wire Support

rusty-git can drive Git HTTP and SSH protocol negotiation completely in-process:

Note on Transport & Auth (SSH / TLS): rusty-git operates directly on request/response byte streams rather than managing sockets. Transport encryption and authentication (SSH keys via asyncssh/OpenSSH, or TLS/mTLS client certs via Nginx/Caddy/HTTPX) happen cleanly at the transport layer before passing payloads to rusty-git.

import rusty_git

# Ref advertisement for git-upload-pack (git clone / fetch)
advert = await rusty_git.advertise("/srv/repos/myorg/backend.git", "git-upload-pack")

# Handle pack negotiation and streaming
upload_res = await rusty_git.upload_pack("/srv/repos/myorg/backend.git", request_body)

# Ingest inbound git push packfile atomically
push_res = await rusty_git.receive_pack("/srv/repos/myorg/backend.git", push_body)
print("Updated Refs:", push_res["updated_refs"])

Virtual Repositories: bytes from anywhere

VirtualRepo reads objects out of packfiles the caller supplies as bytes, so the engine never needs a local clone. Give it a range reader — S3 Range requests, a distributed cache, a mmap, a Pyodide virtual filesystem — and every read path works against it: commits, tree blobs, tree listings, histogram diffs and upload-pack.

Only pack indexes are read up front; object bytes are fetched lazily, one range at a time, and delta chains are resolved without ever downloading the whole pack. With an async def reader even the indexes wait for the first call: VirtualRepo(...) runs on the event loop's own thread, which is blocked until it returns, so nothing could drive the coroutines that fetch them.

import rusty_git


class S3Reader:
    """Synchronous by design: the engine calls this from a worker thread."""

    def __init__(self, bucket, pack_keys):
        self.bucket = bucket
        self.pack_keys = pack_keys

    def list_packs(self) -> list[str]:
        return sorted(self.pack_keys)

    def read_index(self, pack_id: str) -> bytes:
        return self._get(f"{pack_id}.idx")

    def read_range(self, pack_id: str, offset: int, length: int) -> bytes:
        # One HTTP Range request; returning fewer bytes than asked for is fine.
        return self._get(f"{pack_id}.pack", byte_range=(offset, offset + length - 1))

    def _get(self, key, byte_range=None) -> bytes:
        """boto3 ``get_object(Bucket=self.bucket, Key=key, Range=...)``."""


repo = rusty_git.VirtualRepo(S3Reader("my-bucket", ["pack-abc123"]), refs={"HEAD": sha})

info = await repo.get_commit(sha)
source = await repo.get_tree_blob(sha, "src/lib.rs")
hunks = await repo.compare_refs(base_sha, head_sha)
served = await repo.upload_pack(request_body)  # protocol v0

refs is how you express branches: packfiles contain no refs, so either pass a name → object-id map or use SHAs directly (full or abbreviated).

An async def reader works too, so a service can stay on one httpx.AsyncClient/aiobotocore session instead of keeping a thread-safe synchronous pool for the engine:

class S3AsyncReader:
    def __init__(self, client, bucket, pack_keys):
        self.client, self.bucket, self.pack_keys = client, bucket, pack_keys

    async def list_packs(self) -> list[str]:
        return sorted(self.pack_keys)

    async def read_index(self, pack_id: str) -> bytes:
        return await self._get(f"{pack_id}.idx")

    async def read_range(self, pack_id: str, offset: int, length: int) -> bytes:
        return await self._get(f"{pack_id}.pack", Range=f"bytes={offset}-{offset + length - 1}")

    async def _get(self, key: str, **kwargs) -> bytes:
        response = await self.client.get_object(Bucket=self.bucket, Key=key, **kwargs)
        return await response["Body"].read()


async def serve(sha, body):
    async with s3_client() as client:
        # No I/O here: an async reader's indexes are read on the worker that
        # serves the first call, since construction blocks the loop.
        repo = rusty_git.VirtualRepo(
            S3AsyncReader(client, "my-bucket", ["pack-abc123"]),
            refs={"HEAD": sha},
            read_timeout=30,
        )
        return await repo.upload_pack(body)

Each callback is driven on the loop that is awaiting the engine: the worker schedules the coroutine with asyncio.run_coroutine_threadsafe and waits for the result with the GIL released, so the loop keeps running — including the awaits inside the callback. read_timeout bounds one such callback (seconds), so a stuck reader cannot hold the engine indefinitely. Two things are refused rather than hanging: an async reader with no running loop, and the *_sync methods called from the loop's own thread — call those through asyncio.to_thread.

Synchronous twins (get_commit_sync, upload_pack_sync, …) exist for CLI tools and for WebAssembly, where there is no Tokio runtime.

The module-level entry points take the repository as their first argument and accept either form, so a call site never has to know which one it holds:

# `repo` may be a VirtualRepo or a path (str or os.PathLike) — same answers.
info = await rusty_git.get_commit(repo, sha)
blob = await rusty_git.get_tree_blob(repo, sha, "src/lib.rs")
rows = await rusty_git.list_tree(repo, sha, "src", recursive=True)
hunks = await rusty_git.compare_refs(repo, base_sha, head_sha)
served = await rusty_git.upload_pack(repo, request_body)

# ...and on the blocking interface, `rusty_git.sync.get_commit(repo, sha)`.

Operations & Maintenance

Seven operations a long-lived forge needs, all in-process and exposed at the package root and on rusty_git.sync (they live in the ops and gc Rust modules):

import rusty_git

# Do we already have this object? (index lookup, no object is inflated)
if not await rusty_git.has_object("/srv/repos/app.git", sha):
    ...

# Dashboard numbers: refs, branches, tags, loose objects, packs, size
stats = await rusty_git.repository_stats("/srv/repos/app.git", deep=True)
print(stats["branch_count"], stats["object_count"], stats["disk_size_bytes"])

# Integrity check before a user finds the damage
report = await rusty_git.fsck("/srv/repos/app.git")
if not report["ok"]:
    print(report["errors"])  # sorted, each names the object and the ref/edge
print(report["dangling"])  # unreachable objects: reported, not an error

# Fold thousands of loose ref files into one sorted `packed-refs`
packed = await rusty_git.pack_refs("/srv/repos/app.git")  # 0 => already packed

# Fork: copy objects (hardlinked by default) and refs into a new bare repo
await rusty_git.fork_repository("/srv/repos/app.git", "/srv/repos/app-fork.git")

# Compact: fold every pack into one (a crash can only leave debris, never damage)
print(
    await rusty_git.repack("/srv/repos/app.git")
)  # {"packs_before": 12, "packs_after": 1, "objects": 431}

# Full maintenance: repack + pack-refs + de-duplicate loose objects.
# Pruning is opt-in because "unreachable" is not "unwanted": pass a grace period.
stats = await rusty_git.gc("/srv/repos/app.git", prune_expire_days=14)
print(stats["pack_count"], stats["loose_objects"], stats["disk_size_bytes"])

What each operation promises, and what it does not

• has_object answers from the object index (loose names and pack indexes), so True means "present", not "decodes". A sha that is not 40 hex characters raises ValueError — a typo must not look like an absent object.

• repository_stats reads the layout for loose_objects, pack_count and disk_size_bytes (so the numbers match git count-objects/du) and the object store for object_count. object_count is None unless deep=True because it walks the whole store and holds one hash set of ids: counts and sizes are O(files), object_count is O(objects) in time and memory. disk_size_bytes covers the git directory (.git for a non-bare repo), never the worktree, and is None under Pyodide.

• fsck detects two classes of damage: dangling references (a ref, commit parent, commit tree, tag target or tree entry naming an object that is absent) and — with strict=True, the default — undecodable objects. It does not verify pack trailer hashes or recompute object ids, so a valid pack holding an object under the wrong id is invisible here; that is the same trade-off git fsck makes without --full. strict=False is a connectivity check that still reads object headers (the walk cannot descend without them) but never decodes or validates a payload — useful as a cheap pre-push gate. Submodule (gitlink) targets are not required to exist, exactly as in git fsck. Peak memory is two hash sets of object ids, a few dozen bytes per object.

• pack_refs folds the loose ref files under refs/ into one sorted packed-refs (# pack-refs with: peeled fully-peeled sorted, ^-peeled annotated tags, atomic rename) and returns how many refs it moved; 0 means the store was already packed, so calling it twice is a no-op. It goes through gix's packed-ref transaction, which takes packed-refs.lock, renames the new file into place and only then deletes the loose files — an interruption leaves a duplicate ref, never a missing one, and a lock held elsewhere for more than a second is an error rather than a bypass. all=True (default) packs everything; all=False packs only refs/tags/*, matching git pack-refs without --all, which keeps actively-updated branches loose and therefore cheap to compare-and-swap. HEAD and other symbolic refs stay untouched (the format cannot represent them), and no reflog entry is written because no ref value changes. git show-ref output is identical afterwards, and update_ref keeps working on the packed entries — including target=None, which removes the packed-refs line.

• fork_repository only ever reads the source, so it is safe to fork a repository that is serving traffic. Objects are copied before refs: a push landing mid-copy therefore makes the fork stale, never inconsistent — re-fork to catch up. hardlink=True (the default) hardlinks object files, so the fork costs ref metadata instead of a full copy and stays readable even if the source is later pruned or deleted; filesystems that refuse a link (EXDEV, EPERM, FAT/SMB) fall back to a copy per file. Refs, HEAD, packed-refs and shallow are always copied, never linked — a shared ref file would let a push to the fork move a branch in the source. config is deliberately not copied (same reason git clone rewrites it): configure the fork yourself. A destination that exists and is not empty is refused with GitRepositoryError, and so is one that is, contains, or sits inside the source.

• repack folds every object a reference (or HEAD) reaches into one pack, writes it beside the existing packs, fsyncs it, and only then deletes the packs it superseded — so a crash at any point leaves a readable repository, at worst with unreferenced debris that the next run folds in. A pack only becomes usable when its .idx exists, so a concurrent list_commits cannot see a half-written pack, and a reader already walking history keeps its open files alive. packs_after == 1 means the compaction worked; objects is the size of the new pack, which equals git rev-list --objects --all. Packs carrying a .keep file are left alone, and unreachable objects are neither copied nor deleted. Requires git2's pack builder, so it is native-only (no Pyodide).

• gc runs repack, folds loose refs into packed-refs, and deletes the loose objects the new pack made redundant — losing the redundant file is what actually shrinks the repository. Pruning is opt-in: prune_expire_days defaults to None ("never delete"), because "no ref reaches it" does not mean "nothing needs it" — a push in flight and anything a reflog could recover are both unreachable and both young, so anything younger than the period survives whatever you pass. prune_expire_days=0 is for a caller who has just read the dangling list from fsck. Packs are never deleted beyond the ones the repack superseded. Run one gc per repository at a time: it does not take git gc's gc.pid lock, so a scheduled task should own the repository, not a request handler.

• Push quarantine — receive_pack indexes the inbound pack into a private object database (objects/incoming-<unique>/, a throwaway bare repository whose objects/info/alternates names the real one, so thin-pack fix-up still resolves) and only migrates those objects into the real ODB once a ref is actually going to move. A refused push — non-fast-forward, stale lease, malformed pack — therefore leaves the repository byte-for-byte unchanged, which is what stops a client that can push from growing it without bound. The quarantine is deleted on every path, including a panic. Pass receive_pack(..., quarantine=False) for the previous behaviour (ingest straight into the real ODB); it is kept for one release as a fallback.

Not implemented, by design

• Server-side clone from a remote URL — there is no network transport in this build: gix's clone/fetch path is gated behind blocking-network-client/async-network-client, and this crate enables neither (compiling PrepareFetch::fetch_only against the shipped feature set fails with error[E0599]). Enabling one means adding a TLS stack and credentials handling to a statically-linked wheel — a packaging decision, not an API gap. Import out of band (git clone --mirror in the pipeline, or a one-shot fetch in the application), then use fork_repository for in-process copies.
• Reflog expiry, commit-graph and multi-pack-index writing — gc prunes by ref-reachability plus a grace period, deliberately not by reflog reachability: it never expires a reflog, so anything a reflog could recover is protected by age alone. Treat the grace period as the safety margin that git gc gets from its reflog expiry.

Development & Testing

Rust Tests

cargo test

Python Integration Tests

uv run pytest

License

Apache-2.0 © Framework M Contributors