🌲 forest

git-like data management for arbitrary data trees.

Forest is the data-side parallel to git's version control. Git tracks code in .git/; forest tracks large data — local layout plus remote sync — in .forest/. It borrows git's mental model and verbs (checkout, status, push, pull, remote, a HEAD-style pointer) so your git intuition carries over, but the two domains never overlap and neither requires the other.

Forest is domain-agnostic and self-contained: it manages any data trees, knows nothing about what the data means, and depends on no other project. It moves files and tracks their sync state; it does not validate or interpret their contents.

Documentation

Full docs live at tmsincomb.github.io/forest. Start with the hands-on tutorial — seven short chapters from forest init to a fully synced repo, with every output block captured from a real session and no cloud credentials required.

Model

Fixed-depth, no arbitrary nesting:

workspace → checkout → stage → unit → files

• Workspace — a per-repo .forest/ control area (registry + active pointer).
• Checkout — a named data view/focus registered in the workspace (like a git branch you stay rooted in). Switching is an O(1) pointer rewrite; data never moves.
• Stage — a named data category inside a checkout, with a remote layout.
• Unit — one addressable item within a stage (a subdirectory, a directory, or a file, per the stage's sync_by).

Install

pip install forest-cli

Requires rclone on $PATH for transfers. For a development install, clone the repo and pip install -e ".[dev]".

Quick start

No config files are hand-written. Onboarding is a few commands:

forest init                      # create the nameless .forest/ workspace container
forest checkout demo             # create if absent, register + activate 'demo'
forest remote add origin s3://my-bucket/prefix   # allowed before any local data exists
forest add raw ./data/raw        # register stage 'raw' and bind it to a local path
forest push                      # sync every bound stage to the active remote

• forest init creates only .forest/config.yaml (version: 1, checkouts: {}) and a managed .gitignore. No root config, no checkout, no active pointer.
• forest checkout <name> switches to the checkout, creating, registering, and activating it first (with .forest/checkouts/<name>/forest.yaml) when the name is not registered. forest checkout create <name> is the explicit form.
• Remotes can be added before any local binding — useful when your data is remote-only at first.
• forest add STAGE PATH registers a new stage and binds it to a local path in one step (use forest bind to rebind an existing stage).

Metadata layout

Everything forest owns lives under .forest/; your data does not.

.forest/
  config.yaml                     # workspace registry: version, checkouts{}
  HEAD                            # active checkout name (gitignored)
  checkouts/
    demo/
      forest.yaml                 # shared: stages, remotes, manifest
      local.yaml                  # user-local: active_remote, stage_paths (gitignored)
      sync_state.json             # user-local push/pull state (gitignored)

Shared metadata (config.yaml, each forest.yaml) is committed so a fresh clone bootstraps with bind + remote use + pull. User-local files (HEAD, local.yaml, sync_state.json) are gitignored.

Commands

Run any command with -C <path> to operate on another repo without cd.

Forest syncs all files in a data unit, skipping OS junk (.DS_Store, AppleDouble ._*, *.tmp). It applies no content-based include/exclude rules. Push and pull skip a clean unit when source hashes prove its content and path are unchanged; a backend that cannot provide a usable hash transfers safely.

Terminal output

On an interactive terminal forest renders rich output — colored status tables, file trees, panels, and live progress bars during push/pull (rclone JSON stats streamed to a bar; needs rclone ≥ 1.49, older versions transfer fine without a bar). When stdout is piped or redirected, output falls back to the plain, machine-parseable format, so scripts never see ANSI codes or box drawing.

Overrides: --color/--no-color per invocation, FOREST_OUTPUT=rich|plain|auto in the environment, and the NO_COLOR/FORCE_COLOR conventions. Precedence: flag > FOREST_OUTPUT > NO_COLOR > FORCE_COLOR > TTY detection. --verbose shows rclone's raw transfer log instead of a progress bar; --quiet suppresses everything but errors.

Shell prompt

Show the active checkout in your prompt (bash and zsh), the way git prompts show the current branch:

# ~/.bashrc
eval "$(forest prompt init bash)"

# ~/.zshrc
eval "$(forest prompt init zsh)"

~/lab/G004 $                # outside a workspace: unchanged
🌲 G004 ~/lab/G004 $        # inside a workspace with an active checkout

The snippet is pure shell — it reads .forest/HEAD directly and never invokes the forest CLI while rendering, so your prompt stays fast. Set FOREST_PROMPT_ICON to replace the 🌲.

Config reference

Checkout forest.yaml (shared, committed):

project: demo
remotes:
  origin:
    url: s3://my-bucket/prefix
    region: us-east-2          # optional; also endpoint, profile, key_file, known_hosts
stages:
  raw:
    remote_path: demo/raw      # optional; defaults to <checkout>/<stage>
    sync_by: subdirectory      # subdirectory | directory | file
    direction: both            # push | pull | both; bare push/pull skip
                               # wrong-direction stages, explicit selection errors

Checkout local.yaml (per-machine, gitignored):

active_remote: origin
stage_paths:
  raw: ../data/raw             # relative resolves from the workspace root

Environment variables

All optional, all off by default — forest is silent and sends nothing anywhere unless configured. Copy .env.example for a commented template. Logs are always secret-scrubbed.

Dogfood: this repo runs forest

This repository manages its own examples/ and tests/data/ trees with forest — a live demonstration that .forest/ and .git/ coexist without overlapping. It was set up with exactly the quick-start commands:

forest init
forest checkout demo
forest remote add origin s3://forest-cli-test --region us-east-2
forest add examples ./examples                        # two-way stage (direction: both)
forest add data ./tests/data --sync-by directory      # two small test files
forest push                                           # seeds both stages

After the seeding push, data was flipped to direction: pull in the checkout's forest.yaml — it is a read-only reference mirror now: a bare forest push skips it with a note, and forest push --stage data refuses with exit 1 (a new stage can start there directly with forest add ... --direction pull). The examples stage keeps syncing both ways.

Inspect the result:

git ls-files .forest        # what a clone gets: config.yaml + checkouts/demo/forest.yaml
cat .gitignore              # forest-managed: HEAD, local.yaml, sync_state.json stay local
forest ls                   # stage overview: data shows (directory, pull)
forest status               # sync state of both stages

A fresh clone bootstraps the local half (HEAD, bindings, sync state are gitignored) with:

forest checkout demo               # select the checkout; clones have no HEAD
forest bind examples ./examples
forest bind data ./tests/data
forest pull --force                # first pull: no local sync state yet, so
                                   # forest refuses to touch the git-tracked
                                   # copies without it

The single configured remote is used automatically. Pulling needs AWS credentials for the bucket; the layout is the demonstration.

Notes

• Single active machine (v1). HEAD/local.yaml/sync_state.json are git-invisible but may be synced by a file-syncing tool; forest assumes one active machine and uses atomic writes plus a per-checkout flock for intra-machine write races.
• Real filenames. Forest stores data under real paths, not a content-addressed blob store.
• See docs/adr/ for the design decisions behind the workspace/checkout model.