🌲 forest

git-like data management for arbitrary data trees.

Forest is the data-side parallel to git's version control. Git tracks code in .git/; forest tracks large data — local layout plus remote sync — in .forest/. It borrows git's mental model and verbs (checkout, status, push, pull, remote, a HEAD-style pointer) so your git intuition carries over, but the two domains never overlap and neither requires the other.

Forest is domain-agnostic and self-contained: it manages any data trees, knows nothing about what the data means, and depends on no other project. It moves files and tracks their sync state; it does not validate or interpret their contents.

Documentation

Full docs live at tmsincomb.github.io/forest. Start with the hands-on tutorial: one runnable path from forest init to push, status and pull, using a local directory as the remote, so no cloud credentials are needed. Every command and option is listed in the CLI reference.

Model

Fixed-depth, no arbitrary nesting:

workspace → checkout → stage → unit → files

• Workspace — a per-repo .forest/ control area (registry + active pointer).
• Checkout — a named data view/focus registered in the workspace (like a git branch you stay rooted in). Switching is an O(1) pointer rewrite; data never moves.
• Stage — a named data category inside a checkout, with a remote layout. Command selectors call it BRANCH.
• Unit — one addressable item within a stage (a subdirectory, a directory, or a file, per the stage's sync_by). In the default subdirectory mode, loose files at the stage root are not units; push, pull and status skip them.

Install

pip install forest-cli

Requires rclone on $PATH for transfers. For a development install, clone the repo and pip install -e ".[dev]".

Install the current stable rclone release (CI pins 1.75.1, verified on 2026-09-25). Native copy reports first appeared in 1.71; versions lacking those flags fail with an upgrade message, never an offline synced verdict.

Quick start

No config files are hand-written. Onboarding is a few commands:

forest init                      # create the nameless .forest/ workspace container
forest remote add origin s3://my-bucket/prefix
forest remote use origin          # select the source for new checkouts
forest checkout demo             # create + activate, remembering origin
forest add raw ./data/raw        # register stage 'raw' and bind it to a local path
forest push                      # sync every bound stage to the checkout's saved remote

• forest init creates only .forest/config.yaml (version: 1, checkouts: {}) and a managed .gitignore. No root config, no checkout, no active pointer.
• forest checkout <name> switches to the checkout, creating, registering, and activating it first (with .forest/checkouts/<name>/forest.yaml) when the name is not registered. forest checkout create <name> is the explicit form.
• Remotes belong to the workspace and can be added before the first checkout. remote use changes only the choice for new checkouts. Existing trees keep their saved remote, including when adding stages. With one workspace remote, a new checkout remembers it automatically in shared metadata.
• forest add STAGE PATH registers a new stage and binds it to a local path in one step (use forest bind to rebind an existing stage).

Metadata layout

Everything forest owns lives under .forest/; your data does not.

.forest/
  config.yaml                     # shared: version, checkouts{}, remotes{}
  local.yaml                      # user-local: remote choice for new checkouts (gitignored)
  HEAD                            # active checkout name (gitignored)
  checkouts/
    demo/
      forest.yaml                 # shared: stages, manifest; legacy remotes if present
      local.yaml                  # user-local: active_remote, stage_paths (gitignored)
      sync_state.json             # user-local push/pull state (gitignored)

Shared metadata (config.yaml, each forest.yaml) is committed so a fresh clone of a bound tree bootstraps with bind + pull. User-local files (HEAD, both local.yaml files, sync_state.json, lock files) are gitignored.

Commands

Run any command with -C <path> to operate on another repo without cd.

Forest syncs all files in a data unit, skipping OS junk (.DS_Store, AppleDouble ._*, *.tmp) and partial *.forest-tmp files. It applies no content-based include/exclude rules. Push and pull use native rclone comparison against real destinations. Changed and missing files transfer; unchanged files are skipped. Counts show files and bytes actually transferred, not the size of the selected unit. Layout-preserving transfers batch selected files in one native copy per root mapping, even when root names differ; true filename renames use copyto. Pull uses rclone's temporary-file rename, with .forest-tmp names and no --inplace.

Select work with positional names: forest push outputs for every unit in that branch, forest pull outputs/run1 outputs/run2 for units of one branch. --stage, --all and add --sync-by are removed without aliases; set sync_by in the checkout's forest.yaml instead. Unknown branches, invalid paths and missing push targets fail before any selected target transfers.

Forest reuses saved local hashes when the resolved path, device/inode, size, mtime and ctime are unchanged. Changed evidence causes a fresh read, including edits that restore the old mtime. This relies on filesystem stat evidence; status --checksum performs an explicit fresh content check. Legacy state warms the cache on its next successful operation. Remote hashes are read live, not cached. Verification still precedes recording, and missing backend hashes limit verification to sizes. --force uses native checksum comparison when a transfer is needed, including edits whose size and mtime were preserved.

Live status

forest status compares bound stages with the current tree's saved remote using rclone copy's dry-run report. It downloads and uploads nothing, changes no file metadata, and never saves sync history. It uses the real paths, not temporary transfer destinations.

Each row names a file: synced, different, local-only, or remote-only. Saved history can further identify local-changed, local-missing, and remote-missing. With --checksum, an unchanged local hash can also establish remote-changed. different does not guess which side changed.

Normal comparison uses rclone's size/time checks and may hash files with changed times. It does not hash the whole tree routinely. --checksum requests available backend hashes and checks local contents against saved hashes; it can be slow. Synced means a match under that comparison, not guaranteed byte equality on backends without suitable hashes. A failed comparison exits with an error and prints no synced rows.

Use status BRANCH/UNIT or status --id ID to select units. --incomplete hides synced rows, for this tree and saved remote only. New remote files appear within configured stage prefixes, even when the bound local directory is missing. A prefix sequencing/run0001 does not discover its sibling run0002. Bare status skips unbound stages with a warning; bind them before comparing. Read-only status inspects both push-only and pull-only stages.

The old ls and list commands are removed. Status always includes file paths, so their --tree and --long views are removed too. Use checkout list for checkout names; status does not scan other trees. Transfer behavior is unchanged.

Terminal output

On an interactive terminal forest renders rich output — colored status tables, panels, and live progress bars during push/pull (rclone JSON stats streamed to a bar; needs rclone ≥ 1.49, older versions transfer fine without a bar). When stdout is piped or redirected, output falls back to the plain, machine-parseable format, so scripts never see ANSI codes or box drawing.

Overrides: --color/--no-color per invocation, FOREST_OUTPUT=rich|plain|auto in the environment, and the NO_COLOR/FORCE_COLOR conventions. Precedence: flag > FOREST_OUTPUT > NO_COLOR > FORCE_COLOR > TTY detection. --verbose shows rclone's raw transfer log instead of a progress bar; --quiet suppresses everything but errors.

Shell prompt

Show the active checkout in your prompt (bash and zsh), the way git prompts show the current branch:

# ~/.bashrc
eval "$(forest prompt init bash)"

# ~/.zshrc
eval "$(forest prompt init zsh)"

~/lab/G004 $                # outside a workspace: unchanged
🌲 G004 ~/lab/G004 $        # inside a workspace with an active checkout

The snippet is pure shell — it reads .forest/HEAD directly and never invokes the forest CLI while rendering, so your prompt stays fast. Set FOREST_PROMPT_ICON to replace the 🌲.

Config reference

Workspace .forest/config.yaml (shared, committed):

version: 1
checkouts:
  demo:
    remote: origin            # saved source for this tree, not a workspace default
remotes:
  origin:
    url: s3://my-bucket/prefix
    region: us-east-2          # optional; also endpoint, profile, key_file, known_hosts

Checkout forest.yaml (shared, committed):

project: demo
stages:
  raw:
    remote_path: demo/raw      # optional; defaults to <checkout>/<stage>
    sync_by: subdirectory      # subdirectory | directory | file
    direction: both            # push | pull | both; bare push/pull skip
                               # wrong-direction stages, explicit selection errors

Checkout local.yaml (per-machine, gitignored):

active_remote: origin
stage_paths:
  raw: ../data/raw             # relative resolves from the workspace root

remote use NAME writes the workspace's local choice for future checkouts. It never changes an existing tree's source. A new tree saves its remote name under checkouts in workspace config, so cloning the shared metadata preserves that source. Stage mappings belong to the tree and use its saved remote.

Legacy trees keep a recorded checkout-local active_remote, or their sole checkout-local remote when no explicit choice was recorded. Forest never guesses from the latest workspace choice. Legacy definitions stay in their original files and keep their addresses. remote list/show still display the selected tree's effective sources; without a selected tree they show workspace sources. remote use selects only workspace definitions for future trees.

A tree with no saved source cannot add stages or transfer data. Restore its verified source metadata, or select a workspace remote and create a new tree. Missing definitions and conflicting saved references also fail clearly. There is no rebinding command in this change.

remote remove refuses workspace sources referenced by any tree, even when its local config is missing. Overlapping legacy names must be resolved explicitly. Removing an unused definition never removes data or sync history.

Environment variables

All optional, all off by default — forest is silent and sends nothing anywhere unless configured. Copy .env.example for a commented template. Logs are always secret-scrubbed.

Notes

• Single active machine (v1). HEAD/local.yaml/sync_state.json are git-invisible but may be synced by a file-syncing tool; forest assumes one active machine and uses atomic writes plus a per-checkout flock for intra-machine write races.
• Real filenames. Forest stores data under real paths, not a content-addressed blob store.