agent-folder-workspace

agent-folder-workspace presents local directories as stable, paginated node hierarchies for agents. It discovers file metadata first and opens document bodies only when a caller expands or reads a node. The default parser path is local Python code: Microsoft Office and LibreOffice are not required.

Project status: 0.1.6 is an alpha contract. Read-only exploration and byte access are implemented; editing, rendering, OCR, and complete semantic coverage of every supported container are not.

Python 3.11 through 3.14 is supported. This repository does not assume that a release has already been published to a package index. Install a checkout with:

python -m pip install .
folderws --help

Runtime installation uses only Python packages from pip; the Windows-only pywin32 dependency is selected by an environment marker. Java, .NET, Microsoft Office, and LibreOffice are never installed or required by this package. Already-installed office suites are optional probe/conversion backends.

Quick start

from pathlib import Path

from agent_folder_workspace import FolderWorkspace

with FolderWorkspace.open(Path("./documents")) as workspace:
    if not workspace.wait_until_ready(timeout=10):
        raise TimeoutError("metadata scan is still running")

    info = workspace.workspace_info()
    first_page = workspace.list_children(info.root_node.id, limit=50)
    for node in first_page.nodes:
        print(node.id, node.kind, node.relative_path, node.content_status)

Call list_children(file_node.id) to expose a format-specific hierarchy, then use read_content() on a semantic node or read_binary() for a bounded byte range. Results are Pydantic models and can be serialized with result.model_dump(mode="json").

Architecture

The package separates transport, workspace policy, node discovery, format adapters, and caching:

1. FolderWorkspace validates and owns one root directory and one generation.
2. NodeRegistry builds an in-memory metadata tree. The initial scan uses directory entries and file stat data; it does not read file payloads.
3. AdapterManager keeps text, CSV, and bounded binary access direct. It sends semantic children/read calls for structured text, SQLite, OOXML, ODF, PDF, and legacy Office to a bounded pool of spawn-based parser workers.
4. Parser workers block Python socket/DNS entry points before importing risky adapters and enforce one deadline across startup, queueing, IPC, and parsing.
5. SizedLru retains bounded decoded pages in memory. ContentIndex stores an optional SQLite search index in each workspace's owned local cache.
6. WorkspaceManager gives one FastMCP process a bounded registry of active and explicitly addressed workspace roots.
7. The Python API, Typer CLI, and FastMCP stdio server share the same workspace methods and versioned response contracts.

Node IDs are deterministic hashes of relative and logical paths; they do not contain the absolute workspace path. Pagination cursors are opaque, integrity-protected, workspace-instance tokens. refresh_workspace() advances the generation, rebuilds metadata, clears decoded pages, synchronizes an open index, and makes older cursors stale.

See the architecture notes for lifecycle, data-flow, cache, and error details.

Python API

The supported top-level entry points are:

API	Purpose
FolderWorkspace.open(root, config=None)	Open one existing local directory.
workspace_info()	Report the root node, generation, scan state, and index coverage counts.
list_children(node_id, cursor=None, limit=100)	Page through filesystem or virtual children.
get_node(node_id)	Fetch a node by stable ID.
read_content(node_id, cursor=None, limit=100, representation="semantic")	Read a semantic/source-record page; raw delegates to bounded binary access.
read_binary(node_id, offset=0, length=..., cursor=None)	Return at most 1 MiB as base64 and an optional continuation cursor.
search_content(query, cursor=None, limit=20)	Search filenames plus progressively indexed content.
index_content()	Walk supported semantic leaves into the persistent content index.
refresh_workspace()	Re-scan and invalidate old generation-bound cursors.
close()	Stop the scan worker/parser pool and close an open index.

FolderWorkspace is a context manager. Known file nodes and shallow requested directories remain usable while the deep metadata scan continues; a lookup for an as-yet undiscovered node can wait for that scan. Full indexing and exact coverage require the inventory to finish.

WorkspaceConfig controls page, binary, cache, embedded-file, parser-worker, archive, XML, and structural limits. It is frozen and rejects unknown fields. The package exports a stable error hierarchy (FolderWorkspaceError, InputValidationError, NodeNotFoundError, StaleCursorError, and ResourceLimitError) and six document wire models plus three workspace-selection models:

• CapabilityReportV1
• ContentPageV1
• DiagnosticV1
• NodePageV1
• NodeV1
• SearchPageV1
• WorkspaceCloseV1
• WorkspaceHandleV1
• WorkspaceListV1

Committed JSON Schemas live in src/agent_folder_workspace/schemas/. Regenerate them with python scripts/generate_json_schemas.py; CI checks that they still equal Pydantic's model_json_schema() output.

MCP server

Run one stdio server with an initial root:

folderws mcp --root ./documents

The server exposes exactly these tools:

• open_workspace
• set_cache_directory
• import_cache
• list_workspaces
• activate_workspace
• close_workspace
• workspace_info
• list_children
• get_node
• read_content
• read_binary
• search_content
• index_content
• refresh_workspace
• probe_backends

open_workspace(path) accepts an explicit absolute directory, returns an opaque, process-lifetime workspace_id, and makes that workspace active. The compatible extended form open_workspace(path, cache_directory=None) accepts an optional absolute cache directory for that document basis. Without it, MCP checks for and reuses an existing owned cache at path/.agent-folder-workspace-cache; a foreign directory with that name is rejected. The selected path is returned as workspace_info.cache_directory. set_cache_directory(cache_directory, workspace_id=None) is the separately discoverable MCP operation for changing the active or explicitly selected open workspace. The path must be absolute. If it already exists, it must be an owned cache with a regular .agent-folder-workspace-cache-v1 marker. A successful change atomically returns a replacement workspace_id; retain it because the previous ID and its cursors become invalid. Selecting the current cache is an idempotent no-op.

import_cache(source_cache_directory=None, workspace_id=None) imports the matching document basis from the historical platformdirs cache used through version 0.1.2. An explicit source must be an absolute, ownership-marked legacy cache root. The import rejects links, reparse points, special files, and oversized payloads; it copies only the selected workspace subtree and never overwrites existing target cache data.

Existing tools accept an optional workspace_id; omission uses the active workspace. Opening the same canonical root again reuses its ID; attempting to reopen it with a different cache returns an input error. list_workspaces, activate_workspace, and close_workspace manage the registry. One process holds at most eight workspaces and never evicts one silently.

On Windows, runtime selection accepts C:\Documents, C:/Documents, UNC paths, and Git Bash paths such as /c/Documents. Relative external paths are rejected. The initial CLI root remains compatible with relative paths such as ..

Results use the same versioned models as the Python API. A method that is absent from an injected workspace implementation returns a structured capability_unavailable response instead of failing server construction. Expected workspace/parser failures and unexpected tool exceptions become structured results; a later MCP request remains available. PDF decoding remains lazy and isolated in bounded parser workers.

The transport is stdio. There is no HTTP listener, authentication layer, or network broker. The process and its MCP client share the operating-system permissions of the account that launched folderws; open roots narrowly.

CLI

All commands accept global options before the subcommand: folderws [--quiet | --verbose] [--log-file PATH] COMMAND .... Human status, progress, warnings, and errors are written to stderr; the JSON reports from doctor, index, clear-cache, and install-opencode remain unadorned on stdout. MCP keeps stdout exclusively for JSON-RPC. By default, durable UTF-8 logs are written to platformdirs' user log directory as folderws.log, with 5 MiB rotation and three backups. An explicit log file must not be a symbolic link, junction, or reparse-point destination; ordinary log permission failures fall back to stderr-only operation.

# Serve a folder to an MCP client over stdio
folderws mcp --root DIR

# Detect optional backends without launching them (default)
folderws doctor

# Opt in to isolated Office/LibreOffice round-trip probes
folderws doctor --probe-office --json --timeout-seconds 10

# Build the complete semantic content index
folderws index --root DIR

# Remove the current document directory's owned cache, or an explicit cache
folderws clear-cache [--cache-dir PATH]

# Materialize the bundled, managed OpenCode skill and plugin
folderws install-opencode --target DIR

Legacy positional roots/targets and doctor --active remain compatibility aliases; the explicit forms above are the primary interface.

install-opencode does not edit opencode.json or opencode.jsonc. It writes two ownership-marked resources atomically and always refuses foreign files, symbolic links, and Windows reparse points. The compatibility --force flag never bypasses ownership checks.

Without --cache-dir, clear-cache selects ./.agent-folder-workspace-cache. It removes only a non-link directory containing the package's cache ownership marker; an arbitrary directory supplied through --cache-dir is refused.

Lazy loading and cache

Lazy means that the initial recursive scan records names, types, sizes, and timestamps, but not document bodies. Expanding a file asks its adapter for virtual children. Reading a semantic leaf parses only the requested format path where the adapter permits it; some container adapters must still parse a whole XML part, table, or package directory to answer that request.

Document processing and storage

Documents are normally split into virtual nodes, not into physical per-page or per-section files. The following flow shows which data stays in memory and which derived data can reach the owned cache:

flowchart LR
    A["Document basis<br/>original files unchanged"] --> B["Metadata scan<br/>name, type, size, timestamps"]
    B --> C["In-memory NodeRegistry"]
    C --> D{"Expand or read a file"}
    D --> E["Direct adapters<br/>text, CSV, bounded binary"]
    D --> F["Spawn worker pool<br/>structured text, SQLite,<br/>OOXML, ODF, PDF, legacy Office"]
    E --> G["Virtual semantic nodes<br/>lines, rows, paragraphs,<br/>tables, sheets, slides, pages"]
    F --> G
    G --> H["RAM page LRU<br/>256 MiB default"]
    G --> I["Optional SQLite/FTS5 index"]
    G --> J["Physical cache artifacts only when needed<br/>embedded files and legacy conversions"]

Text files are grouped into virtual blocks of 1,000 lines and CSV files into virtual blocks of 1,000 records. Other adapters expose format-specific virtual hierarchies such as PDF pages and text blocks, Word paragraphs and tables, spreadsheet sheets and cells, or presentation slides and shapes. These nodes retain source references and are paginated through the normal API. They are not written as individual source documents.

Cache versions and directory structure

All releases use the same on-disk format marker, .agent-folder-workspace-cache-v1. Package releases changed the default location and cache controls, not the cache format:

Package versions	Default cache root	Change
0.1.0–0.1.2	OS user cache returned by platformdirs	All document bases share one root and are isolated by a workspace-path hash.
0.1.3–0.1.4	WORKSPACE_ROOT/.agent-folder-workspace-cache	The owned cache moves beside the document basis and is excluded from discovery.
0.1.5	WORKSPACE_ROOT/.agent-folder-workspace-cache	MCP can change the cache of an open workspace and safely import its matching subtree from an old cache.
0.1.6	WORKSPACE_ROOT/.agent-folder-workspace-cache	Documentation release; cache format and behavior remain unchanged from 0.1.5.

The current default layout is:

WORKSPACE_ROOT/
├── <original documents>                         # authoritative input
└── .agent-folder-workspace-cache/               # owned, excluded from scans
    ├── .agent-folder-workspace-cache-v1         # ownership/format marker
    └── workspaces/
        ├── import-session-<uuid>/                # temporary during cache import
        └── <workspace-path-sha256-prefix>/
            ├── index.sqlite3                    # persistent content index
            ├── index.sqlite3-{wal,shm,journal}  # transient SQLite sidecars
            ├── embedded/
            │   ├── <sha256><original-extension> # materialized attachment/embedded file
            │   └── .<sha256>.<uuid>.partial     # temporary atomic write
            └── alternatives/
                └── <conversion-key>/
                    ├── converted.{docx,xlsx,pptx}
                    ├── converted-<uuid>.<ext>    # temporary conversion output
                    └── conversion-session-*/    # temporary converter scratch

Directories and files are created lazily. For example, index.sqlite3 appears only after content indexing starts, embedded/ only after a supported embedded document or PDF attachment is expanded, and alternatives/ only after an optional legacy Office conversion is requested. Cache import copies only the selected workspace hash, stages it under workspaces/import-session-<uuid>, then installs it atomically without deleting the source or overwriting existing target data.

Additional package-owned files live outside the document-local cache:

<platform user log directory>/
├── folderws.log
└── folderws.log.{1,2,3}                         # rotating 5 MiB backups

<platform user cache directory>/
└── agent-folder-workspace/
    └── probe-scratch/
        └── session-<backend>-<pid>-<uuid>/      # active Office/LibreOffice probe

<OpenCode config directory>/                     # only after install-opencode
├── skills/agent-folder-workspace/SKILL.md
└── plugins/agent-folder-workspace.ts

Text and CSV semantics, metadata scanning, and bounded memory-mapped original-file byte reads run in the workspace process. Structured text, SQLite, OOXML, ODF, PDF, and legacy Office semantic children/read calls, plus byte extraction from their container parts/streams/records, use a lazily started pool of parser_max_workers long-lived spawn processes (two by default). A parser_timeout_seconds deadline covers pool startup, queue wait, request and response IPC, and execution. A timed-out worker is terminated and replaced; the caller receives a stable parser_timeout diagnostic.

Supported embedded package parts and PDF attachments are mounted only when their node is expanded. The bytes are copied atomically, in bounded chunks, to a hashed per-workspace embedded cache file, then exposed below a contents node through the normal adapter/worker path. Depth, count, individual file size, and total disk-cache limits apply. This feature creates derived document copies on disk; folderws clear-cache removes them with the index.

Decoded ContentPageV1 values use a byte-sized in-memory LRU capped by max_content_cache_bytes (256 MiB by default). Reads are also indexed on a best-effort basis. index_content() performs a full supported semantic walk without retaining all decoded bodies in RAM.

The persistent index is an application-owned SQLite database below WORKSPACE_ROOT/.agent-folder-workspace-cache, separated by a hash of the workspace root. Existing owned caches are reused, while the cache subtree is excluded from document discovery. WorkspaceConfig.cache_directory and MCP's optional cache_directory argument can select another cache root explicitly. The dedicated MCP tools can change an already-open workspace's cache or import its matching subtree from the historical shared cache without overwriting current data. The index uses FTS5 when available and a plain-table fallback otherwise. max_disk_cache_bytes is checked after read-through chunks and completed files; least-recently-used indexed files are evicted and SQLite WAL/database pages are compacted. Embedded and alternative views enforce the same cache budget at their own materialization boundaries. Read-through index failures do not make an otherwise successful content read fail. Embedded mounting, by contrast, requires a writable bounded cache. Use folderws clear-cache to remove an explicitly selected cache tree safely.

Normal completion removes scratch immediately. On a later workspace/cache open, package-owned partial files and conversion/probe sessions older than five minutes are removed without traversing linked paths; recent items are retained to avoid interfering with another process.

Search coverage is the fraction of discovered files whose full indexing pass completed. Filename matches are available before content coverage reaches 1. If any directory could not be inventoried, the true denominator is unknown; coverage is then conservatively 0 and complete=false, even when every successfully discovered file has been indexed. The affected directory node is partial and carries a directory_scan_failed diagnostic.

Optional legacy conversion views

WorkspaceConfig.backend_policy accepts four closed values:

• python_first (default) uses the built-in DOC/XLS/PPT parser and attempts an actively usable Microsoft Office backend, then LibreOffice, only if that parser fails;
• python_only never starts an office suite;
• microsoft_office and libreoffice keep the Python view and additionally create a converted, provenance-marked OOXML child when a legacy file is first expanded.

Optional conversion is performed by a timeout-bounded child Python process. The result is verified as OOXML and mounted through the normal OOXML adapter. Its cache key includes the backend, adapter version, and SHA-256 of the source, and both conversion scratch and the retained result stay below the selected workspace cache rather than %TEMP% or /tmp. Microsoft automation disables macros, UI, events, external-link updates, and recent-file writes where the application API exposes those controls. LibreOffice uses a disposable cache-local profile, headless/safe-mode flags, and never touches the user's normal profile. The external applications are optional and are used only after an active round-trip capability check.

Security model

The normal Python parser path is offline and read-only. It does not upload documents and does not execute document macros. Important controls include:

• roots that are symbolic links or reparse points are rejected;
• external roots are opened only by an explicit open_workspace call, are bounded to eight per process, and receive process-lifetime opaque IDs;
• workspace caches default to an owned .agent-folder-workspace-cache below the selected root, are excluded from source discovery, and can be overridden only with an explicit absolute MCP path;
• child links are not followed by default, and resolved sources must remain under the selected root;
• SQLite files are opened with mode=ro and PRAGMA query_only=ON;
• YAML uses safe_load; XML uses defused parsers; HTML parsing disables network access;
• OOXML and ODF apply entry-count, expanded-size, XML-size, and compression-ratio checks; ODF rejects encrypted package entries;
• PDFs reject encryption and enforce configured content/attachment limits;
• binary reads, page sizes, and search inputs are bounded;
• risky semantic adapters run in bounded spawn workers; Python socket/DNS APIs and proxy variables are blocked before those adapters import, and hard call deadlines retire a hung worker;
• supported embedded files are materialized atomically only inside a checked, per-workspace cache path with depth/count/file/disk limits;
• pagination cursors are signed and generation-bound.

These controls reduce accidental traversal and common parser abuse; they are not a sandbox. A caller with read_binary access can intentionally obtain bytes from any regular file below the chosen root. Spawn workers inherit the account's filesystem permissions and are not an OS sandbox; their network block covers Python socket paths, not arbitrary native code. Text/CSV parsing, original-file reads, and embedded-byte materialization remain in the workspace process; container-part range extraction runs in parser workers. Size and timeout limits do not prove that every malformed file is harmless. Run the server under a minimally privileged account for untrusted collections.

folderws doctor --probe-office --json is different from default parsing: it explicitly launches isolated Microsoft Office COM applications or a headless LibreOffice process, copies bundled minimal legacy DOC/XLS/PPT fixtures into application-cache scratch, and verifies real OOXML round trips with timeouts and no-UI/no-macro/no-link settings. It never uses documents from the workspace, but it can still be affected by local application policy.

See SECURITY.md before exposing sensitive folders or reporting a vulnerability.

Supported formats

Family	Extensions	Current semantic surface
Plain/delimited text	.txt, .md, .csv, .tsv	Line or row blocks; header-derived CSV records.
Structured text	.json, .jsonl, .yaml, .yml, .xml, .html, .htm	Hierarchical values/elements, attributes, and text.
SQLite	.sqlite, .sqlite3, .db	Schema objects and paginated table/view rows, read-only.
OOXML	.docx, .docm, .xlsx, .xlsm, .pptx, .pptm	Core document/sheet/slide semantics plus raw package parts.
OpenDocument	.odt, .ods, .odp	Core text/table/sheet/slide semantics plus package parts.
PDF	.pdf	Metadata, pages, text blocks, extracted tables, image descriptors, attachments, structural objects, and original bytes.
Legacy Office	.doc, .xls, .ppt	Partial semantic extraction plus CFB streams and lossless known/unknown record bytes.
Other files	any other extension	Metadata and bounded original-byte reads only.

The detailed coverage and per-format caveats are in docs/formats.md. Windows-specific backend behavior is in docs/windows.md.

Limitations

• This is an explorer, not an editor, renderer, office-suite replacement, or general-purpose file-conversion service; optional legacy conversion exists only to mount an alternative semantic view.
• Semantic extraction is deliberately partial. Raw package/stream access makes omitted structures inspectable but does not turn them into high-level data.
• Formulas are reported where supported but never calculated. Cached values can be absent or stale.
• Scanned PDFs and images have no OCR path. Visual layout, charts, animations, tracked changes, styles, signatures, and embedded-object semantics are not comprehensively reconstructed.
• Encrypted/password-protected documents are unsupported. Macros and external links are exposed only as inert package data where available and are never executed or refreshed.
• Recognized embedded files are mounted recursively only from supported package/attachment nodes and only within configured depth/count/byte limits. Unsupported extensions remain raw; mounted bytes persist in the application cache until eviction or explicit clearing.
• Legacy .doc, .xls, and .ppt parsing recognizes a useful subset of binary records; valid files can contain semantics that appear only in raw streams.
• The filesystem is not watched. Call refresh_workspace() after changes. A refresh invalidates all prior cursors.
• Full content search is opt-in/progressive and limited to semantic leaves the indexer considers searchable. Check coverage instead of assuming complete search results.
• Resource limits are defense in depth, not hard OS isolation. parser_timeout_seconds is a hard total-call deadline for isolated semantic adapters, but not for metadata scanning, text/CSV, or binary reads.
• Optional suite conversion is a semantic alternative view, not a pixel-exact rendering. A hard parent timeout cannot guarantee cleanup of a vendor process that is itself stuck below COM or LibreOffice process control.
• The stated 100,000-node and warm p95 performance figures are design targets, not certified guarantees in 0.1.6; CI covers correctness across Windows, Linux, and macOS but does not emulate a specific Windows-x64 SSD workload.

Development

See CONTRIBUTING.md and the 0.1.0 alpha acceptance record. The complete local gate is:

python scripts/generate_json_schemas.py --check
ruff check .
mypy
pytest
python -m build --no-isolation
python scripts/check_wheel.py dist --install-smoke
python scripts/check_wheel.py --reproducible

The wheel check can also build in a temporary directory when no path is given:

python scripts/check_wheel.py

Use --reproducible to build twice with a fixed ZIP epoch and require identical SHA-256 digests.

License

Licensed under the Apache License 2.0.