agent-folder-workspace 0.1.0

Offline, lazy, agent-oriented exploration of heterogeneous folders

agent-folder-workspace

agent-folder-workspace presents one local directory as a stable, paginated node hierarchy for agents. It discovers file metadata first and opens document bodies only when a caller expands or reads a node. The default parser path is local Python code: Microsoft Office and LibreOffice are not required.

Project status: 0.1.0 is an alpha contract. Read-only exploration and byte access are implemented; editing, rendering, OCR, and complete semantic coverage of every supported container are not.

Python 3.11 through 3.14 is supported. This repository does not assume that a release has already been published to a package index. Install a checkout with:

python -m pip install .
folderws --help

Runtime installation uses only Python packages from pip; the Windows-only pywin32 dependency is selected by an environment marker. Java, .NET, Microsoft Office, and LibreOffice are never installed or required by this package. Already-installed office suites are optional probe/conversion backends.

Quick start

from pathlib import Path

from agent_folder_workspace import FolderWorkspace

with FolderWorkspace.open(Path("./documents")) as workspace:
    if not workspace.wait_until_ready(timeout=10):
        raise TimeoutError("metadata scan is still running")

    info = workspace.workspace_info()
    first_page = workspace.list_children(info.root_node.id, limit=50)
    for node in first_page.nodes:
        print(node.id, node.kind, node.relative_path, node.content_status)

Call list_children(file_node.id) to expose a format-specific hierarchy, then use read_content() on a semantic node or read_binary() for a bounded byte range. Results are Pydantic models and can be serialized with result.model_dump(mode="json").

Architecture

The package separates transport, workspace policy, node discovery, format adapters, and caching:

1. FolderWorkspace validates and owns one root directory and one generation.
2. NodeRegistry builds an in-memory metadata tree. The initial scan uses directory entries and file stat data; it does not read file payloads.
3. AdapterManager keeps text, CSV, and bounded binary access direct. It sends semantic children/read calls for structured text, SQLite, OOXML, ODF, PDF, and legacy Office to a bounded pool of spawn-based parser workers.
4. Parser workers block Python socket/DNS entry points before importing risky adapters and enforce one deadline across startup, queueing, IPC, and parsing.
5. SizedLru retains bounded decoded pages in memory. ContentIndex stores an optional per-workspace SQLite search index in the application cache.
6. The Python API, Typer CLI, and FastMCP stdio server share the same workspace methods and versioned response contracts.

Node IDs are deterministic hashes of relative and logical paths; they do not contain the absolute workspace path. Pagination cursors are opaque, integrity-protected, workspace-instance tokens. refresh_workspace() advances the generation, rebuilds metadata, clears decoded pages, synchronizes an open index, and makes older cursors stale.

See the architecture notes for lifecycle, data-flow, cache, and error details.

Python API

The supported top-level entry points are:

API	Purpose
FolderWorkspace.open(root, config=None)	Open one existing local directory.
workspace_info()	Report the root node, generation, scan state, and index coverage counts.
list_children(node_id, cursor=None, limit=100)	Page through filesystem or virtual children.
get_node(node_id)	Fetch a node by stable ID.
read_content(node_id, cursor=None, limit=100, representation="semantic")	Read a semantic/source-record page; raw delegates to bounded binary access.
read_binary(node_id, offset=0, length=..., cursor=None)	Return at most 1 MiB as base64 and an optional continuation cursor.
search_content(query, cursor=None, limit=20)	Search filenames plus progressively indexed content.
index_content()	Walk supported semantic leaves into the persistent content index.
refresh_workspace()	Re-scan and invalidate old generation-bound cursors.
close()	Stop the scan worker/parser pool and close an open index.

FolderWorkspace is a context manager. Known file nodes and shallow requested directories remain usable while the deep metadata scan continues; a lookup for an as-yet undiscovered node can wait for that scan. Full indexing and exact coverage require the inventory to finish.

WorkspaceConfig controls page, binary, cache, embedded-file, parser-worker, archive, XML, and structural limits. It is frozen and rejects unknown fields. The package exports a stable error hierarchy (FolderWorkspaceError, InputValidationError, NodeNotFoundError, StaleCursorError, and ResourceLimitError) and six versioned wire models:

• CapabilityReportV1
• ContentPageV1
• DiagnosticV1
• NodePageV1
• NodeV1
• SearchPageV1

Committed JSON Schemas live in src/agent_folder_workspace/schemas/. Regenerate them with python scripts/generate_json_schemas.py; CI checks that they still equal Pydantic's model_json_schema() output.

MCP server

Run one stdio server for one root:

folderws mcp --root ./documents

The server exposes exactly these tools:

• workspace_info
• list_children
• get_node
• read_content
• read_binary
• search_content
• index_content
• refresh_workspace
• probe_backends

Results use the same versioned models as the Python API. A method that is absent from an injected workspace implementation returns a structured capability_unavailable response instead of failing server construction.

The transport is stdio. There is no HTTP listener, authentication layer, or multi-root broker. The process and its MCP client share the operating-system permissions of the account that launched folderws; choose the root narrowly.

CLI

# Serve a folder to an MCP client over stdio
folderws mcp --root DIR

# Detect optional backends without launching them (default)
folderws doctor

# Opt in to isolated Office/LibreOffice round-trip probes
folderws doctor --probe-office --json --timeout-seconds 10

# Build the complete semantic content index
folderws index --root DIR

# Remove only the selected application cache tree
folderws clear-cache [--cache-dir PATH]

# Materialize the bundled, managed OpenCode skill and plugin
folderws install-opencode --target DIR

Legacy positional roots/targets and doctor --active remain compatibility aliases; the explicit forms above are the primary interface.

install-opencode does not edit opencode.json or opencode.jsonc. It writes two ownership-marked resources atomically and always refuses foreign files, symbolic links, and Windows reparse points. The compatibility --force flag never bypasses ownership checks.

clear-cache removes only a non-link directory containing the package's cache ownership marker; an arbitrary directory supplied through --cache-dir is refused.

Lazy loading and cache

Lazy means that the initial recursive scan records names, types, sizes, and timestamps, but not document bodies. Expanding a file asks its adapter for virtual children. Reading a semantic leaf parses only the requested format path where the adapter permits it; some container adapters must still parse a whole XML part, table, or package directory to answer that request.

Text and CSV semantics, metadata scanning, and bounded memory-mapped original-file byte reads run in the workspace process. Structured text, SQLite, OOXML, ODF, PDF, and legacy Office semantic children/read calls, plus byte extraction from their container parts/streams/records, use a lazily started pool of parser_max_workers long-lived spawn processes (two by default). A parser_timeout_seconds deadline covers pool startup, queue wait, request and response IPC, and execution. A timed-out worker is terminated and replaced; the caller receives a stable parser_timeout diagnostic.

Supported embedded package parts and PDF attachments are mounted only when their node is expanded. The bytes are copied atomically, in bounded chunks, to a hashed per-workspace embedded cache file, then exposed below a contents node through the normal adapter/worker path. Depth, count, individual file size, and total disk-cache limits apply. This feature creates derived document copies on disk; folderws clear-cache removes them with the index.

Decoded ContentPageV1 values use a byte-sized in-memory LRU capped by max_content_cache_bytes (256 MiB by default). Reads are also indexed on a best-effort basis. index_content() performs a full supported semantic walk without retaining all decoded bodies in RAM.

The persistent index is an application-owned SQLite database below the platform cache directory, separated by a hash of the workspace root. It uses FTS5 when available and a plain-table fallback otherwise. max_disk_cache_bytes is checked after read-through chunks and completed files; least-recently-used indexed files are evicted and SQLite WAL/database pages are compacted. Embedded and alternative views enforce the same application-cache budget at their own materialization boundaries. Read-through index failures do not make an otherwise successful content read fail. Embedded mounting, by contrast, requires a writable bounded cache. Use folderws clear-cache to remove an explicitly selected cache tree safely.

Normal completion removes scratch immediately. On a later workspace/cache open, package-owned partial files and conversion/probe sessions older than five minutes are removed without traversing linked paths; recent items are retained to avoid interfering with another process.

Search coverage is the fraction of discovered files whose full indexing pass completed. Filename matches are available before content coverage reaches 1. If any directory could not be inventoried, the true denominator is unknown; coverage is then conservatively 0 and complete=false, even when every successfully discovered file has been indexed. The affected directory node is partial and carries a directory_scan_failed diagnostic.

Optional legacy conversion views

WorkspaceConfig.backend_policy accepts four closed values:

• python_first (default) uses the built-in DOC/XLS/PPT parser and attempts an actively usable Microsoft Office backend, then LibreOffice, only if that parser fails;
• python_only never starts an office suite;
• microsoft_office and libreoffice keep the Python view and additionally create a converted, provenance-marked OOXML child when a legacy file is first expanded.

Optional conversion is performed by a timeout-bounded child Python process. The result is verified as OOXML and mounted through the normal OOXML adapter. Its cache key includes the backend, adapter version, and SHA-256 of the source, and both conversion scratch and the retained result stay below the platformdirs per-workspace application cache rather than %TEMP% or /tmp. Microsoft automation disables macros, UI, events, external-link updates, and recent-file writes where the application API exposes those controls. LibreOffice uses a disposable cache-local profile, headless/safe-mode flags, and never touches the user's normal profile. The external applications are optional and are used only after an active round-trip capability check.

Security model

The normal Python parser path is offline and read-only. It does not upload documents and does not execute document macros. Important controls include:

• roots that are symbolic links or reparse points are rejected;
• child links are not followed by default, and resolved sources must remain under the selected root;
• SQLite files are opened with mode=ro and PRAGMA query_only=ON;
• YAML uses safe_load; XML uses defused parsers; HTML parsing disables network access;
• OOXML and ODF apply entry-count, expanded-size, XML-size, and compression-ratio checks; ODF rejects encrypted package entries;
• PDFs reject encryption and enforce configured content/attachment limits;
• binary reads, page sizes, and search inputs are bounded;
• risky semantic adapters run in bounded spawn workers; Python socket/DNS APIs and proxy variables are blocked before those adapters import, and hard call deadlines retire a hung worker;
• supported embedded files are materialized atomically only inside a checked, per-workspace cache path with depth/count/file/disk limits;
• pagination cursors are signed and generation-bound.

These controls reduce accidental traversal and common parser abuse; they are not a sandbox. A caller with read_binary access can intentionally obtain bytes from any regular file below the chosen root. Spawn workers inherit the account's filesystem permissions and are not an OS sandbox; their network block covers Python socket paths, not arbitrary native code. Text/CSV parsing, original-file reads, and embedded-byte materialization remain in the workspace process; container-part range extraction runs in parser workers. Size and timeout limits do not prove that every malformed file is harmless. Run the server under a minimally privileged account for untrusted collections.

folderws doctor --probe-office --json is different from default parsing: it explicitly launches isolated Microsoft Office COM applications or a headless LibreOffice process, copies bundled minimal legacy DOC/XLS/PPT fixtures into application-cache scratch, and verifies real OOXML round trips with timeouts and no-UI/no-macro/no-link settings. It never uses documents from the workspace, but it can still be affected by local application policy.

See SECURITY.md before exposing sensitive folders or reporting a vulnerability.

Supported formats

Family	Extensions	Current semantic surface
Plain/delimited text	.txt, .md, .csv, .tsv	Line or row blocks; header-derived CSV records.
Structured text	.json, .jsonl, .yaml, .yml, .xml, .html, .htm	Hierarchical values/elements, attributes, and text.
SQLite	.sqlite, .sqlite3, .db	Schema objects and paginated table/view rows, read-only.
OOXML	.docx, .docm, .xlsx, .xlsm, .pptx, .pptm	Core document/sheet/slide semantics plus raw package parts.
OpenDocument	.odt, .ods, .odp	Core text/table/sheet/slide semantics plus package parts.
PDF	.pdf	Metadata, pages, text blocks, extracted tables, image descriptors, attachments, structural objects, and original bytes.
Legacy Office	.doc, .xls, .ppt	Partial semantic extraction plus CFB streams and lossless known/unknown record bytes.
Other files	any other extension	Metadata and bounded original-byte reads only.

The detailed coverage and per-format caveats are in docs/formats.md. Windows-specific backend behavior is in docs/windows.md.

Limitations

• This is an explorer, not an editor, renderer, office-suite replacement, or general-purpose file-conversion service; optional legacy conversion exists only to mount an alternative semantic view.
• Semantic extraction is deliberately partial. Raw package/stream access makes omitted structures inspectable but does not turn them into high-level data.
• Formulas are reported where supported but never calculated. Cached values can be absent or stale.
• Scanned PDFs and images have no OCR path. Visual layout, charts, animations, tracked changes, styles, signatures, and embedded-object semantics are not comprehensively reconstructed.
• Encrypted/password-protected documents are unsupported. Macros and external links are exposed only as inert package data where available and are never executed or refreshed.
• Recognized embedded files are mounted recursively only from supported package/attachment nodes and only within configured depth/count/byte limits. Unsupported extensions remain raw; mounted bytes persist in the application cache until eviction or explicit clearing.
• Legacy .doc, .xls, and .ppt parsing recognizes a useful subset of binary records; valid files can contain semantics that appear only in raw streams.
• The filesystem is not watched. Call refresh_workspace() after changes. A refresh invalidates all prior cursors.
• Full content search is opt-in/progressive and limited to semantic leaves the indexer considers searchable. Check coverage instead of assuming complete search results.
• Resource limits are defense in depth, not hard OS isolation. parser_timeout_seconds is a hard total-call deadline for isolated semantic adapters, but not for metadata scanning, text/CSV, or binary reads.
• Optional suite conversion is a semantic alternative view, not a pixel-exact rendering. A hard parent timeout cannot guarantee cleanup of a vendor process that is itself stuck below COM or LibreOffice process control.
• The stated 100,000-node and warm p95 performance figures are design targets, not certified guarantees in 0.1.0; CI covers correctness across Windows, Linux, and macOS but does not emulate a specific Windows-x64 SSD workload.

Development

See CONTRIBUTING.md and the 0.1.0 alpha acceptance record. The complete local gate is:

python scripts/generate_json_schemas.py --check
ruff check .
mypy
pytest
python -m build --no-isolation
python scripts/check_wheel.py dist --install-smoke
python scripts/check_wheel.py --reproducible

The wheel check can also build in a temporary directory when no path is given:

python scripts/check_wheel.py

Use --reproducible to build twice with a fixed ZIP epoch and require identical SHA-256 digests.

License

Licensed under the Apache License 2.0.