tokenscope 0.2.0

TokenScope: Token-Aware Directory Explorer for LLMs

TokenScope

Token-Aware Directory Explorer for Large Language Models (LLMs).

A Model Context Protocol (MCP) server that helps LLMs efficiently explore and understand codebases and directory structures.

Overview

TokenScope provides intelligent directory structure analysis and token-aware file content exploration designed for LLMs like Claude. It helps LLMs understand codebases by:

1. Exploring directory structures with token-aware summarization
2. Viewing file contents with token limitations in mind
3. Generating comprehensive reports about directories

Key Features

Token-Aware Directory Exploration

• Automatic summarization for large directories while showing small directories in full
• Respect for token limits to maximize useful information within constraints
• Built-in security with base path validation
• Smart filtering with default patterns and .gitignore support
• Accurate directory statistics for even the largest directories

Simple, Intuitive Tools

TokenScope provides just three core tools:

1. explore_directory - Scan and understand directory structures
2. view_content - Access file contents with token awareness
3. generate_report - Create comprehensive reports (with option to save to file)

Installation

Prerequisites

• Python 3.10 or higher
• uv (recommended for dependency management)

Installation (PyPI)

This is the recommended method for most users who just want to use TokenScope:

# Install from PyPI using uv (recommended)
uv pip install tokenscope

Running TokenScope

The --base-path argument is mandatory for security reasons. It restricts all file operations to the specified directory.

# Run using the installed package
uv run --with tokenscope tokenscope --base-path /path/to/allowed/directory

Configuring in Claude Desktop

1. Locate Claude Desktop's configuration file (typically ~/.config/claude/config.json)

2. Add TokenScope to the mcpServers section:

"mcpServers": {
  "TokenScope": {
    "command": "uv",
    "args": [
      "run",
      "--with",
      "tokenscope",
      "tokenscope",
      "--base-path",
      "/your/secure/base/path"
    ]
  }
}

3. Replace /your/secure/base/path with the directory you want to restrict operations to

4. Save the configuration file and restart Claude Desktop

Usage

Running TokenScope Server

The --base-path argument is required for security (it restricts file operations to the specified directory):

tokenscope --base-path /path/to/allowed/directory

Testing Tools Directly

TokenScope includes a test mode for trying out tools directly:

# Test directory exploration
tokenscope --base-path /path/to/allowed/directory --test "explore:/path/to/directory"

# Test with custom ignore patterns
tokenscope --base-path /path/to/allowed/directory --test "explore:/path/to/directory?ignore=cache,*.log,tmp/&gitignore=false"

# Test file viewing
tokenscope --base-path /path/to/allowed/directory --test "view:/path/to/file"

# Test report generation
tokenscope --base-path /path/to/allowed/directory --test "report:/path/to/directory"

# Test report generation with file output and custom ignore patterns
tokenscope --base-path /path/to/allowed/directory --test "report:/path/to/directory?ignore=*.bak,temp/ > /path/to/output.md"

Example Prompts

Here are some examples of how to use TokenScope with Claude:

Could you explore my project directory at /path/to/project and tell me about its structure?

Can you show me the content of the file at /path/to/file.py?

Please generate a comprehensive report about my project at /path/to/project and save it to /path/to/report.md

Accurate Directory Statistics

TokenScope now provides two levels of directory information:

1. Quick Scan Statistics: Information about files and directories visible in the tree view
2. Full Directory Statistics: Complete counts of ALL files and directories, even in very large repositories

This dual approach ensures that even for massive directories (with thousands or millions of files), you'll get accurate information about the total number of files, directories, and disk usage. This is especially valuable when working with large codebases where a complete directory listing would exceed token limits.

Example Output

QUICK SCAN STATISTICS (files visible in tree):
Files shown in tree: 47
Directories shown in tree: 16
Size shown in tree: 185.9 MB

FULL DIRECTORY STATISTICS (all files):
Total files: 16,059
Total directories: 8
Total disk size: 2.1 GB

Smart Filtering with Ignore Patterns

TokenScope automatically filters out common directories and files that typically don't contribute to understanding a codebase:

• Default ignored patterns: .git/, .venv/, venv/, __pycache__/, node_modules/, build/, dist/, etc.
• Custom ignore patterns: You can specify additional patterns to ignore via the ignore_patterns parameter
• .gitignore support: TokenScope can automatically respect .gitignore files in the directories it scans

This functionality helps prevent token waste on irrelevant files and directories like:

• Dependency directories (e.g., node_modules, virtual environments)
• Build artifacts and cache directories
• Version control metadata
• IDE configuration files

Using Ignore Patterns in CLI Test Mode

# Ignore specific patterns
tokenscope --base-path /path --test "explore:/code?ignore=*.log,temp/"

# Disable .gitignore processing
tokenscope --base-path /path --test "explore:/code?gitignore=false"

# Both together
tokenscope --base-path /path --test "explore:/code?ignore=*.tmp&gitignore=false"

Security Features

TokenScope includes important security features:

• All file operations are validated to ensure they're within the specified base directory
• Attempts to access files outside the base path are rejected
• The base path is set once when starting the server and cannot be changed without restart

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Built with FastMCP
• Uses tiktoken for accurate token counting