AI Repo Mapper

A lightweight, AST-driven repository mapping tool designed to scan codebases, parse source code files, and extract structured signatures (classes, functions, and key declarations) without relying on fragile regular expressions.

Table of Contents

• Overview
• Architecture & Project Structure
• Features
• Prerequisites
• Installation & Deployment
• Quick Start: Testing the App & UX
• Usage
• CLI Arguments
• Export Formats
• Using as a System-Wide Tool
• Publishing to PyPI
• Running Tests
• Performance
• Future Improvements

---

Overview

AI Repo Mapper addresses the challenge of understanding large or unfamiliar codebases by extracting a high-level representation of code structure. Utilizing Tree-sitter for robust Abstract Syntax Tree (AST) parsing, it generates accurate function and class signature maps while strictly respecting .gitignore rules and path exclusion configurations.

---

Architecture & Project Structure

The project is structured into clear modular components:

ai-repo-mapper/
├── main.py              # CLI entry point parsing user arguments
├── map_repo.py          # Repository mapper orchestrator with export formats
├── scanner.py           # Gitignore loader and path matching logic via pathspec
├── parser_engine.py     # Tree-sitter AST extraction engine (Python, JS, TS, Go, HTML)
├── token_counter.py     # Token estimation for LLM context planning
├── config.json          # Configuration for ignored dirs and supported extensions
└── tests/
    ├── run_test.py          # Original test runner with mock snippets
    ├── test_fixes.py        # Unit tests for bug fixes (39 tests)
    ├── test_integration.py  # Integration tests for CLI workflow (53 tests)
    ├── test_security.py     # Security tests (20 tests)
    ├── test_reliability.py  # Reliability tests (17 tests)
    ├── test_ux.py           # UX tests (21 tests)
    └── test_benchmarks.py   # Performance benchmarks with regression thresholds (24 tests)

---

Features

• Multi-Language AST Parsing: Precise extraction of code signatures using Tree-sitter parsers for Python, JavaScript, TypeScript, Go, and HTML.
• Smart Filtering: Automatically respects .gitignore rules and custom exclusion directives using pathspec.
• Multiple Export Formats: Output maps as Markdown (default), JSON, or YAML.
• Smart Import Resolution: --dump-smart follows imports and extracts only the referenced symbols.
• Token Counting: Estimates token count for LLM context planning.
• Robust Error Handling: Graceful handling of invalid config, syntax errors, binary files, and Unicode content.
• Security Features: Path traversal protection, symlink detection, file size limits, binary detection, and automatic secret redaction.
• Performance Benchmarks: Built-in benchmark suite with regression thresholds to catch slowdowns.
• Reliability: Structured logging, config validation, deterministic output, and graceful error handling.
• User Experience: Progress bars, color output, consistent exit codes, and timing info.
• Comprehensive Testing: 174 tests covering unit, integration, security, reliability, UX, and performance scenarios.

---

Prerequisites

• Python: Version 3.8 or higher.
• C Compiler: Required by tree-sitter for building/loading language grammars on some platforms.

---

Installation

Option 1: Install from GitHub (Recommended)

pip install git+https://github.com/siso-kh/project_summeriser.git

Option 2: Install via PyPI (when published)

pip install ai-repo-mapper

Option 3: Install from source (for development)

git clone https://github.com/siso-kh/project_summeriser.git
cd project_summeriser
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -e .

Option 4: Manual installation

git clone https://github.com/siso-kh/project_summeriser.git
cd project_summeriser
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt

---

Quick Start: Testing the App & UX

Follow these steps to install, run, and verify the tool works correctly on your machine.

Step 1: Clone & Install

# Clone the repository
git clone https://github.com/siso-kh/project_summeriser.git
cd project_summeriser

# Create a virtual environment (recommended)
python -m venv venv

# Activate it
# On macOS/Linux:
source venv/bin/activate
# On Windows:
venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Step 2: Verify Installation

# Check that the CLI works
python main.py --help

# You should see:
# usage: repo-mapper [-h] [--output OUTPUT] [--format {markdown,json,yaml}]
#                    ...

# Check the version
python main.py --version
# Output: repo-mapper 1.1.0

Step 3: Scan a Directory (Basic Test)

# Scan the current directory (the project itself)
python main.py

# Expected output:
# Scanning target directory: ./
# Found XX valid files. Starting AST mapping...
# [Success] Processed mapping saved to: codebase_map.md
# [Info] Estimated token count: XXX Tokens

# Open the output file to verify
# On macOS:  open codebase_map.md
# On Linux:  xdg-open codebase_map.md
# On Windows: start codebase_map.md

Step 4: Test Export Formats

# Test JSON export
python main.py --format json --output test_map.json
cat test_map.json | head -20
# Should show valid JSON with "metadata" and "files" keys

# Test YAML export
python main.py --format yaml --output test_map.yaml
cat test_map.yaml | head -20
# Should show valid YAML with metadata and files sections

# Clean up test files
rm -f test_map.json test_map.yaml codebase_map.md

Step 5: Test Security Features

# Test path traversal protection (should be blocked)
mkdir -p /tmp/test_escalation
echo 'SECRET = "password123"' > /tmp/test_escalation/secret.py
python main.py /tmp/test_escalation --output /tmp/test_out.md
# Should work, but secret should be redacted in output
grep -o 'REDACTED' /tmp/test_out.md && echo 'Secret was redacted!'

# Test symlink detection
ln -sf /etc/passwd test_link.txt 2>/dev/null || true
python main.py --output test_out.md
grep -c 'test_link.txt' test_out.md  # Should be 0 (symlink skipped)

# Clean up
rm -rf /tmp/test_escalation /tmp/test_out.md test_link.txt test_out.md

Step 6: Test Verbose Logging

# Run with verbose output
python main.py --verbose
# Should show DEBUG-level log messages with timestamps

Step 7: Test Error Handling

# Test with a file path instead of directory (should fail gracefully)
touch test_file.txt
python main.py test_file.txt 2>&1
# Expected: error message about file instead of directory, exit code 1

echo "Exit code: $?"  # Should print: Exit code: 1

# Test with invalid arguments
python main.py --invalid-flag 2>&1
# Expected: usage message, exit code 2

echo "Exit code: $?"  # Should print: Exit code: 2

# Clean up
rm -f test_file.txt

Step 8: Test on a Real Project

# Scan a well-known open-source project (e.g., requests, flask)
pip install requests  # If not installed
python main.py $(python -c "import requests; print(requests.__path__[0])")

# Or scan any Python project you have
dir="path/to/your/other/project"
python main.py "$dir" --output my_project_map.md --format json
cat my_project_map.md | head -30

rm -f my_project_map.md

Step 9: Run the Automated Test Suite

# Navigate to tests directory
cd tests

# Run each test suite one by one
python run_test.py          # Original AST tests (2 files)
python test_fixes.py        # Unit tests (39 tests)
python test_integration.py  # Integration tests (53 tests)
python test_security.py     # Security tests (20 tests)
python test_reliability.py  # Reliability tests (17 tests)
python test_ux.py           # UX tests (21 tests)
python test_benchmarks.py   # Performance benchmarks (24 tests)

# All should print: "All tests passed! ✅"

Step 10: Verify Exit Codes

# Success (exit code 0)
python main.py .
echo "Exit code: $?"

# Bad arguments (exit code 2)
python main.py --bogus 2>/dev/null
echo "Exit code: $?"

# Expected output:
# Exit code: 0
# Exit code: 2

---

Quick Test Checklist

#	Test	What to Look For	Pass?
1	python main.py --help	Shows usage info	☐
2	python main.py	Creates codebase_map.md	☐
3	python main.py --format json	Creates valid JSON file	☐
4	python main.py --format yaml	Creates valid YAML file	☐
5	python main.py --verbose	Shows debug logs	☐
6	python main.py --version	Shows repo-mapper 1.1.0	☐
7	python main.py <file>	Error message, exit code 1	☐
8	python main.py --bogus	Error message, exit code 2	☐
9	Secret in code → output	API keys are redacted	☐
10	cd tests && python test_ux.py	21/21 tests pass	☐

---

Usage

After installing (pip install ai-repo-mapper), you can use either command:

# Both work identically:
repo_mapper          # underscore
repo-mapper          # hyphen

# Or run as a module:
python -m ai_repo_mapper

Examples

# Scan current directory, output to codebase_map.md
repo_mapper

# Scan specific directory
repo_mapper /path/to/project

# Custom output file
repo_mapper --output my_map.md

# Export as JSON
repo_mapper --format json --output map.json

# Export as YAML
repo_mapper --format yaml --output map.yaml

# Dump all source files
repo_mapper --dump all

# Dump only programming files
repo_mapper --dump logic

# Dump specific files
repo_mapper --dump-files main.py scanner.py

# Smart dump - follows imports
repo_mapper --dump-smart main.py

# Show version
repo_mapper --version

Note: You can also run directly with python main.py if installed from source.

---

CLI Arguments

Argument	Description	Default
target_dir	Directory to scan	./
--output	Output file path	codebase_map.md
--format	Output format: markdown, json, yaml	markdown
--dump	Dump mode: none, logic, all	none
--dump-files	Specific files to dump completely	[]
--dump-smart	Target file + auto-dump its imports	[]
--follow-symlinks	Follow symbolic links (security risk)	false
--max-file-size	Maximum file size in MB	10
--no-redact	Disable automatic secret redaction	false
--verbose	Enable verbose debug logging	false

---

Export Formats

Markdown (default)

python main.py --output map.md

Produces a human-readable Markdown file with code signatures.

JSON

python main.py --format json --output map.json

Produces structured JSON with metadata and file entries:

{
  "metadata": {
    "total_files": 13,
    "generated_by": "AI Repo Mapper"
  },
  "files": [
    {
      "path": "./main.py",
      "extension": ".py",
      "signatures": ["def main():"],
      "source": null
    }
  ]
}

YAML

python main.py --format yaml --output map.yaml

Produces YAML without requiring PyYAML dependency.

---

Using as a System-Wide Tool

After installing via pip install ai-repo-mapper, both commands are available globally:

repo_mapper    # Use underscore
repo-mapper    # Or use hyphen

Quick Access from Any Directory

# From any project directory, just run:
repo_mapper

# This will scan the current directory and create codebase_map.md

Windows Users (PowerShell/CMD)

# After pip install, just type:
repo_mapper

# Or:
repo-mapper

macOS/Linux Users

# After pip install, just type:
repo_mapper

# Or add to your shell profile for autocomplete:
echo 'alias rm="repo_mapper"' >> ~/.bashrc

---

Security

AI Repo Mapper includes built-in security features:

Feature	Description
Path Traversal Protection	Prevents accessing files outside the target directory
Symlink Detection	Skips symbolic links by default (use --follow-symlinks to enable)
File Size Limits	Skips files larger than --max-file-size (default: 10MB)
Binary Detection	Automatically skips binary files
Secret Redaction	Detects and redacts API keys, passwords, and tokens in output

# Enable symlink following (use with caution)
repo-mapper --follow-symlinks /path/to/project

# Set custom file size limit
repo-mapper --max-file-size 5 /path/to/project

# Disable secret redaction
repo-mapper --no-redact /path/to/project

---

Reliability

AI Repo Mapper ensures reliability through:

Feature	Description
Structured Logging	Uses Python logging module with timestamps and log levels
Config Validation	Validates config.json schema on load, falls back to defaults
Deterministic Output	Same input always produces same output (for caching)
Graceful Error Handling	Never crashes, always provides useful error messages
Verbose Mode	Use --verbose for detailed debug output

# Enable verbose logging
repo-mapper --verbose /path/to/project

# Log levels: DEBUG, INFO, WARNING, ERROR
# Default level: INFO
# Verbose level: DEBUG

---

Running Tests

# Run all tests
cd tests
python run_test.py          # Original AST extraction tests
python test_fixes.py        # Unit tests for bug fixes (39 tests)
python test_integration.py  # Integration tests for CLI workflow (53 tests)
python test_security.py     # Security tests (20 tests)
python test_reliability.py  # Reliability tests (17 tests)
python test_ux.py           # UX tests (21 tests)
python test_benchmarks.py   # Performance benchmarks with regression thresholds (24 tests)

# Run from project root
python -c "import sys; sys.path.insert(0, '.'); from tests.run_test import setup_and_execute_tests; setup_and_execute_tests()"

# Run benchmarks with regression checks (exits with code 1 if regression detected)
cd tests && python test_benchmarks.py

---

User Experience

AI Repo Mapper provides a polished CLI experience:

Feature	Description
Progress Bars	Visual progress indicator for large codebases
Color Output	Color-coded terminal output for better readability
Exit Codes	Consistent exit codes (0=success, 1=error, 2=bad args)
Timing Info	Displays scan duration and file count in stderr

# Color output is enabled by default in terminal
repo-mapper /path/to/project

# Disable colors (for piping or non-TTY)
NO_COLOR=1 repo-mapper /path/to/project

---

Performance

Benchmarks are included in tests/test_benchmarks.py with regression thresholds:

Component	Throughput
Scanner	~300 files/sec (with security checks)
Parser	~200 files/sec
Full Pipeline (500 files)	~8 seconds
Token Counter	~1.4M tokens/sec

To run benchmarks:

cd tests && python test_benchmarks.py

---

Architecture Decisions

Decision	Rationale
Tree-sitter over regex	Robust AST parsing, handles edge cases
pathspec for gitignore	Native gitignore syntax support
Structured logging	Observability without cluttering stdout
Security by default	Symlinks blocked, secrets redacted
Deterministic output	Same input = same output (for caching)

---

Future Improvements

• Additional Language Support: Expand Tree-sitter integrations to support Rust, C++, and Java.
• Parallel Processing: Implement multi-process scanning for large monorepos.
• Incremental Scanning: Only scan changed files for faster re-scans.
• Caching: Cache AST parsing results for repeated runs.
• CI/CD Integration: GitHub Actions for automated testing on every commit.
• AI Integration: Support feeding extracted repository maps into LLM context windows for code analysis.

---

Publishing to PyPI (Make Available to All Users)

To make this tool installable by anyone via pip install ai-repo-mapper, follow these steps:

Step 1: Install Build Tools

pip install build twine

Step 2: Create a PyPI Account

1. Go to https://pypi.org
2. Create an account (use a dedicated email)
3. Enable 2FA for security

Step 3: Build the Package

# From the project root (where pyproject.toml is)
python -m build

This creates two files in dist/:

dist/
├── ai_repo_mapper-1.1.0.tar.gz    # Source distribution
└── ai_repo_mapper-1.1.0-py3-none-any.whl  # Wheel (pre-built)

Step 4: Upload to TestPyPI (Recommended First)

# Test on TestPyPI first
twine upload --repository testpypi dist/*

# Test installation from TestPyPI
pip install --index-url https://test.pypi.org/simple/ ai-repo-mapper

# Verify it works
repo-mapper --version

Step 5: Upload to Real PyPI

# Upload to real PyPI
twine upload dist/*

# You'll be prompted for your PyPI username and password

Step 6: Verify Public Installation

# Now anyone can install it!
pip install ai-repo-mapper

# Verify
repo-mapper --version
# Output: repo-mapper 1.1.0

# Or run as module
python -m ai_repo_mapper --help

Step 7: Add Badges to README

Add these badges to the top of your README:

[![PyPI version](https://badge.fury.io/py/ai-repo-mapper.svg)](https://pypi.org/project/ai-repo-mapper/)
[![Downloads](https://static.pepy.tech/badge/ai-repo-mapper)](https://pypi.org/project/ai-repo-mapper/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

---

Publishing Checklist

#	Task	Status
1	pyproject.toml has correct metadata	☐
2	Version matches in __init__.py and pyproject.toml	☐
3	python -m build succeeds	☐
4	Test on TestPyPI	☐
5	Upload to real PyPI	☐
6	pip install ai-repo-mapper works	☐
7	repo-mapper --help shows usage	☐
8	Add badges to README	☐

---

Updating the Package

When you release new versions:

# 1. Update version in pyproject.toml and __init__.py
# 2. Build new version
python -m build

# 3. Upload (PyPI doesn't allow reusing version numbers)
twine upload dist/*

# 4. Clean up old builds
rm -rf dist/ build/ *.egg-info/

---

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Run the test suite (cd tests && python run_test.py)
4. Commit your changes (git commit -m 'Add amazing feature')
5. Push to the branch (git push origin feature/amazing-feature)
6. Open a Pull Request

---

Note: This project was implemented with the help of AI (Gemini) and Codebuff.