ctxctx 0.3.10

Intelligently select, format, and present relevant project files and directory structure as context for LLMs.

LLM Context Builder: Focused Project Context for Large Language Models

A powerful Python package designed to intelligently select, format, and present relevant project files and directory structure as context for Large Language Models (LLMs). Avoid token limits, reduce noise, and get more accurate, actionable responses from your AI assistant.

---

🌟 Why Use This?

Working with LLMs for code-related tasks is incredible, but they often struggle with:

1. Token Limits: Sending an entire codebase is impossible and wasteful.
2. Information Overload: Even if possible, too much irrelevant code confuses the model.
3. Lack of Structure: Raw file dumps lack the directory context a human developer would have.

LLM Context Builder solves these problems by allowing you to:

• Precisely select the files, folders, or specific line ranges you want to include.
• Automatically ignore irrelevant files (like node_modules or build artifacts) using powerful ignore rules (including .gitignore).
• Provide a clear project overview with an automatically generated directory tree.
• Format output in easily parsable Markdown or JSON.

This results in more focused, relevant, and accurate responses from your LLM, helping you code faster and more effectively.

---

🚀 Getting Started

1. Installation: Install ctxctx directly from PyPI:

   pip install ctxctx
   

   Alternatively, if cloning the repository for development:

   git clone https://github.com/gkegke/ctxctx.git
   cd llm-context-builder
   # Install with poetry (recommended for development)
   poetry install --with dev
   

2. Basic Usage: Once installed, you can use the ctxctx command directly from your terminal in any project directory:

   ctxctx
   

   This will generate prompt_input_files.md and prompt_input_files.json containing only the directory tree of your project, up to a default depth of 3.

---

📖 Table of Contents

• LLM Context Builder: Focused Project Context for Large Language Models
  • 🌟 Why Use This?
  • 🚀 Getting Started
  • 📖 Table of Contents
  • ✨ Key Features & Usage Examples
    • 1. Basic Usage: Include Directory Tree
    • 2. Including Specific Files & Folders
    • 3. Ignoring Files & Folders
    • 4. Force Including Files & Folders (Override Ignores)
    • 5. Targeting Specific Line Ranges
    • 6. Using Glob Patterns for Flexible Selection
    • 7. Passing Arguments from a File
    • 8. Pre-defined Context Profiles
    • 9. Output Formats (Markdown & JSON)
    • 10. Dry Run Mode
    • 11. Discovering Files with --list-files
  • ⚙️ Configuration
  • 🪵 Logging Configuration
    • Console Output
    • File-based Logging
    • Benefits for CI/CD
  • 🤝 Contributing
  • 📄 License

---

✨ Key Features & Usage Examples

The tool outputs its results into prompt_input_files.md (Markdown) and prompt_input_files.json (JSON) by default, based on the OUTPUT_FORMATS configuration.

1. Basic Usage: Include Directory Tree

The simplest way to get context is just to include your project's directory structure. This gives the LLM a high-level overview of your project's layout, which is often very helpful.

ctxctx

This will generate prompt_input_files.md and prompt_input_files.json containing only the directory tree of your project, up to a default depth of 3.

2. Including Specific Files & Folders

The most common use case is to provide the content of a few specific files or all files within a specific folder.

• Include a single file:

  ctxctx src/main.py
  

• Include multiple files:

  ctxctx src/utils.js README.md
  

• Include all files within a folder (recursively, up to SEARCH_MAX_DEPTH):

  ctxctx config/
  

• Combine files and folders:

  ctxctx tests/backend/ src/data_models.py
  

3. Ignoring Files & Folders

Crucial for large projects! The tool uses a robust ignore system to ensure you don't send irrelevant or sensitive files to the LLM.

• Automatic Gitignore: By default, the script respects your project's .gitignore file.
• Built-in Ignores: Common build artifacts, temporary files, and environment directories (node_modules, __pycache__, .venv, .git, .DS_Store, etc.) are ignored automatically.
• Additional Ignore Files: The script also looks for other common ignore files like .dockerignore, .npmignore, and .eslintignore defined in ADDITIONAL_IGNORE_FILENAMES.
• ctxctx-Specific Ignores: The tool automatically ignores its own configuration and output files (like .ctxctx.yaml, the .ctxctx_cache/ directory, prompt_input_files.md, and prompt_input_files.json) by default. These are embedded directly in the tool's core configuration's EXPLICIT_IGNORE_NAMES. You can always add more explicit ignore patterns to your .ctxctx.yaml file if needed.

4. Force Including Files & Folders (Override Ignores)

Sometimes, you want to include a file or folder that is normally ignored by ctxctx's default rules, .gitignore, or your custom ignore rules in .ctxctx.yaml. The "force include" feature allows you to explicitly override these ignore rules for specific paths.

• Syntax: Prefix the file or folder path (or glob pattern) with force:.

  • Example: ctxctx 'force:path/to/file.js'

• How it works: When ctxctx encounters a query starting with force:, it marks that path as "force included". During processing, if a file matches any ignore rule, ctxctx first checks if it's explicitly force-included. If it is, the file will be included in the context, regardless of other ignore patterns.

• Important Nuance for Simple Filenames: When using force: followed by a simple filename (i.e., no directory separators like / or \\, and no glob wildcards like * or ?), ctxctx will only look for that file directly in the project's root directory. If you want to force-include a simple filename that resides in a subdirectory, you must specify its full path or a glob pattern that includes its path (e.g., force:src/config.py). This prevents unintended inclusion of identically named files deep within subdirectories, which is particularly useful for project-level files like LICENSE or .gitignore.

  • Example: force:.gitignore If you have /project/.gitignore and /project/frontend/.gitignore, ctxctx 'force:.gitignore' will only include /project/.gitignore. If you wanted the one in frontend/, you would need to specify its path: ctxctx 'force:frontend/.gitignore'.

  • Example: force:README.md If you have README.md at the root and docs/README.md, ctxctx 'force:README.md' will only include the root README.md.

  • This specific root-only behavior does not apply if your force: query includes directory separators (e.g., force:src/config.py) or glob patterns (e.g., force:*.log). In those cases, the search remains recursive up to SEARCH_MAX_DEPTH, and force-include simply overrides ignore rules wherever the pattern matches.

• Examples:

  • Force include a specific log file:

    ctxctx 'force:debug.log'
    

    (Even if *.log is in your .gitignore or debug.log is in your .ctxctx.yaml EXPLICIT_IGNORE_NAMES, it will be included. If debug.log only exists in a subdirectory, this command will not find it, due to the nuance described above.)

  • Force include a file inside an ignored directory:

    ctxctx 'force:node_modules/my_custom_module/index.js'
    

    (Normally node_modules is ignored, but this specific file will be included. This is a path-specific query, so it searches deeply.)

  • Force include all build artifacts (using a glob):

    ctxctx 'force:build/**/*.js'
    

    (If your build directory is typically ignored, this will include all JavaScript files within it. This is a glob query, so it searches deeply.)

  • Combine force-include with line ranges:

    ctxctx 'force:temp/sensitive_data.py:10,20'
    

    (Includes only lines 10-20 from sensitive_data.py, even if temp/ is ignored. This is a path-specific query, so it searches deeply.)

This feature provides granular control, ensuring that critical files are always part of your LLM context, even if they would otherwise be filtered out.

5. Targeting Specific Line Ranges

For very precise context, you can include one or more specific ranges of lines from a file. This is perfect for focusing on a few key sections of code for debugging or refactoring. The output will clearly mark the included line ranges and indicate where content has been omitted.

• Syntax: filepath:start1,end1:start2,end2... (lines are 1-indexed and inclusive).

• Example (Single Range):

  ctxctx 'src/api/user_routes.py:100,150'
  

  This will include lines 100 through 150 from src/api/user_routes.py.

• Example (Multiple Ranges):

  ctxctx 'src/data_processor.py:20,45:200,215'
  

  This will include lines 20-45 and 200-215 from the same file, with a comment indicating the omitted lines in between.

6. Using Glob Patterns for Flexible Selection

Glob patterns provide a powerful way to select multiple files based on wildcards.

• Syntax: Standard Unix-style glob patterns (e.g., *.py, src/**/*.js). Remember to quote patterns to prevent shell expansion.
• Example:
  • All Python files:

    ctxctx '*.py'
    

  • All JavaScript or TypeScript files within src/ and its subdirectories:

    ctxctx 'src/**/*.{js,ts}' # (Note: Shell might expand {js,ts}, quote carefully or run in a compatible shell)
    # Safer alternative for cross-platform (multiple arguments):
    ctxctx 'src/**/*.js' 'src/**/*.ts'
    

  • All Markdown files in the root or docs/ folder:

    ctxctx '*.md' 'docs/*.md'
    

7. Passing Arguments from a File

For very long or complex ctxctx commands, or for commands you use frequently, you can store your queries and flags in a text file and pass that file to ctxctx. This helps keep your terminal commands clean and makes them easily repeatable.

• Syntax: ctxctx @filename

• How it works: ctxctx will read each line from the specified file as if it were a separate command-line argument. Lines starting with # are treated as comments and ignored.

• Example my_queries.txt:

  # This is a comment, it will be ignored
  src/main.py
  tests/unit/test_config.py:10,25:50,60
  '*.md'
  docs/api/
  --profile backend_dev
  

• Usage:

  ctxctx @my_queries.txt
  

  This command would be equivalent to running:

  ctxctx src/main.py 'tests/unit/test_config.py:10,25:50,60' '*.md' docs/api/ --profile backend_dev
  

8. Pre-defined Context Profiles

For common tasks, you can define profiles within your main .ctxctx.yaml file to create reusable context definitions. Profiles now use a powerful funnel-based system:

1. include: A list of glob patterns that defines the initial set of files.
2. queries: (Optional) Ad-hoc queries (like specific files, line ranges, or force: includes) are added to the set.
3. exclude: (Optional) A list of glob patterns that removes files from the final set.

This allows you to build broad contexts with include and then precisely refine them with exclude and queries.

1. Define Profiles in your .ctxctx.yaml file in your project's root directory:

   # .ctxctx.yaml
   ROOT: .
   OUTPUT_FILE_BASE_NAME: prompt_input_files
   OUTPUT_FORMATS:
   - md
   - json
   TREE_MAX_DEPTH: 3
   # ... other global settings ...
   
   profiles: # Profiles are now a top-level key in .ctxctx.yaml
     backend_api:
       description: "Core backend API files, excluding tests and configs."
       include:
         - 'src/server/**/*.py'  # All python files under src/server
       exclude:
         - 'src/server/tests/**'   # Exclude the test subdirectory
         - 'src/server/config.py' # Exclude the config file
       queries:
         - 'requirements.txt'      # Explicitly add the requirements file
   
     frontend_component:
       description: "Focus on a specific component, its styles, and tests."
       include:
         - 'src/components/UserProfile/**' # All files for the component
       exclude:
         - 'src/components/UserProfile/**/*.snap' # Exclude snapshots
       # Override a global config setting just for this profile
       tree_max_depth: 4
   
     refactor_task:
       description: "A specific refactoring task with precise line numbers."
       queries:
         # Use 'queries' when you only need a few specific files/ranges
         - 'src/data/processor.py:10,45:100,120'
         - 'src/utils/helpers.py:1,30'
         - 'tests/test_processor.py'
   

2. Use a profile:

   ctxctx --profile backend_api
   ctxctx --profile refactor_task
   

   You can also combine profiles with additional ad-hoc queries from the command line:

   ctxctx --profile frontend_component 'public/index.html'
   

9. Output Formats (Markdown & JSON)

The tool generates two output files by default (prompt_input_files.md and prompt_input_files.json) to give you flexibility depending on what your LLM prefers or how you want to review the context.

• Markdown (.md): Human-readable, includes directory tree, and uses Markdown code blocks with syntax highlighting hints for file contents. Great for reviewing the context yourself before sending it, or for models that prefer structured text.
• JSON (.json): Machine-readable structured data. Contains the directory tree as a string and an array of file objects, each with path, content, and any line/function details. Ideal for programmatic use or models that perform better with structured JSON input.

You can configure which formats are generated in your .ctxctx.yaml file (or via a profile).

10. Dry Run Mode

Test your queries and configurations without writing any files. The full output will be printed directly to your console.

ctxctx --dry-run 'src/config.py' '*.md'

11. Discovering Files with --list-files

When you need to select from a large number of files, the --list-files command simplifies the process. It prints a clean, sorted list of all files that ctxctx can see after applying all ignore rules (from .gitignore, .ctxctx.yaml EXPLICIT_IGNORE_NAMES, etc.).

Its primary use is to generate an argument file that you can edit and pass back to ctxctx.

• How it works:

  1. Generate a list of all potential files. The command directs logs to stderr, so you can safely redirect the output.

     ctxctx --list-files > my_context.txt
     

  2. Open my_context.txt in your editor. Comment out (#) or delete the lines for files you wish to exclude.
  3. Feed the curated list back into ctxctx using the @ prefix.

     ctxctx @my_context.txt
     

• Combining with Profiles: You can also use it with profiles to see exactly what files a profile includes, providing a great starting point for a more specific context.

  ctxctx --profile backend_api --list-files > backend_files.txt
  # Now edit backend_files.txt and run:
  ctxctx @backend_files.txt
  

---

⚙️ Configuration

The tool's behavior can be customized by creating a .ctxctx.yaml file in your project's root directory. Any values defined in this file will override the tool's defaults. Additionally, profiles can override these global settings.

Key configurable options include:

• ROOT: The base directory for your project (defaults to . - current directory).
• OUTPUT_FILE_BASE_NAME: Base name for output files (e.g., prompt_input_files).
• OUTPUT_FORMATS: List of desired output formats (markdown, json).
• TREE_MAX_DEPTH: Maximum recursion depth for the directory tree view.
• TREE_EXCLUDE_EMPTY_DIRS: If true, empty directories (after applying ignore rules) will not be included in the tree.
• SEARCH_MAX_DEPTH: Maximum recursion depth for file content search.
• MAX_MATCHES_PER_QUERY: Max number of files a single query can return before an error is raised (prevents accidental large inclusions).
• EXPLICIT_IGNORE_NAMES: A set of exact file/folder names or relative paths to always ignore. This now includes the tool's internal output and cache files by default. You can add your own custom ignore names here.
• SUBSTRING_IGNORE_PATTERNS: A list of substrings that, if found anywhere in a file's relative path, will cause it to be ignored.
• ADDITIONAL_IGNORE_FILENAMES: List of other ignore files (e.g., .dockerignore) to load in addition to .gitignore.
• DEFAULT_CONFIG_FILENAME: The name of the main configuration file (defaults to .ctxctx.yaml).
• USE_GITIGNORE: Boolean to enable/disable .gitignore integration.
• GITIGNORE_PATH: Relative path to your main .gitignore file.
• USE_CACHE: Boolean to enable/disable caching of the project's file list.

---

🪵 Logging Configuration

ctxctx provides flexible logging options to help you debug issues, monitor execution, and capture detailed output, especially useful in automated environments like CI/CD.

Console Output

By default, ctxctx logs informational messages to your console (stdout).

• Enable Debug Mode: Use the --debug flag to increase the verbosity of console output, showing detailed debugging information.

  ctxctx --debug src/main.py
  

File-based Logging

For persistent logs or detailed analysis, you can direct all logging output to a file.

• Log to a File: Use the --log-file <path> argument to write all logs (including DEBUG level) to the specified file. This is highly recommended for CI/CD pipelines or when running ctxctx in non-interactive scripts, as it ensures all details are captured without relying on console output.

  # Log all output to ctxctx.log at DEBUG level
  ctxctx src/cli.py --log-file ctxctx.log
  
  # Combine with other arguments
  ctxctx 'src/**/*.py' --profile backend_dev --log-file debug_output.txt
  

Benefits for CI/CD

The --log-file argument is invaluable for Continuous Integration/Continuous Deployment (CI/CD) pipelines:

• Persistent Records: Capture full execution logs for every build, even if the pipeline fails, allowing for post-mortem analysis.
• Detailed Debugging: Provide engineers with comprehensive information for troubleshooting build issues or unexpected ctxctx behavior within automated workflows.
• Clean Console: Avoids flooding the CI/CD console output with verbose details, keeping the primary build logs focused.
• Auditing: Maintain an auditable trail of what context was generated for specific code changes.

---

🤝 Contributing

Contributions are welcome! If you have ideas for new features, improvements, or bug fixes, please open an issue or submit a pull request.

Areas for future improvement include:

• Git Integration: Automatically include files based on Git status (e.g., staged, modified).
• Code-aware Extraction: Use AST (Abstract Syntax Tree) parsing to extract specific functions, classes, or methods from code files.
• Advanced Ignore Logic: More robust .gitignore parsing, including support for negation patterns (!).
• Interactive Mode: A CLI mode for interactively selecting files and folders to include in the context.

Performance improvements being considered IF real world usage calls for it (increases codebase complexity):

• Content Change Detection for Cache: Currently, the cache only tracks file list changes based on config/ignore file mtimes. Adding content hashing could make cache invalidation more precise.
• Parallel File Content Reading

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.