Skip to main content

RepoHarvester

CI

Clone a git repository (or point at a local folder) and compile all of its text files into one file. Handy for feeding a codebase to an LLM, code review, or offline reading.

Features

  • Works with SSH/HTTPS git URLs (shallow clone into a temp dir) or local folders.
  • Skips binary/noise by default: media, archives, executables, lock files, node_modules, virtualenvs, caches, .git, .github, and more.
  • Optional comment removal for common languages. String literals are kept, so "http://..." and "#fff" stay intact. Python uses the real tokenizer; other languages use regexes.
  • Size limit per file. Non-UTF-8 files are skipped and listed in skipped_files.txt.
  • Respects .gitignore: in a git repo only tracked and untracked-but-not-ignored files are harvested.
  • Logs a rough token estimate (~4 characters per token) so you know if it fits an LLM context.
  • Each file is headed by its path relative to the repo root. The output folder is never harvested into itself.
  • No runtime dependencies. The GUI is optional.

Install

Requires Python 3.12+ and git.

uv tool install repoharvester   # or: pipx install repoharvester / uv run repoharvester ... from a clone

Usage

repoharvester <repo-url-or-folder> [options]
Option Description
-r, --remove Remove comments from code files
--no-skip GROUP[,GROUP] Include groups that are skipped by default (repeatable)
--max-size KB Skip files larger than this (default 1000). Files over 500 KB are logged
--exclude DIR[,DIR] Folder names to exclude, along with their contents (repeatable)
-o, --output-dir Output directory (default output)
--log FILE Also write the log to a file

Groups: archive, audio, cache_files, certificate, compiled_code, configuration, database, executables, font, log_files, media, node_modules, office, package_locks, system, temporary, video, virtual_env. See EXTENSION_GROUPS, FILENAME_GROUPS and FOLDER_GROUPS in repoharvester.py.

Example:

repoharvester git@github.com:username/repo.git -r --no-skip configuration --max-size 250 --exclude vendor,docs

GUI

uv run --extra gui python gui.py

Paste a git URL or choose a folder, tick anything extra to include, press Harvest (or Enter), then Open file. While a clone runs the button turns into Cancel. Settings are remembered in ~/.config/repoharvester/gui.json. On Linux the project prefers the system Python: uv's bundled builds ship Tk without font antialiasing.

Output format

## <repo-name>
### <relative/path/to/file.ext>
<file contents>
### end of file
...

Written to <output-dir>/<repo-name>_all_files.txt, with skipped_files.txt next to it.

Development

uv run --group dev pytest
uvx ruff check .

License

MIT. See LICENSE.

Metadata

Release files for repoharvester 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for repoharvester 0.2.0
File Size Uploaded Provenance Actions
repoharvester-0.2.0.tar.gz 15.7 kB Without provenance Details for repoharvester-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for repoharvester 0.2.0
File Interpreter ABI Platform Actions
repoharvester-0.2.0-py3-none-any.whl Python 3 none any Details for repoharvester-0.2.0-py3-none-any.whl

Total release size: 24.4 kB

Release files / repoharvester-0.2.0.tar.gz

Download URL repoharvester-0.2.0.tar.gz
Size 15.7 kB
Tags Source
SHA-256 checksum
How to use checksums
1697bb72eaedc89b1ffb4cb32ee45bef7ce0700a6121a8ad9caba6e5b56c9283
BLAKE2b-256 checksum
How to use checksums
184ab76b2f8b10a7d275c81d2d615a4644c8e0c4f625037432a90e22233d3e36
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.19 {"installer":{"name":"uv","version":"0.12.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / repoharvester-0.2.0-py3-none-any.whl

Download URL repoharvester-0.2.0-py3-none-any.whl
Size 8.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a833b64e13279b499001afdfdda265ea2ff2cd71e2978bd17dbf2cb3c6aab68f
BLAKE2b-256 checksum
How to use checksums
022b8f566462b525eba060bc213a718c1504551baeead579affd81e19f355804
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.19 {"installer":{"name":"uv","version":"0.12.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for repoharvester 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page