Skip to main content

RepoHarvester

CI

Clone a git repository (or point at a local folder) and compile all of its text files into one file. Handy for feeding a codebase to an LLM, code review, or offline reading.

Features

  • Works with SSH/HTTPS git URLs (shallow clone into a temp dir) or local folders.
  • Skips binary/noise by default: media, archives, executables, lock files, node_modules, virtualenvs, caches, .git, .github, and more.
  • Optional comment removal for common languages. String literals are kept, so "http://..." and "#fff" stay intact. Python uses the real tokenizer; other languages use regexes.
  • Size limit per file. Non-UTF-8 files are skipped and listed in skipped_files.txt.
  • Respects .gitignore: in a git repo only tracked and untracked-but-not-ignored files are harvested.
  • Logs a rough token estimate (~4 characters per token) so you know if it fits an LLM context.
  • Each file is headed by its path relative to the repo root. The output folder is never harvested into itself.
  • No runtime dependencies. The GUI is optional.

Install

Requires Python 3.12+ and git.

uv tool install repoharvester   # or: pipx install repoharvester / uv run repoharvester ... from a clone

Usage

repoharvester <repo-url-or-folder> [options]
Option Description
-r, --remove Remove comments from code files
--no-skip GROUP[,GROUP] Include groups that are skipped by default (repeatable)
--max-size KB Skip files larger than this (default 1000). Files over 500 KB are logged
--exclude DIR[,DIR] Folder names to exclude, along with their contents (repeatable)
-o, --output-dir Output directory (default output)
--log FILE Also write the log to a file

Groups: archive, audio, cache_files, certificate, compiled_code, configuration, database, executables, font, log_files, media, node_modules, office, package_locks, system, temporary, video, virtual_env. See EXTENSION_GROUPS, FILENAME_GROUPS and FOLDER_GROUPS in repoharvester.py.

Example:

repoharvester git@github.com:username/repo.git -r --no-skip configuration --max-size 250 --exclude vendor,docs

GUI

uv tool install 'repoharvester[gui]'   # then: repoharvester-gui
uv run --extra gui repoharvester-gui    # from a clone

Paste a git URL or choose a folder, tick anything extra to include, press Harvest (or Enter), then Open file. While a clone runs the button turns into Cancel. Settings are remembered in ~/.config/repoharvester/gui.json. On Linux the project prefers the system Python: uv's bundled builds ship Tk without font antialiasing.

Output format

## <repo-name>
### <relative/path/to/file.ext>
<file contents>
### end of file
...

Written to <output-dir>/<repo-name>_all_files.txt, with skipped_files.txt next to it.

Development

uv run --group dev pytest
uvx ruff check .

License

MIT. See LICENSE.

Metadata

Release files for repoharvester 0.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for repoharvester 0.2.1
File Size Uploaded Provenance Actions
repoharvester-0.2.1.tar.gz 15.8 kB Without provenance Details for repoharvester-0.2.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for repoharvester 0.2.1
File Interpreter ABI Platform Actions
repoharvester-0.2.1-py3-none-any.whl Python 3 none any Details for repoharvester-0.2.1-py3-none-any.whl

Total release size: 29.0 kB

Release files / repoharvester-0.2.1.tar.gz

Download URL repoharvester-0.2.1.tar.gz
Size 15.8 kB
Tags Source
SHA-256 checksum
How to use checksums
dd74ceb59d69ced851a806c78013e13140eabb642b0f015d079f1ce52372c3f6
BLAKE2b-256 checksum
How to use checksums
00a406673cadb2d3b5fc5d08dceeaca3ccb95e48a6bacb5199f14b9458edbbb1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.19 {"installer":{"name":"uv","version":"0.12.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / repoharvester-0.2.1-py3-none-any.whl

Download URL repoharvester-0.2.1-py3-none-any.whl
Size 13.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c6d39279fe26261bbaab9290de0a3ea2208356b209bd8d29e4787aec3e2f298f
BLAKE2b-256 checksum
How to use checksums
9bb7054d94738247f2aebd99c776a952f63fcb0f12a92f4b3975d7bedb886c2b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.19 {"installer":{"name":"uv","version":"0.12.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for repoharvester 0.2.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page