create-dump

Enterprise-Grade Code Dump Utility for Monorepos

create-dump is a production-ready CLI tool for automated code archival in large-scale monorepos. It generates branded Markdown dumps with Git metadata, integrity checksums, flexible archiving, retention policies, path safety, full concurrency, and SRE-grade observability.

Designed for SRE-heavy environments (Telegram bots, microservices, monorepos), it ensures reproducible snapshots for debugging, forensics, compliance audits, and CI/CD pipelines. It also includes a rollback command to restore a project from a dump file.

Built for Python 3.11+, leveraging AnyIO, Pydantic, Typer, Rich, and Prometheus metrics.

⚡ Quick Start

Prerequisites

• Python: 3.11 or higher
• Git: Optional, but recommended for metadata and git ls-files support.
• Docker/Podman: Optional, if running via container.

Installation

Via PyPI:

pip install create-dump

Via Source (Dev):

git clone https://github.com/dhruv13x/create-dump.git
cd create-dump
pip install -e .[dev]

Run (The "Hello World")

Navigate to your project root and run:

create-dump

This creates a markdown snapshot of your current directory in create_dump_output/ (or root).

Demo Snippet

Copy-paste this to see it in action:

# 1. Install
pip install create-dump

# 2. Dump your current folder (excluding hidden files)
create-dump single --use-gitignore --no-toc

# 3. View the result
head -n 20 *_all_create_dump_*.md

✨ Features

Core

• Branded Markdown: Auto-generated TOC (list or tree), language detection, and metadata headers.
• Smart Collection: Respects .gitignore automatically. Use --git-ls-files for blazing fast, Git-native file discovery.
• Multi-Mode:
  • single: Dump one project/directory.
  • batch: Recursively dump multiple subprojects in a monorepo.
  • rollback: Restore a project from a dump file.
• Live Watch: Run with --watch to auto-update the dump whenever files change.

Performance

• Async & Concurrent: Powered by anyio with up to 16 parallel workers for massive speedups on large repos.
• Smart Caching: Hashes config and file metadata to skip processing unchanged files.
• Low Footprint: Optimized for CI/CD pipelines.

Security & SRE

• Secret Scanning: Integrated detect-secrets scanning.
  • Fail on secret detection: --scan-secrets
  • Auto-redact secrets: --hide-secrets
• Safe Paths: Anti-traversal guards to prevent Zip-Slip attacks.
• Observability: Prometheus metrics server (default port 8000) and structured JSON logging.
• ChatOps: Native push notifications to Slack, Discord, Telegram, and ntfy.sh.

🛠️ Configuration

You can configure create-dump via CLI arguments, Environment Variables (loaded into config), or a TOML file (create_dump.toml or pyproject.toml).

Environment Variables & TOML Keys

Define these in [tool.create-dump] section of pyproject.toml or create_dump.toml.

CLI Arguments (create-dump single)

Run create-dump --help for the full list.

🏗️ Architecture

Directory Tree

src/create_dump/
├── cli/             # Entry points (main, single, batch)
├── scanning/        # Secret scanning & security
├── collector/       # File gathering (glob/git)
├── workflow/        # Processing pipelines
├── writing/         # Output generation (MD/JSON)
├── archive/         # Compression & retention
├── rollback/        # Restore functionality
└── core.py          # Config & Models

Data Flow

1. CLI Entry: User invokes create-dump (via Typer).
2. Config Load: Merges defaults, pyproject.toml, and CLI args.
3. Collector: Finds files via glob or git ls-files. Applies ignores.
4. Processor (Async):
   • Reads file content.
   • Middlewares: Secret Scan -> TODO Scan -> Language Detect.
5. Writer: Aggregates processed files into a Markdown/JSON artifact.
6. Post-Process:
   • Archiver: Rotates old dumps.
   • Notifier: Sends Slack/Discord alerts.

🐞 Troubleshooting

Debug Mode: Run with -v or --verbose to see detailed logs and stack traces.

create-dump single -v

🤝 Contributing

We love contributions! Please check our CONTRIBUTING.md for details.

Dev Setup:

1. Clone: git clone ...
2. Install: pip install -e .[dev]
3. Test: pytest
4. Lint: ruff check .

🗺️ Roadmap

• Smart Caching: Re-use processing for unchanged files.
• Rollback Command: Restore projects from dumps.
• Database Dumps: Postgres/MySQL integration.
• S3/Cloud Upload: Direct upload of dumps/archives.
• PDF Export: Generate PDF reports instead of Markdown.
• Plugin System: Allow custom middleware via Python entrypoints.