abx-dl 1.0.2

All-in-one CLI tool to download and extract content from URLs

⬇️ abx-dl

A simple all-in-one CLI tool to auto-detect and download everything available from a URL.

pip install abx-dl
abx-dl 'https://example.com/page/to/download'

---

✨ Ever wish you could yt-dlp, gallery-dl, wget, curl, puppeteer, etc. all in one command?

abx-dl is an all-in-one CLI tool for downloading URLs "by any means necessary".

It's useful for scraping, downloading, OSINT, digital preservation, and more. abx-dl provides a simpler one-shot CLI interface to the ArchiveBox plugin ecosystem.

---

🍜 What does it save?

abx-dl --plugins=title,favicon,headers,wget,singlefile,screenshot,pdf,dom,readability,git,... 'https://example.com'

abx-dl runs all plugins by default, or you can specify --plugins=... for specific methods:

• HTML, JS, CSS, images, etc. rendered with a headless browser
• title, favicon, headers, outlinks, and other metadata
• audio, video, subtitles, playlists, comments
• snapshot of the page as a PDF, screenshot, and Singlefile HTML
• article text, git source code
• and much more...

🧩 How does it work?

abx-dl uses the ABX Plugin Library (shared with ArchiveBox) to run a collection of downloading and scraping tools.

Plugins are discovered from the plugins/ directory and execute hooks in order:

1. Crawl hooks run first (setup/install dependencies like Chrome)
2. Snapshot hooks run per-URL to extract content

Each plugin can output:

• Files to its output directory
• JSONL records for status reporting
• Config updates that propagate to subsequent plugins

⚙️ Configuration

Configuration is handled via environment variables:

• CHROME_BINARY, WGET_BINARY, etc. - binary paths
• TIMEOUT=60 - default timeout for hooks
• {PLUGIN}_ENABLED=true/false - enable/disable specific plugins
• {PLUGIN}_TIMEOUT=120 - per-plugin timeout overrides

---

📦 Install

pip install abx-dl
abx-dl install           # optional: install plugin dependencies

🔠 Usage

# Basic usage - download URL with all plugins:
abx-dl 'https://example.com'

# Download with specific plugins only:
abx-dl --plugins=title,favicon,screenshot 'https://example.com'

# Specify output directory:
abx-dl --output=./downloads 'https://example.com'

# Set timeout:
abx-dl --timeout=120 'https://example.com'

Commands

abx-dl <url>                    # Download URL (default command)
abx-dl plugins                  # List available plugins
abx-dl info <plugin>            # Show plugin details
abx-dl install [plugins]        # Install plugin dependencies
abx-dl check [plugins]          # Check dependency status

Installing Dependencies

Many plugins require external binaries (e.g., wget, chrome, yt-dlp, single-file). Use abx-dl install to automatically install them:

# Install all plugin dependencies
abx-dl install

# Install dependencies for specific plugins only
abx-dl install wget,singlefile,ytdlp

# Check which dependencies are available/missing
abx-dl check

Dependencies are installed to ~/.config/abx/lib/{arch}/ using the appropriate package manager:

• pip packages → ~/.config/abx/lib/{arch}/pip/venv/
• npm packages → ~/.config/abx/lib/{arch}/npm/
• brew/apt packages → system locations

You can override the install location with LIB_DIR=/path/to/lib abx-dl install.

---

Output Structure

./
├── index.jsonl             # Snapshot metadata and results (JSONL format)
├── title/
│   └── title.txt
├── favicon/
│   └── favicon.ico
├── screenshot/
│   └── screenshot.png
├── pdf/
│   └── output.pdf
├── dom/
│   └── output.html
├── wget/
│   └── example.com/
│       └── index.html
├── singlefile/
│   └── output.html
└── ...

All Outputs

• index.jsonl - snapshot metadata and plugin results (JSONL format, ArchiveBox-compatible)
• title/title.txt - page title
• favicon/favicon.ico - site favicon
• screenshot/screenshot.png - full page screenshot (Chrome)
• pdf/output.pdf - page as PDF (Chrome)
• dom/output.html - rendered DOM (Chrome)
• wget/example.com/... - mirrored site files
• singlefile/output.html - single-file HTML snapshot
• ... and more via plugin library ...

---

Architecture

abx-dl is built on these components:

• abx_dl/plugins.py - Plugin discovery from plugins/ directory
• abx_dl/executor.py - Hook execution engine with config propagation
• abx_dl/config.py - Environment variable configuration
• abx_dl/cli.py - Rich CLI with live progress display

Plugins are symlinked from ArchiveBox's plugin directory.

---

For more advanced use with collections, parallel downloading, a Web UI + REST API, etc. See: ArchiveBox/ArchiveBox