pywget
A robust, resumable, multi-connection command-line downloader and Python library.
A drop-in-style wget alternative with zero third-party dependencies.
Features
| Area | What you get |
|---|---|
| Speed | Parallel segmented download (-n 1..32), automatic single-stream fallback |
| Reliability | Crash-safe resume, retries with exponential backoff + jitter, Retry-After, atomic rename |
| Integrity | --checksum sha256:..., size verification, disk-space pre-check |
| Security | http/https only, credentials stripped on cross-origin redirects, path-traversal-proof filenames, TLS verification on by default |
| Control | Bandwidth limit, custom headers, basic auth, proxy, custom CA, timeouts |
| Automation | Batch input (-i), --json output, stable exit codes, SIGINT/SIGTERM-safe |
| Server mode | pywget serve: persistent queue, HTTP/JSON API, SSE events, web UI, token auth |
| Agent mode | pywget agent / --server: send links from anywhere; offline-tolerant relay to a server |
| Library | Typed API, progress callbacks, cancel support, legacy wget.download() kept |
| Portability | Python 3.8+, Windows / Linux / macOS, no dependencies |
Installation
pip install pywget
v2 note: Python 2 is no longer supported. Python 3.8+ is required. The CLI and
wget.download(url, dir)keep working as before.
Quick start
# simple download into a directory (same as before)
pywget "https://example.com/file.iso" -p ~/Downloads
# 8 parallel connections, 2 MiB/s cap, verify checksum
pywget https://example.com/file.iso -n 8 --limit-rate 2M --checksum sha256:<hex>
# many URLs from a file (or stdin with -i -), JSON report
pywget -i urls.txt -p downloads --json
# auth + custom header
pywget https://api.example.com/export.zip --user me -H "Accept: application/zip"
# download manager mode: run a server, push links to it (see "Server mode and agent")
pywget serve --root ~/Downloads/pywget &
pywget send https://example.com/file.iso -r https://example.com/ --wait
pywget jobs list
Interrupted (Ctrl+C, crash, network loss)? Run the same command again - it resumes.
CLI reference
| Option | Description |
|---|---|
URL ... |
http(s) URLs, c = read links from the clipboard, or a text file of links |
-i, --input-file FILE |
Read URLs from file (- = stdin); # comments allowed |
-p, -P, --path DIR |
Download directory (default .) |
-O, --output NAME |
Output file name (single URL) |
-n, --connections N |
Parallel connections per file, 1-32 (default 4) |
-t, --retries N |
Retries per request/segment (default 5) |
-r, --referrer URL |
Referer header (many sites require it) |
-C, --cookie STR |
Cookie string a=b; c=d or JSON object |
-T, --timeout SEC |
Network timeout (default 30) |
--limit-rate RATE |
Total speed cap, e.g. 500k, 2M |
-H, --header 'K: V' |
Extra header (repeatable) |
-U, -ua, --user-agent UA |
Custom User-Agent |
--user, --password |
Basic auth (password also from PYWGET_PASSWORD or prompt) |
--proxy URL |
Proxy (default: environment) |
--no-check-certificate |
Disable TLS verification (not recommended) |
--ca-cert FILE |
Custom CA bundle |
--checksum ALGO:HEX |
Verify result (md5, sha1, sha256, sha512, ...) |
--overwrite |
Replace existing file and restart |
--no-resume |
Never resume partial files |
--no-mtime |
Do not copy Last-Modified to the file |
-q, -v |
Quiet / debug logging |
--json |
Machine-readable results on stdout |
-c |
Accepted for wget compatibility (resume is default) |
-s, --server URL |
Send the links to a pywget server/agent instead of downloading locally |
--token TOKEN |
API token for --server (default: $PYWGET_TOKEN or token file) |
-a, --add-only |
With --server: queue paused, do not start (like IDM's add) |
--wait |
With --server: wait for the jobs to finish |
Exit codes
| Code | Meaning |
|---|---|
| 0 | Success |
| 1 | Unexpected error |
| 2 | Bad usage / invalid URL |
| 3 | Network error |
| 4 | HTTP error status (404, 403, ...) |
| 5 | Checksum mismatch |
| 6 | Storage / I/O error (disk full, permissions) |
| 7 | Server / agent error (unreachable, 401, 409, ...) |
| 130 | Interrupted (partial data kept for resume) |
Python API
import pywget
path = pywget.download(
"https://example.com/file.iso",
"/data/downloads",
progress=True, # terminal bar
connections=8,
limit_rate="5M",
checksum="sha256:ab12...",
)
print(path)
Full control:
from pywget import Downloader, Options, ProgressInfo
def on_progress(p: ProgressInfo) -> None:
print(f"{p.filename}: {p.percent or 0:.1f}% {p.speed/1e6:.1f} MB/s ETA {p.eta}")
dl = Downloader(Options(output_dir="out", retries=8, progress=on_progress,
headers={"Authorization": "Bearer TOKEN"}))
result = dl.fetch("https://example.com/big.bin")
print(result.path, result.size, result.resumed, result.connections)
# from another thread: dl.cancel() -> partial state is kept for resume
Talk to a server / agent, or embed one:
from pywget import ServerClient, ServerConfig, ServerHandle
client = ServerClient("http://127.0.0.1:8765") # token: $PYWGET_TOKEN or token file
job = client.submit({"url": "https://example.com/f.iso", "path": "isos"})[0]
print(client.wait(job["id"])["status"])
cfg = ServerConfig(port=0, token="secret-token", root="/srv/dl", state_dir="/var/lib/pywget")
with ServerHandle(cfg) as server: # background server (tests, embedding)
print(server.url)
Legacy API (unchanged):
from pywget import wget
wget.download("http://www.test.com/test.tar.gz", "/home/root/Downloads")
Errors derive from pywget.PywgetError (InvalidURLError, NetworkError,
HTTPStatusError, ChecksumError, StorageError, DownloadCancelled).
A Downloader runs one download at a time - use one instance per thread.
Server mode and agent
Run pywget as a long-lived download manager and push links to it from anywhere - the same idea as handing a link to IDM, but over a small authenticated HTTP API.
# 1. start the server (jobs may only write below --root)
pywget serve --root ~/Downloads/pywget # http://127.0.0.1:8765, token auto-created
# 2. send links (token is read from the token file on the same machine)
pywget send https://example.com/a.iso -p isos -r https://example.com/ --wait
pywget https://example.com/b.zip --server http://127.0.0.1:8765 # same, legacy form
pywget send c # links found in the clipboard
pywget send links.txt -a # a text file of links, queued paused (add only)
# 3. manage
pywget jobs list
pywget jobs pause|resume|cancel|retry|remove <id>
Open http://127.0.0.1:8765/ for the web UI (paste the token once).
Agent (relay)
An agent is a tiny server that does not download: it accepts links on a local port and forwards them to an upstream server, keeping them in a persistent outbox when the upstream is offline and mirroring progress back. Put it on the machine where links appear (laptop, browser helper script) and let a server/NAS do the downloading.
# on the download box
pywget serve --host 0.0.0.0 --root /srv/downloads --tls-cert c.pem --tls-key k.pem
# on your workstation
pywget agent --upstream https://nas:8765 --upstream-token <TOKEN> --port 8766
pywget send https://example.com/file.iso --server http://127.0.0.1:8766 --wait
System overview
flowchart LR
CLI["pywget send / --server"] --> AG
BR["Browser helper,<br/>scripts, curl"] --> AG
subgraph WS["Workstation"]
AG["Agent<br/>pywget agent<br/>persistent outbox"]
end
AG -- "HTTPS + token<br/>forward job, poll status" --> API
CLI -. "direct" .-> API
subgraph SV["Download server"]
API["HTTP API + web UI<br/>auth, Host check, root jail"] --> MGR["JobManager<br/>queue, N workers"]
MGR --> ENG["Downloader engine<br/>segments, resume, retry"]
MGR <--> ST[("jobs.json<br/>0600")]
ENG --> FS[("download root")]
end
Job flow
sequenceDiagram
autonumber
actor U as User
participant A as Agent
participant S as Server
participant W as Worker
U->>A: pywget send URL --wait
A->>A: persist job (outbox)
loop until accepted
A->>S: POST /api/v1/jobs (Bearer token)
S-->>A: 201 job id or error / offline then backoff
end
S->>W: enqueue
W->>W: Downloader.fetch (segments, resume)
loop every second
A->>S: GET /api/v1/jobs/id
S-->>A: status, percent, speed, ETA
end
W-->>S: completed (atomic rename in root)
A-->>U: OK path
Job states
stateDiagram-v2
[*] --> queued: add
[*] --> paused: add --add-only
queued --> running: worker free
running --> paused: pause (partial data kept)
paused --> queued: resume
running --> completed
running --> failed: error after retries
running --> cancelled: cancel (partial files removed)
queued --> cancelled
paused --> cancelled
failed --> queued: retry
cancelled --> queued: retry
running --> queued: server shutdown (auto-resumes on start)
completed --> [*]: remove
failed --> [*]: remove
cancelled --> [*]: remove
HTTP API
All /api/v1/* calls need Authorization: Bearer <token> (or X-Pywget-Token).
| Method and path | Purpose |
|---|---|
GET /health |
Liveness, version, mode (no auth) |
GET / |
Web UI (no auth, token entered in the page) |
POST /api/v1/jobs |
Create job(s): a job object, {"urls": [...], ...common} or {"jobs": [...]} |
GET /api/v1/jobs[?status=] |
List jobs |
GET /api/v1/jobs/{id} |
One job (secrets are never returned) |
POST /api/v1/jobs/{id}/pause|resume|cancel|retry |
Control a job |
DELETE /api/v1/jobs/{id} |
Remove a finished job record |
GET /api/v1/stats |
Counts, total speed |
GET /api/v1/events |
Server-sent events (event: job) |
Job fields: url, filename, path (folder below the root), referrer, cookie
(string or object), headers, user_agent, username, password, connections,
checksum, limit_rate, overwrite, add_only.
curl -s -H "Authorization: Bearer $(cat ~/.config/pywget/token)" \
-d '{"url":"https://example.com/f.iso","path":"isos","referrer":"https://example.com/"}' \
http://127.0.0.1:8765/api/v1/jobs
Security model
- Token required for every API call; generated once (
tokenfile, mode 0600) or given via--token/$PYWGET_TOKEN; compared in constant time. - Loopback by default. Binding elsewhere prints a warning; use
--tls-cert/--tls-keyor an HTTPS reverse proxy. - Root jail: job
pathcan never leave--root(absolute paths and..are rejected, symlinks resolved). - No CORS and a
Hostheader allow-list (--allowed-host) against DNS-rebinding and drive-by browser requests. --deny-private-networksrefuses URLs resolving to private/loopback/link-local addresses (SSRF guard for exposed servers).- Request bodies capped at 1 MiB; headers checked for CR/LF injection; credentials and cookies are stored in
jobs.json(0600) and never returned by the API. - Not supported: POST-body downloads (IDM
post data), FTP.
How it works
Download engine
flowchart LR
CLI["CLI<br/>pywget.cli"] --> DL
API["Python API<br/>pywget.download()"] --> DL
LEG["Legacy shim<br/>pywget.wget"] --> DL
JOB["LocalExecutor<br/>(server jobs)"] --> DL
subgraph Engine["pywget.core"]
DL["Downloader"] --> PROBE["Probe<br/>Range: bytes=0-0"]
PROBE --> PLAN{"Size known and<br/>ranges supported?"}
PLAN -- yes --> SEG["Segment workers<br/>1..N threads"]
PLAN -- no --> STR["Single stream<br/>worker"]
SEG --> MON["Monitor loop<br/>progress + state save"]
STR --> MON
MON --> VER["Verify<br/>size + checksum"]
end
SEG -. throttled by .-> RL["RateLimiter"]
STR -. throttled by .-> RL
MON --> TR["Tracker / ProgressBar"]
SEG <--> ST[("state file<br/>.pywget")]
VER --> OUT[("final file<br/>atomic rename")]
Download sequence
sequenceDiagram
autonumber
actor U as User
participant D as Downloader
participant S as Server
participant F as Disk
U->>D: fetch(url)
D->>S: GET Range: bytes=0-0
S-->>D: 206 + Content-Range (size, ETag)
D->>F: load .pywget state (if any)
alt state matches size / ETag / mtime
D->>D: resume remaining segments
else fresh start
D->>F: preallocate .part, split into N segments
end
par segment 1..N
D->>S: GET Range: bytes=a-b
S-->>D: 206 data
D->>F: write at offset
end
loop every second
D->>F: save state
D-->>U: progress callback
end
D->>F: verify size and checksum
D->>F: rename .part to final name
D-->>U: DownloadResult
Segment lifecycle and retries
stateDiagram-v2
[*] --> Probing
Probing --> Planning: size and ranges known
Probing --> Streaming: no range support
Planning --> Downloading
Streaming --> Verifying
Downloading --> Retrying: timeout / reset / 5xx / 429
Retrying --> Downloading: backoff + jitter, resume at offset
Retrying --> Failed: retries exhausted
Downloading --> Interrupted: Ctrl+C / SIGTERM / cancel()
Interrupted --> Planning: re-run, state validated
Downloading --> Verifying: all segments complete
Verifying --> Completed: size and checksum OK
Verifying --> Failed: mismatch, partial files removed
Completed --> [*]
Failed --> [*]
Resume decision
flowchart TD
A["Start"] --> B{".pywget state<br/>and .part exist?"}
B -- no --> N["Fresh download"]
B -- yes --> C{"URL, size, ETag,<br/>Last-Modified equal?"}
C -- no --> N
C -- yes --> D{"Segments valid and<br/>.part size correct?"}
D -- no --> N
D -- yes --> R["Resume each segment<br/>from its saved offset"]
Main types - engine
classDiagram
class Downloader {
+fetch(url) DownloadResult
+download(url) Path
+cancel()
+partial
}
class Options {
connections
retries
limit_rate
rate_limiter
checksum
progress
}
class DownloadResult {
path
size
resumed
connections
checksum
}
class Tracker
class ProgressBar
class RateLimiter
Downloader o-- Options
Downloader --> DownloadResult
Downloader --> Tracker
Downloader --> RateLimiter
ProgressBar ..> Tracker : renders snapshots
Main types - server and agent
classDiagram
class ServerHandle {
+start()
+close()
+url
}
class PywgetHTTPServer
class ApiHandler {
auth, Host check
routes /api/v1/*
}
class JobManager {
+add(spec) Job
+pause(id)
+resume(id)
+cancel(id)
+retry(id)
+remove(id)
}
class Job
class JobStore
class EventBus
class LocalExecutor
class RelayExecutor
class ServerClient
class ServerConfig
ServerHandle --> PywgetHTTPServer
ServerHandle --> JobManager
PywgetHTTPServer --> ApiHandler
ApiHandler --> JobManager
JobManager o-- Job
JobManager --> JobStore : persists
JobManager --> EventBus : SSE
JobManager --> LocalExecutor : server mode
JobManager --> RelayExecutor : agent mode
LocalExecutor --> Downloader
RelayExecutor --> ServerClient : forwards to upstream
ServerHandle o-- ServerConfig
Module map
| Module | Role |
|---|---|
pywget.core |
Download engine (Downloader, Options) |
pywget.progress, pywget.utils, pywget.errors |
Progress bar, helpers, exceptions and exit codes |
pywget.cli |
CLI: direct download, serve, agent, send, jobs |
pywget.server |
HTTP API, SSE, web UI hosting (ServerHandle, run_server) |
pywget.jobs |
Job, JobManager, persistence, LocalExecutor / RelayExecutor |
pywget.client |
ServerClient (used by send, jobs and the agent) |
pywget.config |
Token file, config dir, ServerConfig |
pywget.clipboard |
c input: clipboard and URL extraction |
pywget.webui |
Embedded single-page web UI |
pywget.wget |
Backward-compatible wget.download() |
Behaviour details
- Probing: one
Range: bytes=0-0request yields size, range support, ETag,Last-ModifiedandContent-Disposition. - Segments: a file is split into
min(connections, size / 2 MiB)parts; small files use one connection. - Existing file: same size means skipped; different size is saved as
name.1(wget style) unless--overwrite. - Partial files: stored as
name.part+name.pywget; removed on success. Changed remote file (size/ETag/mtime) restarts cleanly. - Redirects: only http/https;
Authorization/Cookieare dropped when the origin changes. - Server jobs: persisted in
jobs.json; a job running at shutdown is re-queued and resumes from its.partdata. - Not supported (by design): FTP, recursive mirroring, POST-body downloads.
Development
git clone https://github.com/cumulus13/pywget && cd pywget
pip install -e ".[dev]"
pytest --cov=pywget
ruff check pywget tests && mypy pywget
python -m build
License
Author
Metadata
Release files for pywget 2.0.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pywget-2.0.5.tar.gz | 56.1 kB | With provenance | Details for pywget-2.0.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pywget-2.0.5-py3-none-any.whl | Python 3 | none | any | Details for pywget-2.0.5-py3-none-any.whl |
Total release size: 105.9 kB
Release files / pywget-2.0.5.tar.gz
| Download URL | pywget-2.0.5.tar.gz |
|---|---|
| Size | 56.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c3aaadfcec9816ae76b9fb14c87eae49b98327a38fa60e0e4e901b2245bfe0fe
|
|
BLAKE2b-256 checksum How to use checksums |
833256954af6a061844ba78e2b165eb94a53a485fe0bcaf0d888a760a287f09c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 25, 2026.
Transparency logRelease files / pywget-2.0.5-py3-none-any.whl
| Download URL | pywget-2.0.5-py3-none-any.whl |
|---|---|
| Size | 49.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9fa7171029f5f6fcb13b252a33dbedf18b78ae39f07b6e73e818148e276dde74
|
|
BLAKE2b-256 checksum How to use checksums |
f51d87a6a1c24073c4ae1700355649647cc0b9dde1cbbbec79f031ccba3149ac
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 25, 2026.
Transparency logSecurity for pywget 2.0.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for pywget-2.0.5.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 25, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.