Geler
Help convert dynamic websites to static ones.
Install
=> dans un venv
pip install geler-CERTIC
=> de manière isolée et indépendante (mais disponible globalement)
pipx install geler-CERTIC
Usage
As a library in your own program:
import geler
freezer = geler.Freezer(
"https://acme.tld/",
"/save/to/path/",
thread_pool_size=10,
http_get_timeout=30,
skip_extensions=[".mp4", ".mp3"],
).freeze()
for err in result.http_errors:
print(f'{err.get("status_code")}: {err.get("url")}')
As a CLI tool:
$> geler --help
usage: geler [-h] [-t THREAD_POOL_SIZE] [--http-get-timeout HTTP_GET_TIMEOUT] [-s SKIP_EXTENSIONS] [-v] start-from-url save-to-path
positional arguments:
start-from-url -
save-to-path -
optional arguments:
-h, --help show this help message and exit
-t THREAD_POOL_SIZE, --thread-pool-size THREAD_POOL_SIZE
1
--http-get-timeout HTTP_GET_TIMEOUT
30
-s SKIP_EXTENSIONS, --skip-extensions SKIP_EXTENSIONS
-
-v, --verbose False
Thread pool size (--thread-pool-size) defaults to 1. Increase the number to have multiple downloads in parallel.
HTTP get timeout (--http-get-timeout) default to 30s. This includes the time needed to download the file. Increase the number to increase the timeout, or set it to 0 for no timeout.
List of skipped (--skip-extensions) is a comma-separated list of extensions that won't be downloaded.
Verbose mode (--verbose) will show downloaded URLs and HTTP errors.
Complete example:
geler --http-get-timeout 30 --thread-pool-size 10 --skip-extension ".mp4,.zip" https://acme.tld/ /path/to/local/dir
Why ?
For MaX and associated tools, we needed a lightweight, portable, pure Python solution to convert small dynamic websites to static ones.
Alternatives
This tool has a narrow scope, on purpose. Please turn to these solutions if you need more:
Known Limitations
- only works with HTTP GET
- does not submit forms (even with GET method)
- only considers URLs in
src,href, or URL-like JavaScript string literals inonclickattributes - only considers URLs with
httporhttpsschemes - only downloads what is in the same netloc (same domain, same port) as the start URL
- patches URLs in
*.html,*.cssand JavaScript module imports in*.jsfiles, but does not parse arbitrary JS URL construction patterns - does not throttle requests
- does not respect
robots.txtor<meta name="robots" content="noindex">or any other robot directives.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file geler_certic-0.5.4.tar.gz.
File metadata
- Download URL: geler_certic-0.5.4.tar.gz
- Upload date:
- Size: 16.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.9.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
214681721b6e6e4a08bc7e81234d552015132bc4ef67dea9a1aed6bb3f3354fa
|
|
| MD5 |
9fabfef7bbea15317899713ba771c593
|
|
| BLAKE2b-256 |
ad3fb07f94b7e17ce772c4ad8c02624d780a191d5f4832a1f6db02333809ee36
|
File details
Details for the file geler_certic-0.5.4-py3-none-any.whl.
File metadata
- Download URL: geler_certic-0.5.4-py3-none-any.whl
- Upload date:
- Size: 17.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.9.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8a666a6aaeee415f88b1ce1b3b9a04355237378e75025ac3f9169b53aeacc559
|
|
| MD5 |
3b8a8f686afb1d0a1846ee5046a0cd43
|
|
| BLAKE2b-256 |
e7a5d3013ffc50bb2814aaf7050a0bc795bedfb3924d68041d4bc1ca607df192
|
