geler-CERTIC 0.1.5

Help convert dynamic webistes to static ones.

Geler

Help convert dynamic websites to static ones.

Install

pip install geler-CERTIC

Usage

As a library in your own program:

from geler import freeze
freeze("https://acme.tld/", "/path/to/local/dir/", thread_pool_size=1, http_get_timeout=30)

As a CLI tool:

$> geler --help
usage: geler [-h] [-t THREAD_POOL_SIZE] [--http-get-timeout HTTP_GET_TIMEOUT] start-from-url save-to-path

positional arguments:
  start-from-url        -
  save-to-path          -

optional arguments:
  -h, --help            show this help message and exit
  -t THREAD_POOL_SIZE, --thread-pool-size THREAD_POOL_SIZE
                        1
  --http-get-timeout HTTP_GET_TIMEOUT
                        30

Why ?

For MaX and associated tools, we needed a lightweight, portable, pure Python solution to convert small dynamic websites to static ones.

Alternatives

This tool has a narrow scope, on purpose. Please turn to these solutions if you need more:

• wget
• pywebcopy
• HTTrack

Limitations

• only works with HTTP GET
• does not submit forms (even with GET method)
• only considers URLs in src or href attributes
• only considers URLs with http or https schemes
• only downloads what is in the same netloc (same domain, same port) as the start URL
• only patches URLs in *.html files, not in *.js, not in *.css (watch out for those url(...) and those @import in your CSS)
• does not throttle requests
• does not respect robots.txt