Skip to main content

Command line tool that crawls a local webserver with a headless browser and outputs static html files. Works well with html5validator to validate HTML5 from dynamic content with Javascript.

https://travis-ci.org/svenkreiss/localcrawl.svg?branch=master

Run localcrawl --help:

https://raw.githubusercontent.com/svenkreiss/localcrawl/master/docs/help.png

PhantomJS is required. It is pre-installed on TravisCI. On a Mac run brew install PhantomJS.

Example Command

localcrawl --start _build/html/index.html --out _crawled/ --depth 3

Mustache Example

This can be used to convert templated files to HTML files (e.g. for validation with html5validator).

Input:

<html>
<head>
  <title>Mustache Test</title>
</head>
<body>
  <div id="output"></div>

  <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mustache.js/2.2.1/mustache.min.js"></script>
  <script>
    var data = {
      item: 'Fork',
      price: function() { return (1.10 * 1.08).toFixed(2); },
    };
    var html = Mustache.render('{{item}}: <b>${{price}}</b>', data);
    document.getElementById('output').innerHTML = html;
  </script>
</body>
</html>

The crawled output includes the output from processing the template (Fork: <b>$1.19</b>):

<html><head>
  <title>Mustache Test</title>
</head>
<body>
  <div id="output">Fork: <b>$1.19</b></div>

  <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mustache.js/2.2.1/mustache.min.js"></script>
  <script>
    var data = {
      item: 'Fork',
      price: function() { return (1.10 * 1.08).toFixed(2); },
    };
    var html = Mustache.render('{{item}}: <b>${{price}}</b>', data);
    document.getElementById('output').innerHTML = html;
  </script>


</body></html>

Should play nice with:

JavaScript template engines / JS frameworks:

Static site generators:

Metadata

Release files for localcrawl 0.2.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for localcrawl 0.2.3
File Size Uploaded
localcrawl-0.2.3.tar.gz 5.2 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for localcrawl 0.2.3
File Interpreter ABI Platform
localcrawl-0.2.3-py2.py3-none-any.whl Python 3, Python 2 none any Details

Total release size: 11.2 kB

Release files / localcrawl-0.2.3.tar.gz

Download URL localcrawl-0.2.3.tar.gz
Size 5.2 kB
Tags Source
SHA-256 checksum
How to use checksums
9b222f2ee69860a7b4c96ecb7f98aaf71d24cee55d1441b737c59a5811dc17a5
BLAKE2b-256 checksum
How to use checksums
1a9653ac58886509486ea2181677d33f809e0a2b5644b5a59bb5f80cf3e58631
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release files / localcrawl-0.2.3-py2.py3-none-any.whl

Download URL localcrawl-0.2.3-py2.py3-none-any.whl
Size 6.0 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
2f7c629bc110bf3f3b0c8773ab87f8c11c5adc85bcb2d2a1000a1aad382f4c7f
BLAKE2b-256 checksum
How to use checksums
5a94dab8a257e468a5f91b5bc3ade9cd92d6c5c33c97989a3643f2213ff3b16e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release history Release notifications | RSS feed

This release

0.2.3 This release

2 release files

0.2.2

2 release files

0.2.1

2 release files

0.2.0

2 release files

0.1.3

2 release files

0.1.2

2 release files

0.1.1

2 release files

0.1.0

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page