Skip to main content

Crawl rendered JavaScript templates from a local server.

Project description

Command line tool that crawls a local webserver with a headless browser and outputs static html files. Works well with html5validator to validate HTML5 from dynamic content with Javascript.
https://travis-ci.org/svenkreiss/localcrawl.svg?branch=master

Run localcrawl --help:

https://raw.githubusercontent.com/svenkreiss/localcrawl/master/docs/help.png

PhantomJS is required. It is pre-installed on TravisCI. On a Mac run brew install PhantomJS.

Example Command

localcrawl --start _build/html/index.html --out _crawled/ --depth 3

Mustache Example

This can be used to convert templated files to HTML files (e.g. for validation with html5validator).

Input:

<html>
<head>
  <title>Mustache Test</title>
</head>
<body>
  <div id="output"></div>

  <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mustache.js/2.2.1/mustache.min.js"></script>
  <script>
    var data = {
      item: 'Fork',
      price: function() { return (1.10 * 1.08).toFixed(2); },
    };
    var html = Mustache.render('{{item}}: <b>${{price}}</b>', data);
    document.getElementById('output').innerHTML = html;
  </script>
</body>
</html>

The crawled output includes the output from processing the template (Fork: <b>$1.19</b>):

<html><head>
  <title>Mustache Test</title>
</head>
<body>
  <div id="output">Fork: <b>$1.19</b></div>

  <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mustache.js/2.2.1/mustache.min.js"></script>
  <script>
    var data = {
      item: 'Fork',
      price: function() { return (1.10 * 1.08).toFixed(2); },
    };
    var html = Mustache.render('{{item}}: <b>${{price}}</b>', data);
    document.getElementById('output').innerHTML = html;
  </script>


</body></html>

Should play nice with:

JavaScript template engines / JS frameworks:

Static site generators:

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
localcrawl-0.2.3-py2.py3-none-any.whl (6.0 kB) Copy SHA256 hash SHA256 Wheel 3.6 May 20, 2018
localcrawl-0.2.3.tar.gz (5.2 kB) Copy SHA256 hash SHA256 Source None May 20, 2018

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page