tilecloud-chain 1.16.1

Tools to generate tiles from WMS or Mapnik, to S3, Berkeley DB, MBTiles, or local filesystem in WMTS layout using Amazon cloud services.

TileCloud Chain

The goal of TileCloud Chain is to provide tools around tile generation on a chain like:

Source: WMS, Mapnik.

Optionally using an SQS queue, AWS host, SNS topic.

Destination in WMTS layout, on S3, on Berkeley DB (bsddb), on MBTiles, or on local filesystem.

Features:

• Generate tiles.
• Drop empty tiles.
• Drop tiles outside a geometry or a bbox.
• Use MetaTiles.
• Generate the legend images.
• Generate GetCapabilities.
• Generate OpenLayers example page.
• Generate the Apache configuration.
• Obtain the hash of an empty tile.
• In the future, measure tile generation speed.
• Calculate cost and generation time.
• In the future, manage the AWS hosts that generate tiles.
• Delete empty tiles.
• Copy files between caches.
• Be able to use an SQS queue to dispatch the generation.
• Post processing the generated tiles.
• ...

Legacy features:

• bsddb support
• sqlite (mbtiles) support
• mapnik support (should be updated for Python3)
• mapcache (replaced by internal mapcache)

Get it

With Docker

# Login to docker hub
docker login
docker pull camptocamp/tilecloud-chain

# Initialize the project
docker run -ti \
     --volume .:/project \
     camptocamp/tilecloud-chain \
     pcreate -s tilecloud_chain .

# Run the commands
DOCKER_ADRS=`ifconfig docker0 | head -n 2 | tail -n 1 | awk -F : '{print $2}' | awk '{print $1}'`
docker run -ti \
     --volume .:/project \
     --add-host=db:${DOCKER_ADRS} \
     --add-host=mapserver:${DOCKER_ADRS} \
     --env=USER_NAME=`whoami` \
     --env=USER_ID=`id -u` \
     --env=GROUP_ID=`id -g` \
     --env=UMASK=`umask` \
     camptocamp/tilecloud-chain \
     run <a command>

To share the home folder you should add the arguments:

--volume=${HOME}:${HOME} \
--env=HOME=${HOME} \

The image also contains some tools needed to render OSM data like: fonts-dejavu, node-carto.

With pip

Requirements:

pg_config and a build environment.

Install:

pip install tilecloud-chain
pcreate -s tilecloud_chain .

Edit your layers configuration in ./tilegeneration/config.yaml.

Default configuration file.

Support

Only the latest release is supported and version < 1.11 contains security issues.

From sources

Build it:

git submodule update --recursive
python3 -m venv .build/venv
.build/venv/bin/pip install -r requirements.txt
.build/venv/bin/pip install -e .
.build/venv/bin/pip install -r dev-requirements.txt

Run prospector

.build/venv/bin/prospector

Run the tests

Setup your environment:

touch tilecloud_chain/OpenLayers.js
docker build --tag camptocamp/tilecloud-chain .
docker-compose -p tilecloud up

To run the tests:

docker-compose -p tilecloud exec test python setup.py nosetests --logging-filter=tilecloud,tilecloud_chain --attr '!'nopy3

Documentation

As documentation you can read the https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst.

VSCode

You can add that in your workspace configuration to use the JSON schema:

.. code:: json

{
    "yaml.schemas": {
        "../tilecloud-chain/tilecloud_chain/schema.json": [
            "tilecloud-chain/tilecloud_chain/tests/tilegeneration/*.yaml"
        ]
    }
}

Changelog

Release 1.16

1. Change the config validator who is a little bit more strict.

Release 1.4

1. Add optional metadata section to the config file. See the scaffolds for example.

Release 0.9

1. Correct some error with slash.
2. Better error handling.
3. Be able to have one error file per layer.

Release 0.8

1. Correct some error with slash.
2. Add pre_hash_post_process and post_process.
3. Add copy command.

Release 0.7

1. Support of deferent geoms per layers, requires configuration changes, old version:

   connection: user=www-data password=www-data dbname=<db> host=localhost
   sql: <column> AS geom FROM <table>
   

   to new version:

   connection: user=www-data password=www-data dbname=<db> host=localhost
   geoms:
   -   sql: <column> AS geom FROM <table>
   

   More information in the Configure geom/sql chapter.

2. Update from optparse to argparse, and some argument refactoring, use --help to see the new version.

3. Add support of Blackbery DB (bsddb).

4. The tile server is completely rewrite, now it support all cache, REST and KVP interface, GetFeatureInfo request, and it can be used as a pyramid view or as a WSGI server. More information in the istribute the tiles chapter.

5. Add three strategy to bypass the proxy/cache: Use the headers Cache-Control: no-cache, no-store, Pragma: no-cache (default). Use localhost in the URL and the header Host: <host_name> (recommended). Add a SALT random argument (if the above don't work). More information in the Proxy/cache issue chapter.

6. Improve the dimensions usage by adding it ti the WMS requests, And add a --dimensions argument of generate_tiles to change the dimensions values.

7. Extract generate_cost and generate_amazon from generate_controler.

8. Now we can creates legends, see the Legends chapter.

9. Now the tiles generation display generation statistics at the ends.

10. The EC2 configuration is moved in a separate structure, see README for more information.

Release 0.6

1. Now the apache configuration can be generated with .build/venv/bin/generate_controller --generate-apache-config, it support filesystem cache and MapCache.
2. Windows fixes.
3. Use console rewrite (r) to log generated tiles coordinates.
4. Now if no layers is specified in generation:default_layers we generate all layers by default.
5. Now bbox to be floats.
6. New --get-bbox option to get the bbox of a tile.
7. Add coveralls support (https://coveralls.io/r/camptocamp/tilecloud-chain).
8. Add an config option generation:error_file and a command option --tiles to store and regenerate errored tiles.

Release 0.5

1. SQS config change:

layers:
    layer_name:
        sqs:
            # The region where the SQS queue is
            region: eu-west-1
            # The SQS queue name, it should already exists
            queue: the_name

2. Add debug option (--debug), please use it to report issue.
3. Now the sql request can return a set of geometries in a column names geom but the syntax change a little bit => <column> AS geom FROM <table>