Skip to main content

tile-based geodata processing

Project description

Tile-based geodata processing.

https://badge.fury.io/py/mapchete.svg https://travis-ci.org/ungarj/mapchete.svg?branch=master https://coveralls.io/repos/github/ungarj/mapchete/badge.svg?branch=master Documentation Status https://img.shields.io/pypi/pyversions/mapchete.svg

Developing a script which does some geoprocessing is usually an iterative process where modifying code, running the script and inspecting the output repeat until the desired result. This can take a long time as processing and visualizing the output data repeat very often and therefore sum up. Especially when using a remote machine because the input data is huge, the time to wait for the script to finish, download and open the output can be tedious.

Mapchete aims to facilitate this development circle by providing tools to quickly inspect the output (from a remote or local machine) and allows larger scale processing jobs by running multiple tiles in parallel.

Python is used a lot because it is a very user-friendly language to quickly develop working processing chains and it provides a rich ecosystem of packages which help to efficiently process geodata (e.g. shapely for features, numpy for rasters).

Mapchete takes care about dissecting, resampling and reprojecting geodata, applying user defined Python code to each tile and writing the output into a WMTS-like tile pyramid which is already optimized to be further used for web maps.

Usage

You need a .mapchete file for the process configuration

process: my_python_process.py  # or a Python module path: mypythonpackage.myprocess
zoom_levels:
    min: 0
    max: 12
input:
    dem: /path/to/dem.tif
    land_polygons: /path/to/polygon/file.geojson
output:
    format: PNG_hillshade
    path: /output/path
pyramid:
    grid: mercator

# process specific parameters
resampling: cubic_spline

and a .py file or a Python module path where you specify the process itself

def execute(mp, resampling="nearest"):
    # Open elevation model.
    with mp.open("dem", resampling=resampling) as src:
        # Skip tile if there is no data available.
        if src.is_empty(1):
            return "empty"
        dem = src.read(1)
    # Create hillshade.
    hillshade = mp.hillshade(dem)
    # Clip with polygons and return result.
    with mp.open("land_polygons") as land_file:
        return mp.clip(hillshade, land_file.read())

You can then interactively inspect the process output directly on a map in a browser (first, install dependencies by pip install mapchete[serve] go to localhost:5000):

mapchete serve hillshade.mapchete --memory

The serve tool recognizes changes in your process configuration or in the process file. If you edit one of these, just refresh the browser and inspect the changes (note: use the --memory flag to make sure to reprocess each tile and turn off browser caching).

Once you are done with editing, batch process everything using the execute tool.

mapchete execute hillshade.mapchete

There are many more options such as zoom-dependent process parameters, metatiling, tile buffers or interpolating from an existing output of a higher zoom level. For deeper insights, please go to the documentation.

Mapchete is used in many preprocessing steps for the EOX Maps layers:

  • Merge multiple DEMs into one global DEM.
  • Create a customized relief shade for the Terrain Layer.
  • Generalize landmasks & coastline from OSM for multiple zoom levels.
  • Extract cloudless pixel for Sentinel-2 cloudless.

Installation

via PyPi:

pip install mapchete

from source:

pip install -r requirements.txt
python setup.py install

To make sure Rasterio and Fiona are properly built against your local GDAL installation, don’t install the binaries but build them on your system:

pip install "rasterio>=1.0.2" "fiona>=1.8b1" --no-binary :all:

To keep the core dependencies minimal if you install mapchete using pip, some features are only available if you manually install additional dependencies:

# for contour extraction:
pip install mapchete[contours]

# for S3 bucket reading and writing:
pip install mapchete[s3]

# for mapchete serve:
pip install mapchete[serve]

# for VRT generation:
pip install mapchete[vrt]

License

MIT License

Copyright (c) 2015 - 2019 EOX IT Services

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
mapchete-0.27-py3-none-any.whl (95.0 kB) Copy SHA256 hash SHA256 Wheel py3
mapchete-0.27.tar.gz (88.2 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page