Server to on-the-fly extract and serve vector tiles from an mbtiles file on S3

These details have not been verified by PyPI

Project links

Homepage

Project description

mbtiles-s3-server

Python server to on-the-fly extract and serve vector tiles from mbtiles files on S3. Javascript, maps styles, fonts, and sprites are included so you can get setup quickly, especially with OpenMapTiles mbtiles files, but these are not required to be used. Versioning must be enabled on the underlying S3 bucket.

https://user-images.githubusercontent.com/13877/178137820-9734a903-6efb-4b1a-a7f0-44e59a290176.mov

Video showing expected performance. The server hosting the HTML is running locally, but mbtiles-s3-server is running on AWS, communicating with a real S3 bucket.

Installation

pip install mbtiles-s3-server

The libsqlite3 binary library is also required, but this is typically already installed on most systems. The earliest version of libsqlite3 known to work is 2012-12-12 (3.7.15).

Example usage

Create or obtain an mbtiles file, for example from https://openmaptiles.org/.
(Optional) Create another file with a page size of 65536 bytes (64KiB) using VACUUM or VACUUM INTO
```
sqlite3 my-map.mbtiles "PRAGMA page_size=65536; VACUUM INTO 'mytiles-65536.mbtiles';"
```
While this step is optional, performance with default mbtiles files that have smaller page sizes can be horrible to the point of being unusable - loading of a single tile can take many seconds. Performance of this server is limited by the latency of calls to S3, and this step effectively reduces the number of calls to S3 per map tile.

Note both VACUUM and VACUUM INTO need disk space since they create another database which is approximately the size of the original file.
Upload this file to S3, for example to https://my-bucket.s3.eu-west-2.amazonaws.com/mytiles-65536.mbtiles

Ensure to have a IAM user that has s3:GetObject and s3:GetObjectVersion permissions on this S3 object, for example

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket/mytiles-65536.mbtiles"
            ]
        }
    ]
}

Start this server, configured with the location of this object and credentials for this user - it's configured using environment variables. You can assign the tiles file any version you like, in this case, 1.0.0.

PORT=8080 \
MBTILES__1__URL=https://my-bucket.s3.eu-west-2.amazonaws.com/mytiles-65536.mbtiles \
MBTILES__1__MIN_ZOOM=0 \
MBTILES__1__MAX_ZOOM=14 \
MBTILES__1__IDENTIFIER=mytiles \
MBTILES__1__VERSION=1.0.0 \
AWS_REGION=eu-west-2 \
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE \
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
HTTP_ACCESS_CONTROL_ALLOW_ORIGIN="*" \
    python -m mbtiles_s3_server

The above is only to show what environment variables are needed - usually credentials should be avoided in shell commands. When developing locally, instead of specifying the AWS_* variables, consider using aws-vault, which populates them for you.

For example if you have a local AWS profile called tiles, you can run the below.

PORT=8080 \
MBTILES__1__URL=https://my-bucket.s3.eu-west-2.amazonaws.com/mytiles-65536.mbtiles \
MBTILES__1__MIN_ZOOM=0 \
MBTILES__1__MAX_ZOOM=14 \
MBTILES__1__IDENTIFIER=mytiles \
MBTILES__1__VERSION=1.0.0 \
HTTP_ACCESS_CONTROL_ALLOW_ORIGIN="*" \
    aws-vault exec tiles -- python -m mbtiles_s3_server

On your user-facing site, include HTML that loads these tiles from this server, for example to load maps from a server started as above running locally serving OpenMapTiles

 <!DOCTYPE html>
 <html>
   <head>
     <meta charset="utf-8">
     <title>Example map</title>
     <meta name="viewport" content="initial-scale=1,maximum-scale=1,user-scalable=no">
     <script src="http://localhost:8080/v1/static/maplibre-gl@2.1.9/maplibre-gl.js"></script>
     <link href="http://localhost:8080/v1/static/maplibre-gl@2.1.9/maplibre-gl.css" rel="stylesheet">
     <style>
       body, html, #map {margin: 0; padding: 0; height: 100%; width: 100%}
     </style>
   </head>
   <body>
     <div id="map"></div>
     <script>
     var map = new maplibregl.Map({
         container: 'map',
         style: 'http://localhost:8080/v1/styles/osm-bright-gl-style@1.0.0/style.json?fonts=fonts-gl@1.0.0&tiles=mytiles@1.0.0',
         center: [0, 0],
         zoom: 1,
         attributionControl: false
     });
     map.addControl(new maplibregl.AttributionControl({
         customAttribution: '<a href="https://openmaptiles.org/">© OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright">© OpenStreetMap contributors</a>'
     }), 'bottom-right');
     </script>
   </body>
 </html>

This HTML is included in this repository in example.html. A simple server can be started to view it by

python -m http.server 8081 --bind 127.0.0.1

and going to http://localhost:8081/example.html

For the curious, advanced, or developers of this server itself

Hosting your own vector map tiles to show them in a browser requires quite a few components:

JavaScript and CSS

A Javascript and CSS library, such as MapLibre GL, and your own code to run this library, pointing it to a style file
Style file

A JSON file that defines how the library should visually style the map data, and where it should find the map tiles, glyphs (fonts), and the sprite. This server transforms the built-in Style files on the fly to be able to refer to any map data.
Glyphs (fonts)

Different fonts can be used for different labels and zoom levels, as defined in the Style file. The fonts must Signed Distance Field (SDF) fonts wrapped in a particular Protocol Buffer format. The Style file can refer to "stacks" of fonts; but unlike CSS, the server combines the fonts on the fly in an API where the resulting "font" has at most one glyph from each source font.
Sprite

A sprite is actually 4 URLs: a JSON index file and a single PNG file, and a "@2x" JSON index file and PNG files for higher pixel ratio devices (e.g. Retina). The JSON files contains the offsets and sizes of images within corresponding PNG file. The style file refers the common "base" of these. For example, if the style file has "sprite":"https://my.test/sprite" then the 4 files must be at https://my.test/sprite.json, https://my.test/sprite.png, https://my.test/sprite@2x.json and https://my.test/sprite@2x.png.
Vector map tiles

A set of often millions of tiles each covering a different location and different zoom level. These can be distributed as a single mbtiles file, but this is not the format that the Javascript library accepts. This on-the-fly conversion from the mbtiles file to tiles is the main feature of this server.

The mbtiles file is a SQLite file, containing gzipped Mapbox Vector Tile tiles. This server leaves the un-gzipping to the browser, by sending tiles with a content-encoding: gzip header, which results in browser un-gzipping the tile data before it hits the Javascript.

Licenses

The code of the server itself is released under the MIT license. However, several components included in mbtiles_s3_server/vendor/ are released under different licenses.

Project details

These details have not been verified by PyPI

Project links

Homepage

Release history Release notifications | RSS feed

0.0.20

Jul 17, 2022

This version

0.0.19

Jul 17, 2022

0.0.18

Jul 17, 2022

0.0.17

Jul 16, 2022

0.0.16

Jul 16, 2022

0.0.15

Jul 15, 2022

0.0.14

Jul 11, 2022

0.0.13

Jul 11, 2022

0.0.12

Jul 11, 2022

0.0.11

Jul 10, 2022

0.0.10

Jul 9, 2022

0.0.9

Jul 9, 2022

0.0.8

Jul 9, 2022

0.0.7

Jul 9, 2022

0.0.6

Jul 9, 2022

0.0.5

Jul 9, 2022

0.0.4

Jul 9, 2022

0.0.3

Jul 9, 2022

0.0.2

Jul 8, 2022

0.0.1

Jul 7, 2022

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mbtiles-s3-server-0.0.19.tar.gz (67.8 MB view details)

Uploaded Jul 17, 2022 Source

Built Distribution

mbtiles_s3_server-0.0.19-py3-none-any.whl (67.8 MB view details)

Uploaded Jul 17, 2022 Python 3

File details

Details for the file mbtiles-s3-server-0.0.19.tar.gz.

File metadata

Download URL: mbtiles-s3-server-0.0.19.tar.gz
Upload date: Jul 17, 2022
Size: 67.8 MB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.25.1 requests-toolbelt/0.9.1 urllib3/1.26.5 tqdm/4.62.3 importlib-metadata/4.12.0 keyring/23.5.0 rfc3986/1.5.0 colorama/0.4.3 CPython/3.9.7

File hashes

Hashes for mbtiles-s3-server-0.0.19.tar.gz
Algorithm	Hash digest
SHA256	`b21681e0e05aa0c20666cfa05d04aa39560b429e2cd12e09924644f56d40219c`
MD5	`a7d7642b74a277abbc4576cdf466226c`
BLAKE2b-256	`a38dada9e4718b7c01a948eca5d89bc9364d812e12de401b0fcfeb15610a42fd`

See more details on using hashes here.

File details

Details for the file mbtiles_s3_server-0.0.19-py3-none-any.whl.

File metadata

Download URL: mbtiles_s3_server-0.0.19-py3-none-any.whl
Upload date: Jul 17, 2022
Size: 67.8 MB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.25.1 requests-toolbelt/0.9.1 urllib3/1.26.5 tqdm/4.62.3 importlib-metadata/4.12.0 keyring/23.5.0 rfc3986/1.5.0 colorama/0.4.3 CPython/3.9.7

File hashes

Hashes for mbtiles_s3_server-0.0.19-py3-none-any.whl
Algorithm	Hash digest
SHA256	`5397c3e5dd9e57d0d4655103ae969eab6f6960282b67e54f6b97ca485b16e3b8`
MD5	`96a4fec56e543503c16e15f7f48c637e`
BLAKE2b-256	`e13a9467562054e669606e7fb906a65056620298d48eb30738005da8bb87af48`