Skip to main content

Inspect HEAD/LIST/GET requests withing Rasterio

Project description

tilebench

tilebench

Inspect HEAD/LIST/GET requests within Rasterio

Test Coverage Package version Downloads Downloads


Source Code: https://github.com/developmentseed/tilebench


Inspect HEAD/GET requests withing Rasterio.

Note: In GDAL 3.2, logging capabilities for /vsicurl, /vsis3 and the like was added (ref: https://github.com/OSGeo/gdal/pull/2742).

Install

You can install tilebench using pip

$ python -m pip install -U pip
$ python -m pip install -U tilebench

or install from source:

git clone https://github.com/developmentseed/tilebench.git
cd tilebench

python -m pip install -U pip
python -m pip install -e .

API

from tilebench import profile
import rasterio

[@profile](https://github.com/profile)()
def info(src_path: str):
    with rasterio.open(src_path) as src_dst:
        return src_dst.meta

meta = info("https://noaa-eri-pds.s3.amazonaws.com/2022_Hurricane_Ian/20221002a_RGB/20221002aC0795145w325100n.tif")

> 2023-10-18T23:00:11.184745+0200 | TILEBENCH | {"HEAD": {"count": 1}, "GET": {"count": 1, "bytes": 32768, "ranges": ["0-32767"]}, "Timing": 0.7379939556121826}
from tilebench import profile
from rio_tiler.io import Reader

[@profile](https://github.com/profile)()
def _read_tile(src_path: str, x: int, y: int, z: int, tilesize: int = 256):
    with Reader(src_path) as cog:
        return cog.tile(x, y, z, tilesize=tilesize)

img = _read_tile(
    "https://noaa-eri-pds.s3.amazonaws.com/2022_Hurricane_Ian/20221002a_RGB/20221002aC0795145w325100n.tif",
    9114,
    13216,
    15,
)

> 2023-10-18T23:01:00.572263+0200 | TILEBENCH | {"HEAD": {"count": 1}, "GET": {"count": 2, "bytes": 409600, "ranges": ["0-32767", "32768-409599"]}, "Timing": 1.0749869346618652}

Command Line Interface (CLI)

$ tilebench --help
Usage: tilebench [OPTIONS] COMMAND [ARGS]...

  Command line interface for the tilebench Python package.

Options:
  --help  Show this message and exit.

Commands:
  get-zooms  Get Mercator Zoom levels.
  profile    Profile COGReader Mercator Tile read.
  random     Get random tile.
  viz        WEB UI to visualize VSI statistics for a web mercator tile request

Examples

$ tilebench get-zooms https://noaa-eri-pds.s3.amazonaws.com/2022_Hurricane_Ian/20221002a_RGB/20221002aC0795145w325100n.tif | jq
{
  "minzoom": 14,
  "maxzoom": 19
}

$ tilebench random https://noaa-eri-pds.s3.amazonaws.com/2022_Hurricane_Ian/20221002a_RGB/20221002aC0795145w325100n.tif --zoom 15
15-9114-13215

$ tilebench profile https://noaa-eri-pds.s3.amazonaws.com/2022_Hurricane_Ian/20221002a_RGB/20221002aC0795145w325100n.tif --tile 15-9114-13215 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR | jq
{
  "HEAD": {
    "count": 1
  },
  "GET": {
    "count": 2,
    "bytes": 409600,
    "ranges": [
      "0-32767",
      "32768-409599"
    ]
  },
  "Timing": 0.9715230464935303
}

$ tilebench profile https://noaa-eri-pds.s3.amazonaws.com/2022_Hurricane_Ian/20221002a_RGB/20221002aC0795145w325100n.tif --tile 15-9114-13215 --config GDAL_DISABLE_READDIR_ON_OPEN=FALSE | jq
{
  "HEAD": {
    "count": 8
  },
  "GET": {
    "count": 3,
    "bytes": 409600,
    "ranges": [
      "0-32767",
      "32768-409599"
    ]
  },
  "Timing": 2.1837549209594727
}

Starlette Middleware

Warning: This is highly experimental and should not be used in production (https://github.com/developmentseed/tilebench/issues/6)

In addition of the viz CLI we added a starlette middleware to easily integrate VSI statistics in your web services.

from fastapi import FastAPI

from tilebench.middleware import VSIStatsMiddleware

app = FastAPI()
app.add_middleware(VSIStatsMiddleware)

The middleware will add a vsi-stats entry in the response headers in form of:

vsi-stats: list;count=1, head;count=1, get;count=2;size=196608, ranges; values=0-65535|65536-196607

Some paths may be excluded from being handeld by the middleware by the exclude_paths argument:

app.add_middleware(VSIStatsMiddleware, exclude_paths=["/foo", "/bar"])

GDAL config options

  • CPL_TIMESTAMP: Add timings on GDAL Logs
  • GDAL_DISABLE_READDIR_ON_OPEN: Allow or Disable listing of files in the directory (e.g external overview)
  • GDAL_INGESTED_BYTES_AT_OPEN: Control how many bytes GDAL will ingest when opening a dataset (useful when a file has a big header)
  • CPL_VSIL_CURL_ALLOWED_EXTENSIONS: Limit valid external files
  • GDAL_CACHEMAX: Cache size
  • GDAL_HTTP_MERGE_CONSECUTIVE_RANGES
  • VSI_CACHE
  • VSI_CACHE_SIZE

See the full list at https://gdal.org/user/configoptions.html

Internal tiles Vs Mercator grid

$ tilebench viz https://noaa-eri-pds.s3.amazonaws.com/2022_Hurricane_Ian/20221002a_RGB/20221002aC0795145w325100n.tif --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR

Blue lines represent the mercator grid for a specific zoom level and the red lines represent the internal tiles bounds

We can then click on a mercator tile and see how much requests GDAL/RASTERIO does.

Docker

Ready to use docker image can be found on Github registry.

docker run \
  --volume "$PWD":/data \
  --platform linux/amd64 \
  --rm -it -p 8080:8080 ghcr.io/developmentseed/tilebench:latest \
  tilebench viz --host 0.0.0.0 https://noaa-eri-pds.s3.us-east-1.amazonaws.com/2020_Nashville_Tornado/20200307a_RGB/20200307aC0865700w360900n.tif

Contribution & Development

See CONTRIBUTING.md

License

See LICENSE

Authors

See contributors for a listing of individual contributors.

Changes

See CHANGES.md.

Changelog

0.12.0 (2024-01-24)

  • allow tms options in CLI (profile, random and get-zooms) to select TileMatrixSet

0.11.0 (2023-10-18)

  • update requirements

    • rio-tiler>=6.0,<7.0
    • fastapi>=0.100.0
    • rasterio>=1.3.8
  • remove wurlitzer dependency

  • only use rasterio logs

  • remove LIST information breaking change

0.10.0 (2023-06-02)

  • update rio-tiler requirement
  • fix log parsing when CPL_TIMESTAMP=ON is set

0.9.1 (2023-03-24)

  • handle dateline crossing dataset and remove pydantic serialization

0.9.0 (2023-03-14)

  • update pre-commit and fix issue with starlette>=0.26
  • re-write NoCacheMiddleware as pure ASGI middleware
  • rename analyse_logs to parse_logs
  • add python 3.11 support

0.8.2 (2022-11-21)

  • update hatch config

0.8.1 (2022-10-31)

  • fix issue with min/max zoom when there is no overviews
  • calculate windows from block_shapes

0.8.0 (2022-10-25)

  • update rio-tiler/rasterio dependencies
  • remove python 3.7 support
  • add python 3.10 support
  • add image endpoint to show the data footprint
  • switch from mapbox to maplibre

0.7.0 (2022-06-14)

  • add cProfile stats

0.6.1 (2022-04-19)

  • Remove usage of VSIStatsMiddleware in tilebench viz

0.6.0 (2022-04-19)

  • switch to pyproject.toml

0.5.1 (2022-03-04)

  • make sure we don't cache previous request when using tilebench profile without --tile option

0.5.0 (2022-02-28)

  • update rio-tiler requirement
  • add reader option

0.4.1 (2022-02-14)

  • update Fastapi requirement
  • use WarpedVRT to get dataset bounds in epsg:4326

0.4.0 (2021-12-13)

  • update rio-tiler's version requirement
  • add more information about the raster in the Viz web page (author @drnextgis, https://github.com/developmentseed/tilebench/pull/14)
  • fix bug for latest GDAL/rasterio version
  • add default STAMEN basemap in viz and remove mapbox token/style options.
  • update fastapi requirement

0.3.0 (2021-03-05)

0.2.1 (2021-02-19)

  • fix typo in UI

0.2.0 (2021-01-28)

  • add warp-kernels in output in profile CLI
  • add rasterio/curl stdout in output
  • add dataread time in Viz

0.1.1 (2021-01-27)

  • update requirements

0.1.0 (2021-01-04)

  • add web UI for VSI stats visualization
  • add starlette middleware

0.0.2 (2020-12-15)

  • Update for rio-tiler==2.0.0rc3

0.1.0 (2020-07-13)

  • Initial release

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tilebench-0.12.0.tar.gz (23.3 kB view hashes)

Uploaded source

Built Distribution

tilebench-0.12.0-py3-none-any.whl (22.4 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page