overturemapsdownloader 0.1.9

Overture Maps Downloader simplifies geospatial data manipulation

Overture Maps Downloader

This repository is still under heavy development. Features may change, and documentation will be updated accordingly. Use at your own risk and feel free to contribute!

Overview

OvertureMapsDownloader is a monorepo that contains two main projects aimed at simplifying the process of working with Overture Maps data in Parquet format.

DuckDB-WASM with Map

The first project is a browser-based application that leverages DuckDB-WASM and mapping functionalities. This allows you to query Overture Maps data in Parquet format directly within your web browser. It offers a seamless and interactive way to work with geospatial data without the need for any backend services. The codebase for this project is located in the /app folder.

Data Manipulation/Downloading and Conversion

The second project focuses on data manipulation, downloading, and conversion of Overture Maps data. It utilizes technologies like DuckDB, Python, and IPython Notebooks, along with GDAL/OGR for data conversion. This project is containerized using Docker, making it easy to integrate into existing pipelines and ETL workflows. For hands-on examples, check out the Jupyter notebooks available in the examples folder.

Whether you're a data scientist, a geospatial analyst, or a developer, OvertureMapsDownloader offers comprehensive solutions for your geospatial data needs.

Table of Contents

• Overture Maps Downloader
  • Overview
    • DuckDB-WASM with Map
    • Data Manipulation/Downloading and Conversion
  • Table of Contents
    • DuckDB-WASM in Browser (Developers)
    • Getting Started
    • Visual Demo
    • Data Manipulation/Downloading and Conversion
  • Prerequisites
  • Installation
    • the docker image manily contains the following tools:
  • Usage
    • Download Geospatial Data
      • Commands
  • Examples of downloading data CLI:
    • With Docker:
    • Without Docker you can use the following commands with (pip install overturemapsdownloader):
    • Convert Parquet to GeoPackage
    • Convert Parquet to MBTiles (will support tippecanoe in the future)
    • Convert Parquet to ESRI File Geodatabase vector (OpenFileGDB)
  • Using Jupyter Notebooks (Data Scientists/GIS Analysts)
  • Configuration
  • Roadmap
    • DuckDB-WASM in Browser (Developers) (ReactJS)
    • Data Manipulation/Downloading and Conversion (Python)
  • Contributing

DuckDB-WASM in Browser (Developers)

For developers interested in running DuckDB-WASM in the browser, the relevant codebase is located in the /app folder at the root of the repository. This setup allows you to develop, run, and build the DuckDB-WASM component seamlessly.

Getting Started

1. Navigate to the /app folder:

   cd /app
   

2. Install the required packages:

   yarn install
   

   Note: Make sure you have npm version 18 or above for compatibility.

3. Start the development server:

   yarn dev
   

then open http://localhost:5173/OvertureMapsDownloader in your browser

Visual Demo

Data Manipulation/Downloading and Conversion

Prerequisites

• Docker and Docker Compose
• GeoJSON file containing the bounding box polygon (e.g., bbox.geojson) you can make yours easly on https://geojson.io or use examples/bbox.json for testing

Installation

To get started, you'll need to pull the Docker image from the GitHub Container Registry:

docker pull ghcr.io/youssef-harby/overturemapsdownloader:latest

the docker image manily contains the following tools:

• GDAL/OGR v3.8.4
• Python v3.10 (and GDAL/OGR Python bindings + duckdb python bindings)
• DuckDB v0.10.0
• Jupyter.. and more

Usage

Download Geospatial Data

1. Create a folder and place your bounding box polygon in GeoJSON format inside it (e.g., bbox.geojson).

2. Navigate to the folder:

   cd /examples # e.g., cd /path/to/your/folder contains bbox.geojson and will contain the output file
   

3. Run the following command to download geospatial data:

   docker run -v $(pwd):/examples ghcr.io/youssef-harby/overturemapsdownloader:latest OMDownloader omaps --theme places --ptype place --bbox /examples/bbox.geojson --output /examples/places.parquet
   

Commands

The omaps Command

OMDownloader omaps [OPTIONS]

options:

• --theme [admins|buildings|buildings|places|transportation] Theme of the data to download
• --ptype [locality|administrativeBoundary|building|place|water|connector|segment] Type of the data to download
• --bbox PATH Bounding box polygon in GeoJSON format as a path to a file
• --output PATH Output file path (e.g., places.parquet)
• --help Show this message and exit.

Examples of downloading data CLI:

With Docker:

# admins/locality
docker run -v $(pwd):/examples ghcr.io/youssef-harby/overturemapsdownloader:latest OMDownloader omaps --theme admins --ptype locality --bbox /examples/bbox.geojson --output /examples/locality.parquet

# admins/administrativeBoundary
docker run -v $(pwd):/examples ghcr.io/youssef-harby/overturemapsdownloader:latest OMDownloader omaps --theme admins --ptype administrativeBoundary --bbox /examples/bbox.geojson --output /examples/admins.parquet

# buildings/building
docker run -v $(pwd):/examples ghcr.io/youssef-harby/overturemapsdownloader:latest OMDownloader omaps --theme buildings --ptype building --bbox /examples/bbox.geojson --output /examples/building.parquet

# places/place
docker run -v $(pwd):/examples ghcr.io/youssef-harby/overturemapsdownloader:latest OMDownloader omaps --theme places --ptype place --bbox /examples/bbox.geojson --output /examples/place.parquet

# transportation/connector
docker run -v $(pwd):/examples ghcr.io/youssef-harby/overturemapsdownloader:latest OMDownloader omaps --theme transportation --ptype connector --bbox /examples/bbox.geojson --output /examples/connector.parquet

# transportation/segment
docker run -v $(pwd):/examples ghcr.io/youssef-harby/overturemapsdownloader:latest OMDownloader omaps --theme transportation --ptype segment --bbox /examples/bbox.geojson --output /examples/segment.parquet

Without Docker you can use the following commands with (pip install overturemapsdownloader):

# admins/locality
OMDownloader omaps --theme admins --ptype locality --bbox examples/bbox.geojson --output examples/locality.parquet

# admins/administrativeBoundary
OMDownloader omaps --theme admins --ptype administrativeBoundary --bbox examples/bbox.geojson --output examples/admins.parquet

# buildings/building
OMDownloader omaps --theme buildings --ptype building --bbox examples/bbox.geojson --output examples/building.parquet

# places/place
OMDownloader omaps --theme places --ptype place --bbox examples/bbox.geojson --output examples/place.parquet

# transportation/connector
OMDownloader omaps --theme transportation --ptype connector --bbox examples/bbox.geojson --output examples/connector.parquet

# transportation/segment
OMDownloader omaps --theme transportation --ptype segment --bbox examples/bbox.geojson --output examples/segment.parquet

Convert Parquet to GeoPackage

To convert the downloaded data to GeoPackage format, run the following command:

docker run -v $(pwd):/examples ghcr.io/youssef-harby/overturemapsdownloader:latest ogr2ogr /examples/output.gpkg /examples/places.parquet

Convert Parquet to MBTiles (will support tippecanoe in the future)

docker run -v $(pwd):/examples ghcr.io/youssef-harby/overturemapsdownloader:latest ogr2ogr -dsco MAXZOOM=14 /examples/output.mbtiles /examples/places.parquet

Convert Parquet to ESRI File Geodatabase vector (OpenFileGDB)

docker run -v $(pwd):/examples ghcr.io/youssef-harby/overturemapsdownloader:latest ogr2ogr /examples/output.gdb /examples/places.parquet

Using Jupyter Notebooks (Data Scientists/GIS Analysts)

If you prefer to use Jupyter notebooks for your geospatial data manipulation tasks, you can easily set up a Jupyter environment using Docker Compose.

1. Clone the repository:

   git clone https://github.com/Youssef-Harby/OvertureMapsDownloader.git
   

2. Navigate to the repository folder:

   cd OvertureMapsDownloader
   

3. Run the following Docker Compose command:

   docker compose up
   

This will start a Jupyter notebook server accessible at http://localhost:8888/lab.

Configuration

For advanced configurations, please refer to the config.yml file.

Roadmap

DuckDB-WASM in Browser (Developers) (ReactJS)

• Add Popup on click of the feautre to show the result of the query
• Add a button to use local parquet url instead of the remote one
• Add a button to download the result of the query as GeoJSON..etc
• Add a loading spinner while the query is running
• Query by bbox drawn on the map
• feel free to add more ...

Data Manipulation/Downloading and Conversion (Python)

• Resolve JSON values to be parsed to the correct data type OSGeo/gdal#8227

Contributing

Everyone is invited to contribute to this project. Whether you're a beginner or experienced, your help is valuable. Don't worry about not getting it right the first time; the idea is to grow and learn together.

You can contribute by:

• Commenting on or adding to existing issues
• Starting or participating in discussions

If you're new and have questions or need guidance, the discussions page is a great place to start.