Skip to main content

Filtering of OSM pbf-files and exporting to geojson

Project description

esy-osmfilter

esy-osmfilter is a Python library to read and filter OpenStreetMap data files in the Protocol Buffers (PBF) format and export them to a Python dictionary and/or JSON file.

Install

foo@bar:~$ pip install esy-osmfilter

Usage

In the following example the prefilter of esy-osmfilter is used to extract all pipelines from Liechtenstein. The easiest way to run esy-osmfilter is to download the sample.py from https://gitlab.com/dlr-ve-esy/esy-osmfilter and run it. It is quite similar to the more detailed desription below.

In this example, we start by importing all necessary libraries and methods.

>>> import configparser, contextlib
>>> import os, sys
>>> from esy.osmfilter import osm_colors as CC
>>> from esy.osmfilter import run_filter 
>>> from esy.osmfilter import Node, Way, Relation

Thereafter, the IO filepaths are defined, assuming the osm-pbf file is already downloaded.

>>> PBF_inputfile = os.path.join(os.getcwd(),
...                              'tests/input/liechtenstein-191101.osm.pbf')
>>> JSON_outputfile = os.path.join(os.getcwd(),
...                              'tests/output/LI/liechtenstein-191101.json')

Alternatively, you could also make use of urllib library to retrieve a OSM file:

>>> import urllib.request
>>> if not os.path.exists('tests/input/liechtenstein-191101.osm.pbf'):
...    filename, headers = urllib.request.urlretrieve(
...        'https://download.geofabrik.de/europe/liechtenstein-191101.osm.pbf',
...        filename='liechtenstein-191101.osm.pbf'
...    )
...    PBF_inputfile = filename

In the next step, a prefilter for all pipeline objects is defined. With the prefilter, we accept all way-items that have "man_made" as key and "pipeline" as value in their taglist. The white and black filter are left empty for the moment.

>>> prefilter   = {Node: {}, Way: {"man_made":["pipeline",],}, Relation: {}}
>>> whitefilter = []
>>> blackfilter = []

The run_filter function will allow to filter for OSM items from a pbf-file. We confirm the prefilter phase by setting the boolean variable NewPreFilterData=True.

>>> [Data,_]=run_filter('noname',
...                     PBF_inputfile, 
...                     JSON_outputfile, 
...                     prefilter,
...                     whitefilter, 
...                     blackfilter, 
...                     NewPreFilterData=True, 
...                     CreateElements=False, 
...                     LoadElements=False,
...                     verbose=True)

The prefilter returns the filter results to the Data dictionary. This means all OSM way-items with the tag "man_made"="pipeline" are stored there. But not enough, additionally, all referenced node items of these pipelines are stored there too.

>>> len(Data['Node'])
13
>>> len(Data['Relation'])
0
>>> len(Data['Way'])
2

In this example, we have only found two pipelines and their correspondent 13 nodes.

PLEASE NOTICE:
You can also set "man_made":True to accept items independently of a key value.

In the next step we use run_filter to load the Data dictionary and specify the main filtering results. In this example, we use the blackfilter to exclude possible pipelines substations from our prefiltering results.

>>> blackfilter = [("pipeline","substation"),]

We further only accept the drain pipelines that have the really great name "Wäschgräbli".

>>> whitefilter =[(("waterway","drain"),("name","Wäschgräble")), ]

We initiate the mainfilter phase by setting CreateElements=True.

>>> [_,Elements]=run_filter('funny-waterway-pipelines',
...                            PBF_inputfile, 
...                            JSON_outputfile, 
...                            prefilter,
...                            whitefilter, 
...                            blackfilter, 
...                            NewPreFilterData=False, 
...                            CreateElements=True, 
...                            LoadElements=False,
...                            verbose=True)
>>> len(Elements['funny-waterway-pipelines']['Node'])
0
>>> len(Elements['funny-waterway-pipelines']['Relation'])
0
>>> len(Elements['funny-waterway-pipelines']['Way'])
1

We see, that there is only one way-item left in the Elements dictionary, the other has been filtered out. There are no referenced nodes (or relation members) of the remaining way-item passed to the Elements dictionary.

However, these are still accessible in the Data dictionary.

Esy-osmfilter comes with an export function for GeoJSON files (not implemented for relations yet) which will make thinks a lot easier:

>>> from esy.osmfilter import export_geojson
>>> export_geojson(Elements['funny-waterway-pipelines']['Way'],Data,
... filename='test.geojson',jsontype='Line')

To visualize the output-file just open http://geojson.io and drag it on the screen.

image

For more details, jump to the documentation.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

esy-osmfilter-1.0.11.tar.gz (17.6 kB view details)

Uploaded Source

Built Distribution

esy_osmfilter-1.0.11-py3-none-any.whl (30.1 kB view details)

Uploaded Python 3

File details

Details for the file esy-osmfilter-1.0.11.tar.gz.

File metadata

  • Download URL: esy-osmfilter-1.0.11.tar.gz
  • Upload date:
  • Size: 17.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.7.4

File hashes

Hashes for esy-osmfilter-1.0.11.tar.gz
Algorithm Hash digest
SHA256 97a0aef58fb34822ef1309942bbeb31170cae10f7d61e245cecffe3d2a467ddd
MD5 da41eb452b3a8cb52f34cb43f84c7560
BLAKE2b-256 c18396107674bdef29fb3d403e730d79e5bd2bdf201594ba35cdb8985eefb178

See more details on using hashes here.

File details

Details for the file esy_osmfilter-1.0.11-py3-none-any.whl.

File metadata

  • Download URL: esy_osmfilter-1.0.11-py3-none-any.whl
  • Upload date:
  • Size: 30.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.7.4

File hashes

Hashes for esy_osmfilter-1.0.11-py3-none-any.whl
Algorithm Hash digest
SHA256 877c19edcf7340caf2b4cea776ceea008a8fa3a1bdd1966c2bf7642ed82af314
MD5 055cb81d024001480dc9f2cd23ee866a
BLAKE2b-256 edfea4c19693261c9d8e37d73d4cfdc668b5aca78a95daaa80c29da8e45501de

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page