OSM bot abstraction layer is a python package making easier to automate OpenStreetMap edits without causing problems.

This projects contains code shared between OpenStreetMap bots, to avoid writing the same functionality more than once.

For example, automated edit may change many objects across large area. In that case it is preferable to split it into multiple edits to avoid country-sized bounding boxes. This logic is available as one of tools included in this project.

Installation

pip install osm-bot-abstraction-layer --user

Testing

python3 -m unittest

Contributing

First contact with any project is crucial, so please report any and all issues in this readme. If you needed tool like this but decided to not use it for some fixable reason - please open an issue!

Reports about how documentation can be improved, bug reports, pull requests are welcomed!

Pull requests are welcomed from smallest typo to big new features - though in case of huge changes it is always a good idea to start from opening an issue.

Purpose

This tool is for people who are at once

• programmers
• experienced OSM editors
• following OSM rules

Anyone who runs automated edits is responsible for all problems that appear, including ones caused by bugs in external libraries like this one.

I eliminated all bugs that I noticed, after all I am using this code myself, but some are still lurking. Issue reports and pull requests are welcomed!

Reminder about OSM rules

Note that automated must not be done without consultation or agreement of a community.

See the Import/Guidelines and Automated Edits/Code of Conduct for more information.

Note that automated edits violating rules mentioned above are routinely undone. Undiscussed automatic edits may be reverted by anybody, without any consultation.

And yes, it means that some automated edits that would save time and make perfect sense were rejected and should not be made. It is still preferable over unrestricted automated edits.

Configuration

Sadly, we need to deal with OAuth.

Create application at https://www.openstreetmap.org/oauth2/applications (using urn:ietf:wg:oauth:2.0:oob as redirect URI).

Create secret.json file with content like this:

{
	"bot_account": {
		"script_client_id": "replace",
		"script_secret": "replace",
		"username": "replace with OSM username, optional field needed only for documentation"
	},
	"human_account": {
		"script_client_id": "replace",
		"script_secret": "replace",
		"username": "replace with OSM username, optional field needed only for documentation"
	}
}

On first run script should instruct how to obtain token that will be added to that field.

After that it should have following format (XXXXX are placeholders):

{
	"bot_account": {
		"script_client_id": "replace",
		"script_secret": "replace",
		"username": "replace with OSM username, optional field needed only for documentation",
    "token": {"access_token": "XXXXX", "token_type": "Bearer", "scope": ["write_api", "write_notes"], "created_at": XXXXX}
	},
	"human_account": {
		"script_client_id": "replace",
		"script_secret": "replace",
		"username": "replace with OSM username, optional field needed only for documentation",
    "token": {"access_token": "XXXXX", "token_type": "Bearer", "scope": ["write_api", "write_notes"], "created_at": XXXXX}
	}
}

Usage example

Bot edit

Following is example based on a real automated edit, following guidelines for the automatic edits.

1. Relevant community was asked. In this case it affected Polish mappers, so thread appeared in Polish section of forum.openstreetmap.org. Note that different communities may use different forums or mailing lists as their communication channels.
2. In addition Page documenting the automated edit was created at OSM wiki.
3. OSM community accepted the edit.
4. Following code was created using run_simple_retagging_task component. Running this script will result in:
   • downloading OSM data using Overpass Turbo as specified in objects_to_consider_query parameter
   • iterate over all and objects, ignoring ones where function passed as parameter is_element_editable_checker_function returns false
   • for all other edit_element_function is applied
   • changes are automatically split in multiple changesets (if necessary) to avoid too large bounding boxes or too many objects in one edit
   • changeset_comment, discussion_url, osm_wiki_documentation_page parameter values are used to apply correct changeset tags
5. Running this code resulted in two edits: #64628901 and #64628951

from osm_bot_abstraction_layer.generic_bot_retagging import run_simple_retagging_task

def edit_element(tags):
    if tags.get('amenity') != ("atm"):
        return tags
    if tags.get('name') == ("bankomat"):
        tags.pop('name', None)
    if tags.get('name') == ("Bankomat"):
        tags.pop('name', None)
    return tags

def main():
    run_simple_retagging_task(
        max_count_of_elements_in_one_changeset=500,
        objects_to_consider_query="""
[out:xml][timeout:25000];
area[name="Polska"]->.a;
(
  nwr[amenity='atm'][name='Bankomat'](area.a);
  nwr[amenity='atm'][name='bankomat'](area.a);
);
out body;
>;
out skel qt;
""",
        cache_folder_filepath='/media/mateusz/5bfa9dfc-ed86-4d19-ac36-78df1060707c/OSM-cache',
        is_in_manual_mode=False,
        changeset_comment='usuwanie nazw opisowych z bankomatów (name="bankomat" i name="Bankomat")',
        discussion_url='https://forum.openstreetmap.org/viewtopic.php?id=66038',
        osm_wiki_documentation_page='https://wiki.openstreetmap.org/wiki/Mechanical_Edits/Mateusz_Konieczny_-_bot_account/fix_descriptive_name_on_ATMs_in_Poland',
        edit_element_function=edit_element,
    )

main()

Downloading data

Using Overpass

from osm_bot_abstraction_layer.overpass_downloader import download_overpass_query

vienna_download = """
  [timeout:250];
  (
    nwr(48.10,16.26,48.26,16.51);
  );
out body;
>;
out skel qt;
"""
download_overpass_query(vienna_download, '/tmp/vienna.osm')

If I need to iterate over such file I typically use https://github.com/matkoniecz/osm_iterator

For looking at bounding boxes I typically use http://bboxfinder.com - for example see 48.10,16.26,48.26,16.51

Using OSM Editing API

Create secret.json file with

{
	"bot_account": {
		"username": "OSM user name",
		"password": "password in plaintext"
	}
}

import osm_bot_abstraction_layer.osm_bot_abstraction_layer as osm_bot_abstraction_layer
import json

id = 1
type = "node"
returned = osm_bot_abstraction_layer.get_data(id, type)
json.dumps(returned, default=str, indent=3)

it will give output like

{
   "id": 1,
   "visible": true,
   "version": 33,
   "changeset": 124176968,
   "timestamp": "2022-07-28 09:47:39",
   "user": "owene",
   "uid": 29598,
   "lat": 42.7957187,
   "lon": 13.5690032,
   "tag": {
      "communication:microwave": "yes",
      "communication:radio": "fm",
      "description": "Radio Subasio",
      "frequency": "105.5 MHz",
      "man_made": "mast",
      "name": "Monte Piselli - San Giacomo",
      "tower:construction": "lattice",
      "tower:type": "communication"
   }
}

Handling note spam

from osm_bot_abstraction_layer import osm_bot_abstraction_layer
import osmapi

for id in range(3158344, 3158710):
	print(id)
	data = osm_bot_abstraction_layer.get_data(id, "note")
	text = data['comments'][0]['text']
	print("----")
	print(text)
	print("----")
	if text in ["tf"]:
		try:
			osm_bot_abstraction_layer.close_note(id, "Bot goes brrrrrrr. Spam cleanup.", 'bot_account')
		except osmapi.errors.NoteAlreadyClosedApiError as e:
			pass

Show internal administrative division of Iran

See docs at OSM Wiki how internal administrative structure is tagged

import osm_bot_abstraction_layer.world_data as world_data
print(world_data.list_of_area_division_data('IR', 4, ["name", "wikidata"], '/tmp/boundary_data.osm'))

Returns list of dicts with names (from name tag, so should contain local ones) and wikidata codes.

Download file

import osm_bot_abstraction_layer.util_download_file

directory = '/tmp/'
filename = 'planet-latest.osm.pbf.torrent'
url = 'https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf.torrent'
osm_bot_abstraction_layer.util_download_file.download_file_if_not_present_already(url, directory, filename)

List valid shop= values

import osm_bot_abstraction_layer.tag_knowledge as tag_knowledge

for value in tag_knowledge.valid_shop_values():
  print(shop, "=", value)

Further documentation

Documentation is currently mostly missing - please, open an issue if it would be useful for you (pull requests are also welcomed).

History and etymology

I created this library as part of writing bot editing OpenStreetMap.

Parts of the project built upon osmapi and provide an additional abstraction layer. This part was initial and was source of the project name.

Project location

This project resides at https://github.com/matkoniecz/osm_bot_abstraction_layer