Skip to main content

Rapidly search your text for large list of cities and countries

Project description

ahogeotagger

If you have thousands of passages of text and you want to search them for hundreds or thousands of search strings, the process can become very cumbersome very soon. This is where ahocorasick search becomes very relevant and blazingly fast. You can read more about the algorithm here.

This code combines the very efficient implementation of ahocorasick in python with a pre-bundled list of cities, states and countries to tag locations in text.

Requirements

  • Python 3.6 or higher
  • pyahocorasick 1.4.0 or higher

Installation

Using PIP via PyPi

pip install ahogeotagger

Usage

Right now the usage is pretty simple. You import and init the tagger with the number of cities you want to search your text for. The cities are in order of population (Tokyo, New york, Mexico City etc.).

The data for these cities has been prepopulated from the free version of simplemaps database which you can find here.

from ahogeotagger import tagger
tagger.init(num_cities = 10000)

Optionally, if you dont want to use the built-in database of cities, you can provide your own list of cities. The list needs to be a list of tuples with each tuple's values in the following order:

(id,city,state,country,iso2,iso3,population,lat,lng)

The types for id and population are int, lat and lng are floats and all the rest are strings.

tagger.init(num_cities = 500, cities = [a,b,c])

where a,b,c are tuples described above.

To search whether text contains locations, supply any plain text to the search function like this:

results = tagger.search('New york and London are are competing for tech talent')
print(results)

This produces the following list of tuples as a result

[(0, 7, 'New York', 'New York', 'United States', 'US', 'USA', 19354922, 40.6943, -73.9249), 
 (13, 18, 'London', 'London, City of', 'United Kingdom', 'GB', 'GBR', 8567000, 51.5, -0.1167)]

Each tuple always contains values in this order start_index (for the match in the source string), end_index, city, state, country, iso2, iso3, population, latitude, longitude

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for ahogeotagger, version 0.1.5
Filename, size File type Python version Upload date Hashes
Filename, size ahogeotagger-0.1.5.tar.gz (449.1 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page