Tag Clustering for Tag Maps
Project description
Tag Maps
Spatio-Temporal Tag and Photo Location Clustering for generating Tag Maps
Tag Maps are similar to Tag Clouds, but Tag Maps use the spatial information that is attached to geotagged photographs, in addition to tag frequency, to visualize tags on a map. This Library uses the single-linkage tree that is available from HDBSCAN to cut trees at a specific user-defined distance for all available tags in the given dataset. Afterwards, Alpha Shapes are generated as a means to allow 'soft' placement of tags on a map, according to their area of use. Two Shapefiles are generated that can be used to visualize results, for example, in ESRI ArcGIS.
Based on the papers:
Dunkel, A. (2015). Visualizing the perceived environment using crowdsourced photo geodata. Landscape and Urban Planning, 142. DOI / PDF
Dunkel, A. (2016). Assessing the perceived environment through crowdsourced spatial photo content for application to the fields of landscape and urban planning. Thesis, TU Dresden Landscape and Environmental Planning. DOI / PDF
Overview of processing steps (Toronto High Park example):
- a) individual photo locations (raw data)
- b) photo locations combined to clusters
- c) tag location clustering (HDBSCAN) and Alpha-Shape generation
- d) soft placement of all relevant tag clusters using alpha shapes
Installation
- The easiest way for Windows users is to download the newest pre-compiled build from releases and run
tagmaps.exe
- The recommended way to install the package is with
conda install tagmaps
. Before, make sure that conda-forge is added as a channel:conda config --add channels conda-forge
- If you need to install with pip, it is recommended to install all dependencies first (e.g. Windows users: use Gohlke wheels if available) and then run
pip install tagmaps
See the Documentation for a complete guide to setup tagmaps package in Windows 10.
Documentation
See the tagmaps documentation for additional information, guides and tutorials.
Quickstart
- Place geotagged data in
/01_Input
subfolder- example data is available in the pre-compiled build zip-file above
- Output files will be saved in
/02_Output
(2 shapefiles in auto-selected UTM projection, one containing all tag cluster and one with the overall location clusters) - Visualize shapefiles, e.g. using ESRI ArcGIS (I haven't tried other GIS Software such as QGIS, but it should theoretically be possible..)
- download
BasemapLayout_World.mxd
from resources folder and replace missing links with 2 resulting Shapefiles in/02_Output
- adjust minimum and maximum Font Sizes, Weighting Formula or other metrics to your needs.
- download
Some background:
Tag Maps Package can be used with any tagged & spatially referenced data, but it has been specifically deveoped with Social Media data in mind (Flickr, Twitter etc.). There're two ways to feed input data:
- Unfiltered raw data
- Use
tagmaps.add_record(record)
where record is of typePostStructure
(see shared_structure.py) - How you clean up data totally depends on the type, have a look at LoadData class in load_data.py for Twitter and Flickr cleanup
- Use
- Filtered data
- the result from 1 is a
UserPostLocation
(UPL), which is a reference of type 'CleanedPost'. A UPL simply has all posts of a single user at a single coordinate merged, e.g. a reduced list of terms, tags and emoji based on global occurrence (i.e. no duplicates). - there're other metrics used throughout the package:
- UPL - User post location
- UC - User count
- PC - Post count
- PTC - Total tag count ("Post Tag Count")
- PEC - Total emoji count
- UD - User days (counting distinct users per day)
- DTC - Total distinct tags
- DEC - Total distinct emoji
- DLC - Total distinct locations
- the result from 1 is a
- The filtered data that is used for tagmaps can be exported using
tagmaps.write_cleaned_data()
. Since this will remove all terms/tags/emoji that do not appear in the top 1000 (e.g.) occurring global list of terms, this will produce a highly pseudo-anonymized set of information, with only collectively relevant terms remaining. The default value (1000) can be adjusted usingmax_items
arg, e.g. the smaller max_items, the higher is the effect of anonymization/generalization.
Code
The code has been completely refactored in January 2019, but there are still some missing pieces.
Particularly the API (that is: import tagmaps
) is still in an early stage. See method main() in main.py
for examples on how to use tag maps package.
Resources
- Check out this album on Flickr with some more Tag Maps examples
- There's also an semi-interactive interface to explore some Tag Maps here
- Check out my blog here with some background information
- A Jekyll-Reveal presentation on theory & background
Contributors
Some future goals:
- include topic modeling
- improve automatic detection of general vs specific tags for an area
- include mapping/visualization step (replace ArcGIS)
- improve unit testing (pytest) for tagmaps package
- move from tkinter interface to browser based solution
Built With
This project includes and makes use of several other projects/libraries/frameworks:
Alpha Shapes Kevin Dwyer/ Sean Gillies
Generating Concave Hull for Point Clouds
HDBSCAN McInnes, J. Healy, S. Astels - BSD licensed
A high performance implementation of HDBSCAN clustering.
Shapely
Manipulation and analysis of geometric objects
SciPy and Convex Hull
Simple shapes for point clusters are generated using SciPy's excellent Convex Hull functions
Fiona
OGR's neat and nimble API for Python programmers.
License
GNU GPLv3
Changelog & Download
2019-05-08: TagMaps v.0.17.6
- as of this version, tagmaps package is available on conda-forge
- fixed a bug with newer versions of pyproj (>2.0.0) that would result in very slow projection performance
2019-03-08: TagMaps v0.17.4
- First version of public API, e.g. load tagmaps to other packages with
import tagmaps
orfrom tagmaps import TagMaps
- Refactor of LoadData and PrepareData in separate classes, use of contextmanager/ pipeline generator
- Improved generation of Alpha Shapes
- Basic system integration test pipeline
- Jupyter Notebook compatibility
2019-01-23: TagMaps v0.11.1
- complete refactor of code with improved encapsulation, code now largely follows PEP conventions
- bugfix: emoji handling now accurately recognizes grapheme clusters consisting of multiple unicode codepoints.
- interface: add feature to filter based on toplists for tags, emoji and locations
- added sample CC-BY dataset
2018-01-31: TagMaps v0.9.2
- Because Tag Maps can be generated from local to regional to continental scale, finding an algorythm that fits all was not straight forward. The current implementation will produce shapes for all of these scales without any user input.
- This Final Alpha Shape Implementation is motivated from Kevin Dwyer/ Sean Gillies great base code
- Implementation of Auto-Projection from Geographic to Projected Coordinate System. The code will select the most suitable UTM Zone for projecting data.
2018-01-17: TagMaps v0.9.1
- First build
- Initial commit, still lots of unnecessary comments/code parts left in code
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.