Utility for Location History data from Google Takeout.
Project description
Google Location Utility
Utility for Location History data from Google Takeout.
Overview
GoogleLocationUtility, GLU is a command-line interface (CLI) tool for processing and utilizing location history data from Google Takeout built with Python. See below for requirements and installation instructions. A detailed usage guide is provided in the documentation.
Requirements
Python version 3.8 or newer is required. See the beginner's guide for help installing Python on your system.
Installing GLU within a virtual environment (venv) will install all required packages.
The requirements and versions specified during installation are listed below. A list of all installed packages (output to pip freeze > requirements.txt
) is provided in the docs, installed packages.
- Click >= 8.0, used to build CLI
- exif >= 1.3, reads/adds GPS tags from/to image files
- plotly >= 5.6, generates HTML interactive maps
- pandas >= 1.4, main tool for data handling
- numpy >= 1.22, used with pandas for calculations
- matplotlib >= 3.5, graphing engine for location reports
- seaborn >= 0.11.2, used with matplotlib
- fastparquet >= 0.8, efficient storage for processed location data
- Jinja2 >= 3.0, required for pandas HTML exports, report building
Installation, Quickstart
The following steps provide installation instructions within a virtual environment. It may be possible to install and use GLU in your base environment, but it is not recommended.
1. In a directory of your creation, <your-directory>
. Within this folder, create and activate a virtual environment in venv
.
# Unix / macOS
cd <your-directory>
python3 -m venv venv
. venv/bin/activate
# Windows
cd <your-directory>
py -3 -m venv venv
venv\Scripts\activate
The following steps assume you are in <your-directory>
and the virtual environment is activated. These are accomplished in the first, cd ...
, and third, ...activate
, lines.
Option 2A - PyPi installation
2A. Install package from PyPi.
- Download Configuration.ini as
<your-directory>/Configuration.ini
, be sure not to change the file extension.
# Unix / macOS / Windows
pip install GoogleLocationUtility
Obtain Configuration.ini through command line using curl or wget. Note capitilization of "o/O" for curl/wget
# Unix / macOS / Windows
curl https://raw.githubusercontent.com/NBPub/GoogleLocationUtility/main/Configuration.ini -o ./Configuration.ini
# OR
wget https://raw.githubusercontent.com/NBPub/GoogleLocationUtility/main/Configuration.ini -O ./Configuration.ini
Option 2B - Download and install from Github
2B. Download and extract GLU into your directory <your-directory>
. Install package.
- Click the green Code button at the top of the page for various download options.
- Note that the "docs" folder can be deleted
# Unix / macOS / Windows
pip install --editable .
Optional, delete docs folder
# Unix / macOS / Windows
rm -r docs
rm README.md
# Windows
rmdir /s docs
del README.md
3. Add Location History Takeout export.
- Enter
home
to ensure GLU is functioning, andLocationData
,Outputs
folders exist within<your-directory>
- Request, and then extract, your Location History data from Google Takeout.
- Copy location data, Records.json, and optionally Settings.json, into
<your-directory>/LocationData
.- Other exported files are not used by GLU.
4. See Getting Started for detailed usage instructions.
- Filestatus and available functions are listed with
home
, all available options are listed withhome --help
. - Modify
<your-directory>/Configuration.ini
to specify various configuration settings.- Open configuration file for editing with
home --config
- Open configuration file for editing with
- Documentation can be accessed from GLU with
home --docs
orhome --docs_read
.
Features
For more information about the functions available, see their respective files in the documentation. Configuration settings for the functions are detailed in Configuration.ini Usage.
GLU functions stem from the command, home
, which provides an overview of files and functions available. home --help
will provide all the function options.
"home" printout, with no location data to process and no processed data to use
Location History Export
GLU works with exported Location History Records from Google Takeout. Settings are optional, and may provide additional information about devices, which are reported as 10-digit integers in Records. After extraction, the exported Location History files from Google should be in JSON format.
Semantic Location History and Tombstones are not used by GLU.
"home" printout with location history available, as well as processed data and reports.
The details links will open the corresponding function page in the docs.
Processing
home --loc_parse
details
Before location data can be used, Records.json must be processed. GLU detects Records.json within the LocationData folder in the project root: <your-directory>/LocationData
.
Python's JSON decoder is used to convert the data into a dictionary (this process may be memory intensive), which is then read into a pandas DataFrame. Timestamps, GPS coordinates, device IDs, and sources are kept from the exported data. Additionally, the timedelta in between each point is calculated.
The resulting DataFrame is saved as a Parquet file, which allows for data type persistence and fast loading/saving, in the LocationData directory:
<your-directory>/LocationData/parsed_<date>.parquet
Example processing operation, with ~500MB Records.json file
Reports
home --location_report
details
Reports can be generated from any processed data. Accuracy and time-delta statistics are presented, along with a breakdown of accuracy against source(s) and device(s). Maps detailing locations of each device can optionally be generated. Reports are saved as HTML files, static graphs as PNG images, and maps as HTML files.
Each report, containing these files, is saved as a folder in the Outputs directory: <your-directory>/Outputs/<Report Folder>
.
Filtering
home --loc_filter
details
Processed location data from Records.json is considered "bulk" data. These can be further "filtered" by accuracy, source, and device.
A report for bulk data may be useful in determining filter parameters. Reports can be generated from filtered location data, too.
Notes of filter operations are stored in a CSV table: <your-directory>/LocationData/filter_notes.csv
.
Results of filter operations are saved as a new Parquet file in the LocationData directory:
<your-directory>/LocationData/filtered.parquet
Preview
Maps
home --loc_map
details
Location data within an input time range can be used to generate an interactive HTML map with panning and zooming capabilities. Map markers can be styled by "time" or "accuracy". Street tiles from OpenStreetMap are used in the Plotly graphs, so zooming in provides more detail.
Maps are saved in Outputs/Maps: <your-directory>/Outputs/Maps/MAP-<style>_<date>.html
Preview
geoTag
home --geoTag
details
You made it! This was my main purpose in building this program. See a previous version, and inspiration.
geoTag can add or replace GPS coordinates to an image's EXIF metadata, provided a suitable match is found in the processed location data. A detailed HTML report of an operation can be saved, including maps of tagged photos.
If a match is made, image copies are saved in the Outputs directory: <your-directory>/Outputs/<geoTag results>
Preview
GPS metadata can also be removed from images in a folder with home --geoStrip
details. Copies are also saved in the Outputs directory.
Preview
Limitations, Issues
Altitude
Altidude is dropped from the exported Location History in this version, as my data was mostly empty. Altidude (m above sea level) is an available tag for an image's EXIF metadata, provided it's > 0.
PhotoTag - Improper EXIF tags
Various failure mechanisms may prevent GPS metadata from being added to an image. These may be endemic to the file itself, but could also be a limitation of my methodology or the package used to read/write EXIF tags. In my previous iteration of a photo-tagger, I remember issues with certain panoramas from a mobile phone, images converted from a raw image file, and older images I didn't know much about.
Verbose errors are captured in a GeoTag detailed report.
Please report any errors you may encounter.
Future
Completed fixes and improvements after initial upload/commit, for alpha release:
Improved settings load- allow graceful exit when inputting Map settings
- fallback values
- checks for file and sections
- Relevant error messages and diagnosis
publish on PyPi- with release, delete old version and upload alpha.
general fixesdocumentation
Ideas for improvement and future releases:
- Critical / code improvement
- Implement tests
- Utilize Jinja2 HTML templates to clean up code for reports (location report, geotag report)
- Python Packaging:
- Use setuptools_scm properly, harmonize packaging / PyPi / git
- Include Configuration.ini in PyPi distribution, package data / data_files
- Docs:
- Spoof my location data to provide HTML examples of interactive maps
- Features:
- Option for TimeZoneDB integration to check location vs. input timezone
- Include additional location history parameters (altitude, velocity, heading, etc . . .)
- Determine if Pillow vs exif for tag writing is beneficial.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for GoogleLocationUtility-0.1.0a1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3093a831eeb86b8a0441ecf20585071eae2580f9696f5050da30a2a7b1ad13c4 |
|
MD5 | 4d3adedf975aef97cb50979ac6e00db1 |
|
BLAKE2b-256 | fee12c254a30da9942d163cb13afbba1c3af707aaf20c19ad8f46bbfed1eadec |
Hashes for GoogleLocationUtility-0.1.0a1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b1dd56f08a8a9f853c53f5a52466c69cf643cc257f63bcbb7a3f1ae3262c2e37 |
|
MD5 | 6ba31644b27953d2e8227c8ea28cf24a |
|
BLAKE2b-256 | cb0ce11ed21cb73af512db93a0a3588bb0acc11a10b254b4694f2bc90b992385 |