lbsn data structure format & transfer tool
Project description
LBSNTRANSFORM
A python package that uses the [common lbsn data structure concept] (ProtoBuf) to import, transform and export Social Media data such as Twitter.
Description
This tool will read JSONs or CSVs from a Postgres Database or local folder and map Social Media Endpoints (e.g. Twitter)
to a common LBSN Interchange Structure format in ProtoBuf. Output can be either a Postgres Database or local CSV.
The tool can also be imported to other Python projects with import lbsntransform
for on-the-fly conversion. The goal is to provide a common interface to process
Social Media Data, without custom adjustment to the myriad API Endpoints available.
Quick Start
You can install the package with all its dependencies directly from the Git Repository:
pip install --upgrade git+git://gitlab.vgiscience.de:lbsn/lbsntransform.git
(Note: A PyPi package distribution is planned)
.. or, for non-developers, simply download the latest build and run with custom args,
e.g. with the following command line args
lbsntransform.exe --LocalInput 1 --LocalFileType '*.json' --transferlimit 1000 --CSVOutput
.. the the tool will:
- read local json from /01_Input/
- and store lbsn records as CSV and ProtoBuf in /02_Output/
For a full list of possible input args and descriptions see config.py.
Built With
- lbsnstructure - A common language independend and cross-network social-media datascheme
- protobuf - Google's data interchange format
- psycopg2 - Python-PostgreSQL Database Adapter
- ppygis3 - A PPyGIS port for Python
- shapely - Geometric objects processing in Python
- emoji - Emoji handling in Python
Contributing
Field mapping from and to ProtoBuffers from different Social Media sites is provided in field_mapping.py.
As an example, a mapping of Twitter json structure is given (see class FieldMappingTwitter
). This class may be used to extend
functionality to cover other networks such as Flickr or Foursquare.
For development & testing, make a local clone of this repository
git clone git@gitlab.vgiscience.de:lbsn/lbsntransform.git
..and create package in develop mode to symlink the folder to your
Python's site-packages folder with:
python setup.py develop
Now you can run tool in your shell with:
lbsntransform --LocalInput 1 --LocalFileType '*.json' --transferlimit 1000 --CSVOutput
..or import the package to other python projects with:
import lbsntransform
Versioning and Changelog, and Download
For the versions available, see the tags on this repository. Latest version is 0.1.4. The latest windows build is available for download here. For all other systems use cx_freeze to build executable:
python cx_setup.py build
Authors
- Alexander Dunkel - Initial work
See also the list of contributors.
License
This project is licensed under the GNU GPLv3 or any higher - see the LICENSE.md file for details.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.