pybikes 1.0.0.dev0

A python library for scraping bike sharing data

pybikes

pybikes provides a set of tools to scrape bike sharing data from different websites and APIs, thus providing a coherent and generalized set of classes and methods to access this sort of information.

The library is distributed and intended mainly for statistics and data sharing projects. More importantly, it powers the CityBikes project, and is composed of a set of classes and a pack of data files that provide instances for all different systems.

Installation

Install from PyPI:

pip install pybikes

Install directly from GitHub:

pip install git+https://github.com/eskerda/pybikes.git

Or after downloading/cloning the source:

pip install .

The following dependencies are required (example using Ubuntu package manager):

sudo apt-get install python
sudo apt-get install python-setuptools
sudo apt-get install libxml2 libxml2-dev libxslt1-dev libgeos-dev

Usage

>>> import pybikes

# Capital BikeShare instantiation data is in bixi.json file
>>> capital_bikeshare = pybikes.get('capital-bikeshare')

# The instance contains all possible metadata regarding this system
>>> print(capital_bikeshare.meta)
{
    'name': 'Capital BikeShare',
    'city': 'Washington, DC - Arlington, VA',
    'longitude': -77.0363658,
    'system': 'Bixi',
    'company': ['PBSC'],
    'country': 'USA',
    'latitude': 38.8951118
}
# The update method retrieves the list of stations
>>> print(len(capital_bikeshare.stations))
0
>>> capital_bikeshare.update()
>>> print(len(capital_bikeshare.stations))
191
>>> print(capital_bikeshare.stations[0])
--- 31000 - 20th & Bell St ---
bikes: 7
free: 4
latlng: 38.8561,-77.0512
extra: {
    'installed': True,
    'uid': 1,
    'locked': False,
    'removalDate': '',
    'installDate': '1316059200000',
    'terminalName': '31000',
    'temporary': False,
    'name': '20th & Bell St',
    'latestUpdateTime': '1353454305589'
}

Some systems might require an API key to work (for instance, Cyclocity). In these cases, the instance factory can take an extra API key parameter.

>>> key = "This is not an API key"
>>> dublinbikes = pybikes.get('dublinbikes', key)

Note that pybikes works as an instance factory and, choicely, instances can be generated by passing the right arguments to the desired class

>>> from pybikes.cyclocity import BixiSystem
>>> capital_bikeshare = BixiSystem(
        tag = 'foo_tag',
        root_url = 'http://capitalbikeshare.com/data/stations/',
        meta = {'foo':'bar'}
    )

The way information is retrieved can be tweaked using the PyBikesScraper class included on the utils module thus allowing session reusing and niceties such as using a proxy. This class uses Requests module internally.

>>> scraper = pybikes.utils.PyBikesScraper()
>>> scraper.enableProxy()
>>> scraper.setProxies({
        "http" : "127.0.0.1:8118",
        "https": "127.0.0.1:8118"
    })
>>> scraper.setUserAgent("Walrus™ v3.0")
>>> scraper.headers['Foo'] = 'bar'
>>> capital_bikeshare.update(scraper)

Tests

Tests are separated between unit tests and integration tests with the different sources supported.

To run unit tests simply

make test

To run integration tests

make test-update

Note that some systems require authorization keys, tests expect these to be set as environment variables like:

PYBIKES_CYCLOCITY='some-api-key'
PYBIKES_DEUTSCHEBAHN_CLIENT_ID='some-client-id'
PYBIKES_DEUTSCHEBAHN_CLIENT_SECRET='some-client-secret'

# or if using an .env file
# source .env

make test-update

This project uses pytest for tests. Test a particular network by passing a filter expresson

pytest -k bicing
pytest -k gbfs

To speed up tests execution, install pytest-xdist to specify the number of CPUs to use

pytest -k gbfs -n auto

To use Makefile steps and pass along pytest arguments, append to the T_FLAGS variable

make test-update T_FLAGS+='-n 10 -k gbfs'

Integration tests can generate a json report file with all extracted data stored as geojson. Using this json report file, further useful reports can be generated like a summary of the overall health of the library or a map visualization of all the information.

For more information on reports see utils/README.md

Development

We welcome contributions from the community! The best place to get started is by diving into the codebase or checking the issues list.

Join our developer community on Matrix: #citybikes:matrix.org