Skip to main content

A collection of import scripts for converting bus schedule data into the [eflips-model](https://github.com/mpm-tu-berlin/eflips-model) data format.

Project description

Unit Tests Code style: black

eflips-ingest


Part of the eFLIPS/simBA list of projects.


This repository contains code to import bus schedules from various sources into an eFLIPS-Model database.

Installation

  1. Set up a PostgreSQL database with the PostGIS extension and BTREE_gist enabled.

    apt install postgresql postgis
    sudo -u postgres psql createdb eflips
    sudo -u postgres psql eflips -c "CREATE EXTENSION postgis;"
    sudo -u postgres psql eflips -c "CREATE EXTENSION btree_gist;"
    
  2. Clone this git repository (or download a specific release)

    git clone git@github.com:mpm-tu-berlin/eflips-model.git
    
  3. Install the packages listed in poetry.lock and pyproject.toml into your Python environment. Notes:

    • This project depends on pyproj, which may require the proj-bin package (apt install proj-bin on Ubuntu).
    • The supported platforms are macOS and Linux.
    • Using the poetry package manager is recommended. It can be installed accoring to the instructions listed here.
    poetry install
    

Usage

Command line

The code is organized into various Python files under the eflips/ingest folder, each for ingesting a specific data source. These files should be runnable using python eflips/ingest/x.py The following data sources are currently supported:

  • bvgxml.py: XML files emitted by BVG's proprietary software.
    • Requires at least GOOGLE_MAPS_API_KEY, also OPENELEVATION_URL is suggested to save money on Google Maps API calls.
    • Known Limitations:
      • The source data is known to contain some partial rotations of the bus routes. These are imported, which may lead to errors when creating a depot for each start and end spot, as some roations start/end at spots that are not depots.
      • Some routes that are going from the same stop to the stop itself are not imported, as they are considered to be invalid.
      • Some route's distance and time is fudged, when it is found to be unrealistically short or long.

API


All API Classes should inherit from eflip.ingest.base.AbstractIngester. This class also provides documentation on how to implement the required methods.


The eflips-ingest package is designed to provide a standardized access method to the different ingesters. It is designed as a reusable package providing a programmatic API. It provides the Ingester class with the following methods:

  • __init__(self, database_url: str): The constructor. It takes the database URL as a parameter.
  • prepare(self, progress_callback: None | Callable[[float], None] = None, **kwargs: Dict[str, str | int | float | bool | Enum | Path],) -> Tuple[bool, UUID | Dict[str, str]]:: This method is used to validate the data. It has multiple keyword arguments, one for each parameter. It returns a tuple containing a boolean indicating whether the data is valid and a UUID if the data is valid or a dictionary containing the error names and messages if the data is invalid. It is suggested that the input form is generated by introspecting this method. If a UUID is returned, this UUID can be used to start the ingestion process, even with another instance of this Ingester class (the validated input data is saved to a temporary directory).
  • ingest(self, UUID: UUID, progress_callback: None | Callable[[float], None]) -> None: This method is used to start the ingestion process. It takes the UUID returned by the validate method and a callback function that will be called with the progress of the ingestion process. It is suggested that the progress is displayed in the UI. This method may raise an exception if the ingestion process fails. Note that exceptions should not be raised during the ingest process if everything goes well, as we are checking validity in the validate method.
  • prepare_param_names() This property should return a dict with a short description of each parameter that the prepare method takes. This can be used for Naming fields in the UI.
  • prepare_param_descriptions() This property should return a dict with a long description of each parameter that the prepare method takes. This can be used for tooltips in the UI.

In addition, the package provides a get_ingesters function that returns a list of Ingester classes.

Testing


NOTE: Be aware that the tests will clear the database specified in the DATABASE_URL environment variable. Make sure that you are not using a database that you want to keep.


Testing is done using the pytest framework with tests located in the testsdirectory. To run the tests, execute the following command in the root directory of the repository:

   export PYTHONPATH=tests:. # To make sure that the tests can find the eflips package
   export DATABASE_URL=postgis://postgres:postgres@localhost:5432/postgres # Or whatever your database URL is
   export GOOGLE_MAPS_API_KEY=put_your_api_key_here # Required for some tests
   export OPENELEVATION_URL=put_your_url_here # Optional, required for some tests
   pytest

Development

We utilize the GitHub Flow branching structure. This means that the main branch is always deployable and that all development happens in feature branches. The feature branches are merged into main via pull requests.

We use black for code formatting. You can use pre-commit to ensure the code is formatted correctly before committing. You are also free to use other methods to format the code, but please ensure that the code is formatted correctly before committing.

Please make sure that your poetry.lock and pyproject.toml files are consistent before committing. You can use poetry check to check this. This is also checked by pre-commit.

License

This project is licensed under the AGPLv3 license - see the LICENSE file for details.

Funding Notice

This code was developed as part of the project eBus2030+ funded by the Federal German Ministry for Digital and Transport (BMDV) under grant number 03EMF0402.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

eflips_ingest-1.2.76.tar.gz (66.5 kB view details)

Uploaded Source

Built Distribution

eflips_ingest-1.2.76-py3-none-any.whl (80.2 kB view details)

Uploaded Python 3

File details

Details for the file eflips_ingest-1.2.76.tar.gz.

File metadata

  • Download URL: eflips_ingest-1.2.76.tar.gz
  • Upload date:
  • Size: 66.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for eflips_ingest-1.2.76.tar.gz
Algorithm Hash digest
SHA256 78b2fff3f13a3dfdc9d6d48f9e2e0236f22f735d284d2eee0a34bab9acba515b
MD5 45c676f759532799bdbcf449addba6c7
BLAKE2b-256 548d5d76d5884a5520fef66b6f92dc3bcd14d989d417900a5ee53ab0260bbf27

See more details on using hashes here.

File details

Details for the file eflips_ingest-1.2.76-py3-none-any.whl.

File metadata

File hashes

Hashes for eflips_ingest-1.2.76-py3-none-any.whl
Algorithm Hash digest
SHA256 7ded6f7253a049c65eb8ec53308d33ccfab006958d4f5657732f9d516374288b
MD5 9b061bcffd9752c843aff3299a6fba90
BLAKE2b-256 d8c3d37240d9671c3a76bfc77e690bbced46c4d5daaaadbcd81ad28f9509a805

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page