eflips-ingest

Part of the eFLIPS/simBA list of projects.

This repository contains code to import bus schedules from various sources into an eFLIPS-Model database.

Installation

1. Set up a PostgreSQL database with the PostGIS extension and BTREE_gist enabled.

   apt install postgresql postgis
   sudo -u postgres psql createdb eflips
   sudo -u postgres psql eflips -c "CREATE EXTENSION postgis;"
   sudo -u postgres psql eflips -c "CREATE EXTENSION btree_gist;"
   

2. Clone this git repository (or download a specific release)

   git clone git@github.com:mpm-tu-berlin/eflips-model.git
   

3. Install the packages listed in poetry.lock and pyproject.toml into your Python environment. Notes:

   • This project depends on pyproj, which may require the proj-bin package (apt install proj-bin on Ubuntu).
   • The supported platforms are macOS and Linux.
   • Using the poetry package manager is recommended. It can be installed accoring to the instructions listed here.

   poetry install
   

Usage

The code is organized as a set of ingester packages under eflips/ingest/, each covering a specific data source. Use them via the API described below. Supported sources:

• eflips.ingest.bvgxml.BvgxmlIngester: XML files emitted by BVG's proprietary software.
  • Known Limitations:
    • The source data is known to contain some partial rotations of the bus routes. These are imported, which may lead to errors when creating a depot for each start and end spot, as some roations start/end at spots that are not depots.
    • Some routes that are going from the same stop to the stop itself are not imported, as they are considered to be invalid.
    • Some route's distance and time is fudged, when it is found to be unrealistically short or long.
• eflips.ingest.vdv.VdvIngester: VDV 451/452 archives.
• eflips.ingest.gtfs.GtfsIngester: GTFS feeds.

Altitude lookups

All ingesters store stations and route shapes with an altitude (the model's geometries are POINT Z / LINESTRING Z). Altitudes are looked up through eflips.model.util.get_altitudes, which batches all coordinates of an import into as few requests as possible and caches results on disk. The environment variables involved:

• GOOGLE_MAPS_API_KEY: required for real altitudes. The Google Elevation API is billed per request; one request carries up to 512 points.
• OPENELEVATION_URL: optional. If set, an Open-Elevation server is queried first and only the points it cannot answer go to Google.
• ELEVATION_DUMMY_MODE=True: skip all lookups and store a dummy altitude of 9999 m. This is what the test suite and CI use.

API

All API Classes should inherit from eflip.ingest.base.AbstractIngester. This class also provides documentation on how to implement the required methods.

The eflips-ingest package is designed to provide a standardized access method to the different ingesters. It is designed as a reusable package providing a programmatic API. It provides the Ingester class with the following methods:

• __init__(self, database_url: str): The constructor. It takes the database URL as a parameter.
• prepare(self, progress_callback: None | Callable[[float], None] = None, **kwargs: Dict[str, str | int | float | bool | Enum | Path],) -> Tuple[bool, UUID | Dict[str, str]]:: This method is used to validate the data. It has multiple keyword arguments, one for each parameter. It returns a tuple containing a boolean indicating whether the data is valid and a UUID if the data is valid or a dictionary containing the error names and messages if the data is invalid. It is suggested that the input form is generated by introspecting this method. If a UUID is returned, this UUID can be used to start the ingestion process, even with another instance of this Ingester class (the validated input data is saved to a temporary directory).
• ingest(self, UUID: UUID, progress_callback: None | Callable[[float], None]) -> None: This method is used to start the ingestion process. It takes the UUID returned by the validate method and a callback function that will be called with the progress of the ingestion process. It is suggested that the progress is displayed in the UI. This method may raise an exception if the ingestion process fails. Note that exceptions should not be raised during the ingest process if everything goes well, as we are checking validity in the validate method.
• prepare_param_names() This property should return a dict with a short description of each parameter that the prepare method takes. This can be used for Naming fields in the UI.
• prepare_param_descriptions() This property should return a dict with a long description of each parameter that the prepare method takes. This can be used for tooltips in the UI.

In addition, the package provides a get_ingesters function that returns a list of Ingester classes.

Testing

NOTE: Be aware that the tests will clear the database specified in the DATABASE_URL environment variable. Make sure that you are not using a database that you want to keep.

Testing is done using the pytest framework with tests located in the testsdirectory. To run the tests, execute the following command in the root directory of the repository:

   export PYTHONPATH=tests:. # To make sure that the tests can find the eflips package
   export DATABASE_URL=postgis://postgres:postgres@localhost:5432/postgres # Or whatever your database URL is
   export ELEVATION_DUMMY_MODE=True # Skip altitude lookups; the tests never need real altitudes
   pytest

Development

We utilize the GitHub Flow branching structure. This means that the main branch is always deployable and that all development happens in feature branches. The feature branches are merged into main via pull requests.

We use black for code formatting. You can use pre-commit to ensure the code is formatted correctly before committing. You are also free to use other methods to format the code, but please ensure that the code is formatted correctly before committing.

Please make sure that your poetry.lock and pyproject.toml files are consistent before committing. You can use poetry check to check this. This is also checked by pre-commit.

License

This project is licensed under the AGPLv3 license - see the LICENSE file for details.

Funding Notice

This code was developed as part of the project eBus2030+ funded by the Federal German Ministry for Digital and Transport (BMDV) under grant number 03EMF0402.