A collection of import scripts for converting bus schedule data into the [eflips-model](https://github.com/mpm-tu-berlin/eflips-model) data format.
Project description
eflips-ingest
Part of the eFLIPS/simBA list of projects.
This repository contains code to import bus schedules from various sources into an eFLIPS-Model database.
Installation
-
Set up a PostgreSQL database with the PostGIS extension and
BTREE_gist
enabled.apt install postgresql postgis sudo -u postgres psql createdb eflips sudo -u postgres psql eflips -c "CREATE EXTENSION postgis;" sudo -u postgres psql eflips -c "CREATE EXTENSION btree_gist;"
-
Clone this git repository (or download a specific release)
git clone git@github.com:mpm-tu-berlin/eflips-model.git
-
Install the packages listed in
poetry.lock
andpyproject.toml
into your Python environment. Notes:- This project depends on pyproj, which may require the
proj-bin
package (apt install proj-bin
on Ubuntu). - The supported platforms are macOS and Linux.
- Using the poetry package manager is recommended. It can be installed accoring to the instructions listed here.
poetry install
- This project depends on pyproj, which may require the
Usage
Command line
The code is organized into various Python files under the eflips/ingest
folder, each for ingesting a specific data source. These files should be runnable using python eflips/ingest/x.py
The following data sources are currently supported:
bvgxml.py
: XML files emitted by BVG's proprietary software.- Requires at least
GOOGLE_MAPS_API_KEY
, alsoOPENELEVATION_URL
is suggested to save money on Google Maps API calls. - Known Limitations:
- The source data is known to contain some partial rotations of the bus routes. These are imported, which may lead to errors when creating a depot for each start and end spot, as some roations start/end at spots that are not depots.
- Some routes that are going from the same stop to the stop itself are not imported, as they are considered to be invalid.
- Some route's distance and time is fudged, when it is found to be unrealistically short or long.
- Requires at least
API
All API Classes should inherit from eflip.ingest.base.AbstractIngester
. This class also provides documentation on how to implement the required methods.
The eflips-ingest
package is designed to provide a standardized access method to the different ingesters. It is
designed as a reusable package providing a programmatic API. It provides the Ingester
class with the following
methods:
__init__(self, database_url: str)
: The constructor. It takes the database URL as a parameter.prepare(self, progress_callback: None | Callable[[float], None] = None, **kwargs: Dict[str, str | int | float | bool | Enum | Path],) -> Tuple[bool, UUID | Dict[str, str]]:
: This method is used to validate the data. It has multiple keyword arguments, one for each parameter. It returns a tuple containing a boolean indicating whether the data is valid and a UUID if the data is valid or a dictionary containing the error names and messages if the data is invalid. It is suggested that the input form is generated by introspecting this method. If a UUID is returned, this UUID can be used to start the ingestion process, even with another instance of this Ingester class (the validated input data is saved to a temporary directory).ingest(self, UUID: UUID, progress_callback: None | Callable[[float], None]) -> None
: This method is used to start the ingestion process. It takes the UUID returned by thevalidate
method and a callback function that will be called with the progress of the ingestion process. It is suggested that the progress is displayed in the UI. This method may raise an exception if the ingestion process fails. Note that exceptions should not be raised during the ingest process if everything goes well, as we are checking validity in thevalidate
method.prepare_param_names()
This property should return a dict with a short description of each parameter that theprepare
method takes. This can be used for Naming fields in the UI.prepare_param_descriptions()
This property should return a dict with a long description of each parameter that theprepare
method takes. This can be used for tooltips in the UI.
In addition, the package provides a get_ingesters
function that returns a list of Ingester
classes.
Testing
NOTE: Be aware that the tests will clear the database specified in the DATABASE_URL
environment variable. Make sure that you are not using a database that you want to keep.
Testing is done using the pytest
framework with tests located in the tests
directory. To run the tests, execute the following command in the root directory of the repository:
export PYTHONPATH=tests:. # To make sure that the tests can find the eflips package
export DATABASE_URL=postgis://postgres:postgres@localhost:5432/postgres # Or whatever your database URL is
export GOOGLE_MAPS_API_KEY=put_your_api_key_here # Required for some tests
export OPENELEVATION_URL=put_your_url_here # Optional, required for some tests
pytest
Development
We utilize the GitHub Flow branching structure. This means
that the main
branch is always deployable and that all development happens in feature branches. The feature branches
are merged into main
via pull requests.
We use black for code formatting. You can use pre-commit to ensure the code is formatted correctly before committing. You are also free to use other methods to format the code, but please ensure that the code is formatted correctly before committing.
Please make sure that your poetry.lock
and pyproject.toml
files are consistent before committing. You can use poetry check
to check this. This is also checked by pre-commit.
License
This project is licensed under the AGPLv3 license - see the LICENSE file for details.
Funding Notice
This code was developed as part of the project eBus2030+ funded by the Federal German Ministry for Digital and Transport (BMDV) under grant number 03EMF0402.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file eflips_ingest-1.2.76.tar.gz
.
File metadata
- Download URL: eflips_ingest-1.2.76.tar.gz
- Upload date:
- Size: 66.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.0 CPython/3.12.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 78b2fff3f13a3dfdc9d6d48f9e2e0236f22f735d284d2eee0a34bab9acba515b |
|
MD5 | 45c676f759532799bdbcf449addba6c7 |
|
BLAKE2b-256 | 548d5d76d5884a5520fef66b6f92dc3bcd14d989d417900a5ee53ab0260bbf27 |
File details
Details for the file eflips_ingest-1.2.76-py3-none-any.whl
.
File metadata
- Download URL: eflips_ingest-1.2.76-py3-none-any.whl
- Upload date:
- Size: 80.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.0 CPython/3.12.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7ded6f7253a049c65eb8ec53308d33ccfab006958d4f5657732f9d516374288b |
|
MD5 | 9b061bcffd9752c843aff3299a6fba90 |
|
BLAKE2b-256 | d8c3d37240d9671c3a76bfc77e690bbced46c4d5daaaadbcd81ad28f9509a805 |