DearMEP Server

This is the server side of DearMEP. It provides a RESTful API and employs

• FastAPI as its application framework,
• SQLite as its zero-maintenance, self-contained database,
• SQLAlchemy for accessing that database,
• ffmpeg for converting audio,

as well as several other libraries to do its job.

Features

• Pydantic-powered rigorous validation of user-supplied data
• Fully configurable rate limiting and telephony subsystem to protect against malicious users and manage spending
• Prometheus metrics to observe the system's health
• Convenient command line interface and composable import and export tools
• Privacy-first design, keeping information stored about users to a minimum

Installation

DearMEP is available on PyPI and can be installed and run using normal Python packaging tools, e.g. pip install dearmep or uvx dearmep.

The packages include a matching version of the client and are able to serve it via HTTP, see serving static files below.

Installation for Development

1. Install uv.
2. Clone this repository and cd into it.
3. Run uv sync --all-extras.
   • See below on which extras this will install. Alternatively, you can select individual extras with --extra NAME or don’t install any extras at all.
4. Done.

uv takes care of managing a virtual environment in .venv/ for you, containing DearMEP and all of its dependencies, without messing with your global installation. You can update your $PATH accordingly to access the commands (including the dearmep command) available in .venv/bin. We recommend direnv to automate this. Alternatively, you can prefix individual commands with uv run to run just that command inside of the virtual environment.

In the examples below, commands like dearmep or uvicorn should either be run using the command from .venv/bin, or prefixed like uv run dearmep.

Extras

Python packages can have optional dependencies, called extras. These can provide additional, optional features. DearMEP defines the following extras:

• convert: Additional tools for converting different data formats into each other, or general data manipulation. Can be very helpful for importing & exporting data like MEPs, statistics or swayability scores into and out of DearMEP. Will install csvkit and VisiData.
• specs: Dependencies for extended specifications & documentation. Right now, this is just eralchemy2 to generate entity relationship diagrams using dearmep dump erd. You probably don't need this.

Command Line Interface

If you have installed DearMEP correctly, you will have access to the dearmep command in your shell. It provides several subcommands to interact with your DearMEP installation or help you convert data.

Like any good command line tool, it comes with a --help argument that will show you all available commands and options. Use --help with a subcommand to get more information about that subcommand, e.g. dearmep dump --help.

The most accurate documentation about the command line interface is always what --help gives you, but here is a small overview of the subcommands:

• serve: Start an HTTP server that provides the actual DearMEP API.
• convert: Several tools to convert data into into formats that DearMEP can use, or intermediate formats for further manipulation.
• import: Import Destinations (people to contact, e.g. MEPs) or scoring information into the database.
• db: Check the database for issues, add files to the blob storage.
• alembic: Launch a pre-configured version of the Alembic CLI, i.e. with ALEMBIC_CONFIG already pointing to our configuration.
• dump: Print example configuration files and specifications.
• version: Print the version numbers of Python, DearMEP, and its most important libraries.
• check: Run checks against your configuration, e.g. whether all strings are translated in all languages.

Providing a Configuration File

Please set the environment variable DEARMEP_CONFIG (which defaults to config.yaml) to the name of a YAML file containing the configuration. See example-config.yaml for an extensively commented example.

You can use dearmep dump example-config to produce a config file, which you can then modify according to your needs. Make sure to at least change the secrets, i.e. authentication.secrets.pepper and authentication.secrets.jwt.key.

Providing MEPs & Scoring

DearMEP does not come with a built-in list of Members of Parliament. That list would change quite often, and you might only want to offer your users a subset of them, or an entirely different group of people and not MEPs at all. (For that reason, we often call them Destinations instead, i.e. people that can be called. For more on some of our special terms, please see the glossary.)

Also, one of DearMEP's key features is that you can configure, for each Destination, how important it is to contact them. Usually, you'd want to focus on MEPs that are still undecided on the policy your campaign attempts to influence, and not waste time and money calling people who are guaranteed to oppose you (or guaranteed to support you). Therefore, DearMEP uses a concept we call Swayability to calculate a kind of scoring to determine the priority assigned to a Destination. Of course, these criteria vary wildly between the policy issue at hand, and therefore you need to provide DearMEP with your desired scores as well.

We have a separate document on how to convert data for use in DearMEP that talks about tools and methods in general, but also how to get a list of MEPs from publicly available data.

Importing Swayability scores is done using the dearmep import swayability command. Calling it with --help provides information about the CSV format it expects. The scoring algorithm is explained and configured in DearMEP's config file; look for the recommender section.

Providing IVR Audio

When DearMEP calls a User, they are presented with a pretty standard audio-based IVR menu. Of course the menu needs audio files to play.

We have a separate document explaining how the menu works and which files to supply, and we provide prerecorded IVR audio in several languages in the DearMEP media repository. However, you will still need to record at least a few messages to customize them to your own Campaign. Also, while the media repo provides the files as .wav recordings for maximum fidelity and lossless editing, DearMEP expects them in Ogg Vorbis format. See the data conversion document's section on converting the audio files for details.

Database Migrations

DearMEP is using Alembic to manage the database and perform version upgrades to it. Currently, this is a manual process.

Note that Alembic requires a configuration file telling it where to find the database and migrations. Fortunately, DearMEP comes with a prepared alembic.ini as well as a special CLI subcommand, dearmep alembic, that will invoke Alembic with the environment variable ALEMBIC_CONFIG already pointing to the location of this configuration file.

For Administrators of a DearMEP Instance

If you upgrade DearMEP, make sure to run dearmep alembic upgrade head before starting the dearmep serve process again. It will ensure that all changes to the database structure required for the new version are applied. Make sure to have a backup of your database, just in case.

For Developers

If you change the models which are reflected in the database, you'll need to do use Alembic to handle database migrations. You can create migrations via

dearmep alembic revision --autogenerate --message "your short alembic message about the reason of this migration"

Alembic generates a file in dearmep/migrations/versions/. Check this file for sanity. You can edit or delete this file though this usually should not be necessary. If you are happy about the migration, commit to version control.

Make sure to upgrade the database to the latest revision when developing. It is recommended to check the upgrades first, for example by making a copy of the database source and changing the sqlalchemy.uri in the alembic.ini or by checking the raw SQL with the --sql flag.

Please check Alembic's documentation for more.

Running a Development Server

dearmep serve --log-level debug --reload

You can modify the port with -p 1234. By default, DearMEP will serve on port 8000.

See dearmep serve --help for other options, including how to start it via a custom ASGI server.

Accessing Development Tools

You can use the Makefile to access recommended development commands easily.

• make sync to sync your environment, install dependencies, etc. This will run uv sync --all-extras.
• make fmt runs the Ruff formatter on the code.
• make lint runs Ruff's linter.
• make types performs static type checking with mypy.
• make test will run the testsuite using pytest.
• make trans checks for missing translations.

There are also some shortcuts for combining these operations:

• make flint runs formatter and linter. This is super fast and can be done on every save, for example.
• make qa runs the more expensive things: type checking, testsuite, translations.
• And a simple make without any arguments runs all of the above. You can even use make -j to do all of it in parallel and save a few seconds.

Serving Static Files (e.g. the Client)

If you set the environment variable DEARMEP_STATIC_FILES_DIR to a directory path, DearMEP will provide the contents of that directory at /static/. This can be used to serve the JavaScript, CSS and assets of the client snippet. There are no directory indexes and there is no index.html support.

Pre-built releases of DearMEP come bundled with a client snippet (in dearmep/static_files/static), and their DEARMEP_STATIC_FILES_DIR will default to that bundled snippet. (Set the environment variable to an empty string to disable the bundled snippet.)

Even if you use this feature, you are still expected to run a reverse proxy server for additional stability. Also, while the static files will be served with Last-Modified and ETag headers, there is no Cache-Control header; you should configure this in the reverse proxy.

If you set DEARMEP_DEMO_PAGE=y, a basic demo HTML page will be returned at the HTTP root. This option only has an effect if you have also configured DEARMEP_STATIC_FILES_DIR (or you are using a bundled snippet).

Serving Markdown Files

In addition to static files, as described in the previous section, DearMEP can also render and serve Markdown files, in multiple languages, with simple templating support. This can for example be used to serve a privacy policy or other simple documents.

To use this feature, set the environment variable DEARMEP_MARKDOWN_FILES_DIR to a directory path. This directory will need to contain the following structure:

docs
| `- privacy  (1)
|    | `- en.md
|    |    de.md
|    |    …
|    another_folder  (1)
|      `- en.md
|         de.md
|         …
static
| `- styles.css
|    …
templates
  `- default.html.jinja

The folders marked (1) can have any name. However, they have to contain Markdown files named {language}.md, where {language} is one of the configured languages of the instance. These documents will then be served at /pages/{folder_name}/{language}/ (including the trailing slash).

The files in the static directory will be available at /pages/{path}, e.g. /pages/styles.css. Subdirectories below static will be ignored, in order to prevent ambiguousness with Markdown documents.

The templates directory needs to contain a file named default.html.jinja with a Jinja2 HTML template. The Markdown documents will be passed to this document in a content variable. Additional available variables are:

• title: The first level-1 heading of the Markdown document, useful for placing in an HTML <title> element.
• base_path: Usually /pages/. Should be used as a prefix to refer to files in static, e.g. the stylesheet.
• language: The {language} part of the URL, can be used for setting <html lang="…">.

Note: The generated HTML is cached in memory. If you modify the document or the Jinja template after its initial render, these modifications will not show up in the HTML output until DearMEP is restarted.

The example-markdown directory contains, as the name suggests, an example setup.

Retrieving the OpenAPI specification

A running DearMEP server will provide its OpenAPI specification at /openapi.json and GUIs for it at /docs and /redoc.

To quickly dump the OpenAPI spec to stdout, use dearmep dump openapi.

You can also view the specification of the main branch (i.e. the current development version) online, in ReDoc or in Swagger UI.

Running Behind a Reverse Proxy

If you run DearMEP behind a reverse proxy like Caddy, nginx or Træfik, make sure to provide the remote client’s original IP address via the X-Forwarded-For header. This is required for the geolocation to work.

For nginx, and if you’re using dearmep serve, check out Uvicorn’s best practices. dearmep serve currently doesn’t support listening to a Unix domain socket, so please use a normal TCP HTTP connection instead (e.g. proxy_pass http://localhost:8000/;).

Prometheus Metrics

DearMEP exports Prometheus metrics on the usual /metrics endpoint, providing you with insight about the health of the system as well as statistics. The available metrics are documented on a separate page.

Note that you probably don't want to allow public access to these metrics. They don't contain user data, but do expose information about the phone numbers DearMEP itself is using to call people, which destination is being called how often and for how long, as well as cost statistics.