django-ixp-tracker

Library to retrieve and manipulate data about IXPs

v3 warning

v3 completely removes the legacy v1 implementation and hence will delete all the old models from the db. You might want to backup your db before installing this version.

v2 warning

Note the v2 is a complete rewrite to use Event Sourcing. The first time you upgrade you will need to re-run any historical data you have as the history is not migrated between major versions.

v2 does not by default preserve any internal ids for IXPs. If you need to preserve them, you will need to preload the IXPIdMap model table with the mapping to Peering DB ids (use a dummy aggregate_id), e.g. if you use Postgres, by running these queries

INSERT INTO ixp_tracker_ixpidmap (id, peeringdb_id, aggregate_id)
SELECT id, peeringdb_id, concat('aaaabbbb-cccc-dddd-eeee-', lpad(id::text, 12, '0'))::uuid FROM ixp_tracker_ixp;

SELECT setval(pg_get_serial_sequence('ixp_tracker_ixpidmap', 'id')
           , COALESCE(max(id) + 1, 1)
           , false)
FROM   ixp_tracker_ixpidmap;

The legacy implementation is preserved in v2 but is removed in v3. If you would like to upgrade to v2 but continue with the legacy implementation be sure to set IXP_TRACKER_ENABLE_EVENT_SOURCING to False. Note that there are no changes to the legacy implementation in v2 though.

Installation

Install this library using pip:

pip install django-ixp-tracker

Usage

1. Add to your INSTALLED_APPS setting like this:

   INSTALLED_APPS = [
   ...,
   "ixp_tracker",
   ]

Note: this app has no web-facing components so you don't need to add anything to urls.py etc

2. Run python manage.py migrate to create the models.
3. Add the relevant settings to your config.
4. Add IXP_TRACKER_DATA_LOOKUP_FACTORY to config with the path to your factory (see below).
5. Run the management command to import the data: python manage.py ixp_tracker_import (This will sync the current data, if you want historical data you need to backfill first)

ASN country and status data

The lib uses an external component to look up the country of registration (why?) and the status of an ASN. This status is used for the logic to identify when members have left an IXP.

If you don't provide this service yourself, it will default to a noop version. This will mean you will get no country of registration data and the marking of members having left an IXP will not be as efficient.

In order to implement such a component yourself, you should implement the Protocol ixp_tracker.data_lookup.AdditionalDataSources and provide a factory function for your class.

Backfilling data

You have the option of backfilling data from archived PeeringDb data. This can be done by running the import command with the --backfill option for each month you want to backfill:

python manage.py ixp_tracker_import --backfill <YYYMM>

The backfill currently process a single month at a time and will look for the earliest file for the relevant month at https://publicdata.caida.org/datasets/peeringdb/

IMPORTANT NOTE: due to the way the code tries to figure out when a member left an IXP, you should run the backfill strictly in date order and before syncing the current data.

IXP stats

The import process also generates monthly stats per IXP and per country. These are generated as of the 1st of the month used to import the data.

Running programmatically

If you'd like to run the import from code, rather than from the management command, you can call importers.import_data() and stats.generate_stats() directly.

It's not recommended to call any other functions yourself.

Development

To contribute to this library, first checkout the code. Then create a new virtual environment:

cd django-ixp-tracker
python -m venv .venv
source .venv/bin/activate

Now install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

pytest

We use pre-commit for linting etc on push. Run:

pre-commit install

from the repo top-level dir to set it up.

Releases

For now, releasing a new version is manual and can be done by running the following commands from the repo:

python -m build
python -m twine upload --repository pypi dist/*

Peering Db libraries

PeeringDb provide their own vanilla Python and Django libs, but we have decided not to use these.

Both libs are designed to keep a local copy of the current data and to keep that copy in sync with the central copy via the API.

As we need to keep a historical record (e.g. for IXP growth stats over time), we would have to provide some sort of wrapper over those libs anyway.

In addition to that, the historical archives of PeeringDb data use flat lists of the different object types in json. We can retrieve the data from the API directly in the same way, so it makes it simpler to implement.