synchronizer-framework 0.0.3

Bi-directional synchronization of CRUD data sources

Synchronizer Framework

This framework provides a data synchronization pattern based on a declarative bi-directional definition. The definition is a mapping of entity types with identity keys and non-key attributes which support CRUD operations and a synchronization strategy. The data sources may be arbitrarily implemented as long as they inherit from sync.base.Base. Common data sources like ORMs (django-orm, sqlalchemy) or REST APIs may be used as long as they support CRUD.

Data sources may mixin sync.base.Timestamped to apply a last-write wins synchronization strategy based on the updated_at field.

The project targets audiences which have chosen data synchronization as a primary integration method for microservices. Inspiration has been taken from projects Kubernetes and Chef.

Disclaimer: The demo and README mentions systems like Desire and Daphne. Those are no open source projects, but some target audience will understand them. The definition is supposed to cover all use-cases of the Desire architecture.

Features

• Adding missing objects
• Defer adding objects, which miss required foreign keys, because they do not exist or which are not synchronized yet
• Linking synchronized objects with each other to track them subsequently
• Many objects contributing to a single synchronized object on the remote side
• Updating non-key attributes
• Last write wins (entity needs to support sync.base.Timestamped)
• Deep-join attribute sourcing
• Abstract data access to make the pattern work with any CRUD data source (like Microsoft Dynamics, ldap, REST APIs, netconf, gNMI, CSP SDKs)
  • i.e. AWS SDK Create as AllocateHostedConnection, Read as DescribeConnections, ...
  • Data sources need to implement crud operations as defined in sync.datasource.DataSource
• pre-load/include for deep joins
• Optional deletion based on orphaned entities
• Possibility to expedite single object synchronization based on an event stream
  • Celery support
• SQLAlchemy backend support
• Redis linking backend support
• dockerized

More details about the pattern can be found in the tests path.

Missing/TODO

• Deletion based on intent (with cryptographic verification)

Known Limitations

1. Cannot synchronize three-way at once.

   Example:

   Assume entities PA required parent of A, BA required parent of B, CA required parent of C. Mappings are defined as M1: A -> B and M2: B <- C. The pattern prevents each M1 and M2 to progress independently. For M1 , information about PC is missing and for M2 information about PA is missing. Further, assume x(e) to denote an entity e transformed to the namespace of x.

   Solution:

   • Introduce an intermediary entity I which accepts optional parents I(PA) and I(PC)
   • remove M1 and M2
   • create M3: A -> I, M4: I <- C
   • create M5: I -> B

   The sequence of M3, M4, M5 leads to a synchronized B containing both parents B(PA) and B(PC). The order of events always keeps consistent states. For example, M5 may never progress before M4 even if it is executed before.

   The solution is not limited to uni-directional mappings, but the author chose demonstrating uni-directional mappings for easier understanding of how data flows.

2. The orphan deletion may be counter-intuitive and will be changed in a future version. Refrain from using it, if you do not understand the consequences.

Define Entities

Entities below resemble the desire and daphne entity hierarchy and deviate to some extent. The intention is to demonstrate the synchronization pattern instead of resembling existing models accurately. The details which are left out of consideration seem to be straight-forward to implement.

Base

class DemoBase(InMemoryLinks, Hierarchical, InMemoryTimestamp):
    ...

Daphne

class VlanServiceProvider(DemoBase):
    def __init__(self, name=None, children=set()):
        super().__init__()
        self.name = name
        self.children = children


class VlanServiceConnection(DemoBase):
    def __init__(self, name, region, pop):
        super().__init__()
        self.name = name
        self.region = region
        self.pop = pop

Desire

class CloudServiceProvider(DemoBase):
    def __init__(self, name=None, children=None):
        super().__init__()
        self.name = name
        self.children = children or set()


class CloudRegion(DemoBase):
    def __init__(self, name=None, children=None):
        super().__init__()
        self.name = name
        self.children = children or set()


class CloudHandover(DemoBase):
    def __init__(self, name=None, children=None):
        super().__init__()
        self.name = name
        self.children = children or set()


class NIC(DemoBase):
    def __init__(self, name=None):
        super().__init__()
        self.name = name

Define Mapping

mappings = [
   Mapping(
      entity_types=(VlanServiceConnection, CloudRegion),
      modes={Mode.LEFT_TO_RIGHT},
      keys={
         AttributeMap("region", "name"),
         AttributeMap(
            "parent", "parent", VlanServiceProvider, CloudServiceProvider
         ),
      },
      attributes={AttributeMap("externalRef", "external_ref")},
   ),
   Mapping(
      entity_types=(VlanServiceProvider, CloudServiceProvider),
      modes={Mode.LEFT_TO_RIGHT, Mode.RIGHT_TO_LEFT},
      keys={AttributeMap("name", "name")},
      attributes={AttributeMap("canUpgrade", "upgrade_allowed")},
   ),
   Mapping(
      entity_types=(VlanServiceConnection, CloudHandover),
      modes={Mode.LEFT_TO_RIGHT},
      keys={
         AttributeMap("pop", "name"),
         AttributeMap("__self__", "parent", VlanServiceConnection, CloudRegion),
      },
      attributes={AttributeMap("externalRef", "external_ref")},
   ),
   Mapping(
      entity_types=(VlanServiceConnection, NIC),
      modes={Mode.RIGHT_TO_LEFT},
      keys={
         AttributeMap("name", "name"),
         AttributeMap("region", ("parent", "parent", "name")),
         AttributeMap("pop", ("parent", "name")),
         AttributeMap(
            "parent",
            ("parent", "parent", "parent"),
            VlanServiceProvider,
            CloudServiceProvider,
         ),
      },
      attributes={AttributeMap("externalRef", "external_ref")},
   ),
   Mapping(
      entity_types=(VlanServiceConnection, NIC),
      modes={Mode.LEFT_TO_RIGHT},
      keys={
         AttributeMap("name", "name"),
         AttributeMap(
            "__self__", "parent", VlanServiceConnection, CloudHandover
         ),
      },
      attributes={AttributeMap("externalRef", "external_ref")},
   ),
]

Define some objects

daphne_db = Db.from_collection(
    {
        VlanServiceProvider(
            name="AWS",
            children={
                VlanServiceConnection("nic-1", "eu-central-1", "INX6"),
                VlanServiceConnection("nic-2", "eu-central-1", "EqFA5"),
                VlanServiceConnection("nic-3", "eu-west-2", "EqFA5"),
            },
        ),
        VlanServiceProvider(name="AZURE", children=set()),
    }
)

desire_db = Db.from_collection(
    {
        CloudServiceProvider(
            name="AWS",
            children={
                CloudRegion(
                    name="eu-central-1",
                    children={
                        CloudHandover(name="INX6", children={NIC("nic-1")}),
                        CloudHandover(name="EqFA5", children={NIC("nic-2a")}),
                    },
                ),
                CloudRegion(
                    name="eu-west-1",
                    children={
                        CloudHandover(name="LON1", children={NIC(name="nic-3a")})
                    },
                ),
                CloudRegion(
                    name="eu-west-2",
                    children={
                        CloudHandover(
                            name="EqFA5",
                            children={NIC(name="nic-3", external_ref="nic3-eref")},
                        )
                    },
                ),
            },
        ),
        CloudServiceProvider(
            name="IBM",
            children={
                CloudRegion(
                    name="EU-Frankfurt",
                    children={CloudHandover(name="fra03", children={NIC("nic-4")})},
                )
            },
        ),
    }
)

Plug it together

sync(daphne_db, desire_db, mappings)

Demo Output

$ python src/demo/main.py
/home/sspies/git/sync-poc/venv/bin/python3 /home/sspies/git/sync-poc/src/demo/main.py
INFO:root:Initial DB states
INFO:root:{<class 'demo.desire.CloudServiceProvider'>: [CloudServiceProvider(name=IBM),
                                              CloudServiceProvider(name=AWS)],
 <class 'demo.desire.CloudRegion'>: [CloudRegion(name=EU-Frankfurt, parent=CloudServiceProvider(name=IBM)),
                                     CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)),
                                     CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS)),
                                     CloudRegion(name=eu-west-1, parent=CloudServiceProvider(name=AWS))],
 <class 'demo.desire.CloudHandover'>: [CloudHandover(name=fra03, parent=CloudRegion(name=EU-Frankfurt)),
                                       CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)),
                                       CloudHandover(name=INX6, parent=CloudRegion(name=eu-central-1)),
                                       CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1)),
                                       CloudHandover(name=LON1, parent=CloudRegion(name=eu-west-1))],
 <class 'demo.desire.NIC'>: [NIC(name=nic-4, parent=CloudHandover(name=fra03)),
                             NIC(name=nic-3, parent=CloudHandover(name=EqFA5)),
                             NIC(name=nic-1, parent=CloudHandover(name=INX6)),
                             NIC(name=nic-2a, parent=CloudHandover(name=EqFA5)),
                             NIC(name=nic-3a, parent=CloudHandover(name=LON1))]}
INFO:root:{<class 'demo.daphne.VlanServiceProvider'>: [VlanServiceProvider(name=AZURE),
                                             VlanServiceProvider(name=AWS)],
 <class 'demo.daphne.VlanServiceConnection'>: [VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6),
                                               VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5),
                                               VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)]}
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6).parent. VlanServiceProvider(name=AWS) not in CloudServiceProvider
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5).parent. VlanServiceProvider(name=AWS) not in CloudServiceProvider
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5).parent. VlanServiceProvider(name=AWS) not in CloudServiceProvider
INFO:root:✨ Adding CloudServiceProvider(name=AZURE)
INFO:root:✨ Adding VlanServiceProvider(name=IBM)
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6).__self__. VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6) not in CloudRegion
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5).__self__. VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5) not in CloudRegion
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5).__self__. VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5) not in CloudRegion
INFO:root:✨ Adding VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)
INFO:root:✨ Adding VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)
INFO:root:✨ Adding VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6).__self__. VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6) not in CloudHandover
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5).__self__. VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5) not in CloudHandover
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5).__self__. VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5) not in CloudHandover
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03).__self__. VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03) not in CloudHandover
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5).__self__. VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5) not in CloudHandover
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1).__self__. VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1) not in CloudHandover
INFO:root:27 operation(s) made
INFO:root:Operations: [SkipMissingLink(VlanServiceConnection -> CloudRegion),
 SkipMissingLink(VlanServiceConnection -> CloudRegion),
 SkipMissingLink(VlanServiceConnection -> CloudRegion),
 Addition(CloudServiceProvider(name=AZURE)),
 Linking(frozenset({VlanServiceProvider(name=AZURE), CloudServiceProvider(name=AZURE)})),
 Linking(frozenset({VlanServiceProvider(name=AWS), CloudServiceProvider(name=AWS)})),
 AttributeUpdated(CloudServiceProvider(name=AWS).upgrade_allowed: True -> False from VlanServiceProvider(name=AWS)),
 Addition(VlanServiceProvider(name=IBM)),
 Linking(frozenset({VlanServiceProvider(name=IBM), CloudServiceProvider(name=IBM)})),
 SkipMissingLink(VlanServiceConnection -> CloudHandover),
 SkipMissingLink(VlanServiceConnection -> CloudHandover),
 SkipMissingLink(VlanServiceConnection -> CloudHandover),
 Addition(VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)),
 Linking(frozenset({NIC(name=nic-4, parent=CloudHandover(name=fra03)), VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)})),
 Linking(frozenset({NIC(name=nic-3, parent=CloudHandover(name=EqFA5)), VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)})),
 AttributeUpdated(VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5).externalRef:  -> nic3-eref from NIC(name=nic-3, parent=CloudHandover(name=EqFA5))),
 Linking(frozenset({VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6), NIC(name=nic-1, parent=CloudHandover(name=INX6))})),
 Addition(VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)),
 Linking(frozenset({NIC(name=nic-2a, parent=CloudHandover(name=EqFA5)), VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)})),
 Addition(VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)),
 Linking(frozenset({NIC(name=nic-3a, parent=CloudHandover(name=LON1)), VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)})),
 SkipMissingLink(VlanServiceConnection -> NIC),
 SkipMissingLink(VlanServiceConnection -> NIC),
 SkipMissingLink(VlanServiceConnection -> NIC),
 SkipMissingLink(VlanServiceConnection -> NIC),
 SkipMissingLink(VlanServiceConnection -> NIC),
 SkipMissingLink(VlanServiceConnection -> NIC)]
INFO:root:DB states after first and before second run
INFO:root:{<class 'demo.desire.CloudServiceProvider'>: [CloudServiceProvider(name=IBM),
                                              CloudServiceProvider(name=AWS),
                                              CloudServiceProvider(name=AZURE)],
 <class 'demo.desire.CloudRegion'>: [CloudRegion(name=EU-Frankfurt, parent=CloudServiceProvider(name=IBM)),
                                     CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)),
                                     CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS)),
                                     CloudRegion(name=eu-west-1, parent=CloudServiceProvider(name=AWS))],
 <class 'demo.desire.CloudHandover'>: [CloudHandover(name=fra03, parent=CloudRegion(name=EU-Frankfurt)),
                                       CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)),
                                       CloudHandover(name=INX6, parent=CloudRegion(name=eu-central-1)),
                                       CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1)),
                                       CloudHandover(name=LON1, parent=CloudRegion(name=eu-west-1))],
 <class 'demo.desire.NIC'>: [NIC(name=nic-4, parent=CloudHandover(name=fra03)),
                             NIC(name=nic-3, parent=CloudHandover(name=EqFA5)),
                             NIC(name=nic-1, parent=CloudHandover(name=INX6)),
                             NIC(name=nic-2a, parent=CloudHandover(name=EqFA5)),
                             NIC(name=nic-3a, parent=CloudHandover(name=LON1))]}
INFO:root:{<class 'demo.daphne.VlanServiceProvider'>: [VlanServiceProvider(name=AZURE),
                                             VlanServiceProvider(name=AWS),
                                             VlanServiceProvider(name=IBM)],
 <class 'demo.daphne.VlanServiceConnection'>: [VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6),
                                               VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5),
                                               VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5),
                                               VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03),
                                               VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5),
                                               VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)]}
INFO:root:✨ Adding NIC(name=nic-2, parent=CloudHandover(name=EqFA5))
INFO:root:Sync with 16 operation(s) made
INFO:root:Operations: [Linking(frozenset({VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6), CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS))})),
 Linking(frozenset({CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)), VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)})),
 AttributeUpdated(CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)).external_ref:  -> nic3-eref from VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)),
 Linking(frozenset({CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS)), VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)})),
 Linking(frozenset({CloudRegion(name=EU-Frankfurt, parent=CloudServiceProvider(name=IBM)), VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)})),
 Linking(frozenset({VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5), CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS))})),
 Linking(frozenset({VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1), CloudRegion(name=eu-west-1, parent=CloudServiceProvider(name=AWS))})),
 Linking(frozenset({VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6), CloudHandover(name=INX6, parent=CloudRegion(name=eu-central-1))})),
 Linking(frozenset({CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)), VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)})),
 AttributeUpdated(CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)).external_ref:  -> nic3-eref from VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)),
 Linking(frozenset({CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1)), VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)})),
 Linking(frozenset({CloudHandover(name=fra03, parent=CloudRegion(name=EU-Frankfurt)), VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)})),
 Linking(frozenset({VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5), CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1))})),
 Linking(frozenset({CloudHandover(name=LON1, parent=CloudRegion(name=eu-west-1)), VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)})),
 Addition(NIC(name=nic-2, parent=CloudHandover(name=EqFA5))),
 Linking(frozenset({NIC(name=nic-2, parent=CloudHandover(name=EqFA5)), VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)}))]
INFO:root:🙌 Final DB states
INFO:root:{<class 'demo.desire.CloudServiceProvider'>: [CloudServiceProvider(name=IBM),
                                              CloudServiceProvider(name=AWS),
                                              CloudServiceProvider(name=AZURE)],
 <class 'demo.desire.CloudRegion'>: [CloudRegion(name=EU-Frankfurt, parent=CloudServiceProvider(name=IBM)),
                                     CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)),
                                     CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS)),
                                     CloudRegion(name=eu-west-1, parent=CloudServiceProvider(name=AWS))],
 <class 'demo.desire.CloudHandover'>: [CloudHandover(name=fra03, parent=CloudRegion(name=EU-Frankfurt)),
                                       CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)),
                                       CloudHandover(name=INX6, parent=CloudRegion(name=eu-central-1)),
                                       CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1)),
                                       CloudHandover(name=LON1, parent=CloudRegion(name=eu-west-1))],
 <class 'demo.desire.NIC'>: [NIC(name=nic-4, parent=CloudHandover(name=fra03)),
                             NIC(name=nic-3, parent=CloudHandover(name=EqFA5)),
                             NIC(name=nic-1, parent=CloudHandover(name=INX6)),
                             NIC(name=nic-2a, parent=CloudHandover(name=EqFA5)),
                             NIC(name=nic-3a, parent=CloudHandover(name=LON1)),
                             NIC(name=nic-2, parent=CloudHandover(name=EqFA5))]}
INFO:root:{<class 'demo.daphne.VlanServiceProvider'>: [VlanServiceProvider(name=AZURE),
                                             VlanServiceProvider(name=AWS),
                                             VlanServiceProvider(name=IBM)],
 <class 'demo.daphne.VlanServiceConnection'>: [VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6),
                                               VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5),
                                               VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5),
                                               VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03),
                                               VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5),
                                               VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)]}
INFO:root:🎉🥳 Sync with 0 operations made. Synchronization has converged
INFO:root:Operations: []

Process finished with exit code 0

Celery Demo

Start the compose setup

❯ docker-compose up -d --build
Creating network "synchronizer-framework_default" with the default driver
Building worker
Step 1/11 : FROM python:3.8-slim
 ---> b281745b6df9
Step 2/11 : RUN apt-get update ; apt-get install -y sqlite3 vim ; apt-get clean
 ---> Using cache
 ---> 25b681e3e49c
Step 3/11 : WORKDIR /app/synchronizer-framework
 ---> Using cache
 ---> 20c652ac628d
Step 4/11 : COPY ./requirements.txt .
 ---> Using cache
 ---> 9a1f9aa5c9a4
Step 5/11 : RUN pip install -r requirements.txt
 ---> Using cache
 ---> ae9c400626e5
Step 6/11 : COPY src/ ./src
 ---> Using cache
 ---> be73849dfe6f
Step 7/11 : COPY tests ./tests
 ---> Using cache
 ---> 3e59bd21b326
Step 8/11 : COPY conftest.py .
 ---> Using cache
 ---> 3c06b0dcc7f9
Step 9/11 : ENV C_FORCE_ROOT=true
 ---> Using cache
 ---> 2a187de06fcd
Step 10/11 : ENV PYTHONPATH="src/"
 ---> Using cache
 ---> 027396f1f6f9
Step 11/11 : CMD [ "python", "-m", "demo.main"]
 ---> Using cache
 ---> b8b1c92054b7
Successfully built b8b1c92054b7
Successfully tagged synchronizer-framework_worker:latest
Building beat
Step 1/11 : FROM python:3.8-slim
 ---> b281745b6df9
Step 2/11 : RUN apt-get update ; apt-get install -y sqlite3 vim ; apt-get clean
 ---> Using cache
 ---> 25b681e3e49c
Step 3/11 : WORKDIR /app/synchronizer-framework
 ---> Using cache
 ---> 20c652ac628d
Step 4/11 : COPY ./requirements.txt .
 ---> Using cache
 ---> 9a1f9aa5c9a4
Step 5/11 : RUN pip install -r requirements.txt
 ---> Using cache
 ---> ae9c400626e5
Step 6/11 : COPY src/ ./src
 ---> Using cache
 ---> be73849dfe6f
Step 7/11 : COPY tests ./tests
 ---> Using cache
 ---> 3e59bd21b326
Step 8/11 : COPY conftest.py .
 ---> Using cache
 ---> 3c06b0dcc7f9
Step 9/11 : ENV C_FORCE_ROOT=true
 ---> Using cache
 ---> 2a187de06fcd
Step 10/11 : ENV PYTHONPATH="src/"
 ---> Using cache
 ---> 027396f1f6f9
Step 11/11 : CMD [ "python", "-m", "demo.main"]
 ---> Using cache
 ---> b8b1c92054b7
Successfully built b8b1c92054b7
Successfully tagged synchronizer-framework_beat:latest
Creating synchronizer-framework_worker_1   ... done
Creating synchronizer-framework_beat_1     ... done
Creating synchronizer-framework_redis_1    ... done
Creating synchronizer-framework_rabbitmq_1 ... done

Show worker logs

worker_1    | [2021-04-21 18:16:06,166: INFO/MainProcess] Received task: sync.tasks.sync_all[281f9bc5-6ac8-4355-864e-692995abca5c]  
worker_1    | [2021-04-21 18:16:06,341: INFO/ForkPoolWorker-2] Task sync.tasks.sync_all[281f9bc5-6ac8-4355-864e-692995abca5c] succeeded in 0.17173888799879933s: [Linking(frozenset({VlanServiceConnection(id=1, name=nic-1, region=eu-central-1, pop=INX6, externalRef=None, parent=VlanServiceProvider(id=2, name=AWS, canUpgrade=False)), CloudRegion(id=2, name=eu-central-1, external_ref=None, parent=CloudServiceProvider(id=2, name=AWS, upgrade_allowed=False))})), Linking(frozenset({CloudRegion(id=2, name=eu-central-1, external_ref=None, parent=CloudServiceProvider(id=2, name=AWS, upgrade_allowed=False)), VlanServiceConnection(id=2, name=nic-2, region=eu-central-1, pop=EqFA5, externalRef=None, parent=VlanServiceProvider(id=2, name=AWS, canUpgrade=False))})), Linking(frozenset({VlanServiceConnection(id=3, name=nic-3, region=eu-west-2, pop=EqFA5, externalRef=nic3-eref, parent=VlanServiceProvider(id=2, name=AWS, canUpgrade=False)), CloudRegion(id=4, name=eu-west-2, external_ref=None, parent=CloudServiceProvider(id=2, name=AWS, upgrade_allowed=False))})), Linking(frozenset({CloudRegion(id=1, name=EU-Frankfurt, external_ref=None, parent=CloudServiceProvider(id=1, name=IBM,..., Li..., ...]
worker_1    | [2021-04-21 18:16:16,165: INFO/MainProcess] Received task: sync.tasks.sync_all[64b780e0-b5e3-4771-913d-de3f54b2ccec]  
worker_1    | [2021-04-21 18:16:16,300: INFO/ForkPoolWorker-2] Task sync.tasks.sync_all[64b780e0-b5e3-4771-913d-de3f54b2ccec] succeeded in 0.13250631099799648s: []

Issue a new object and synchronize it immediately

from demo.daphne import VlanServiceProvider
from demo.fixtures import daphne_sqa_ds, desire_sqa_ds, make_mappings
from demo.celery import tasks

if __name__ == "__main__":
   ds = daphne_sqa_ds()
   vlsp = VlanServiceProvider(name="test")
   ds.create(vlsp)

   tasks.sync_one.delay(
      daphne_sqa_ds, desire_sqa_ds, make_mappings(), vlsp.__class__, vlsp.id
   )

 docker-compose exec worker python src/demo/create_and_sync.py

Worker log

worker_1    | [2021-04-21 18:22:12,938: INFO/MainProcess] Received task: sync.tasks.sync_one[11dfd53a-1198-47e8-b624-25218b463660]  
worker_1    | [2021-04-21 18:22:12,957: INFO/ForkPoolWorker-2] ✨ Adding CloudServiceProvider(id=4, name=test, upgrade_allowed=None)
worker_1    | [2021-04-21 18:22:12,959: INFO/ForkPoolWorker-2] Task sync.tasks.sync_one[11dfd53a-1198-47e8-b624-25218b463660] succeeded in 0.02017242200236069s: [Addition(CloudServiceProvider(id=4, name=test, upgrade_allowed=None)), Linking(frozenset({CloudServiceProvider(id=4, name=test, upgrade_allowed=None), VlanServiceProvider(id=4, name=test, canUpgrade=None)}))]

Show link and synchronized entry

❯ docker-compose exec redis redis-cli smembers VlanServiceProvider:4:CloudServiceProvider
1) "CloudServiceProvider:4"
❯ docker-compose exec worker sqlite3 /tmp/desire.sqlite "SELECT * FROM cloud_service_providers where id = 4;"
4|test|