meepo 0.1.9

event sourcing for databases.

Meepo is event sourcing and event broadcasting for databases.

Documentation: https://meepo.readthedocs.org/

Installation

Install with pip

$ pip install meepo

To use mysql binlog as event source, install additional requires with

$ pip install meepo[mysqlbinlog]

Features

Meepo can publish the table events from database to the outside, so it can be used to do a lot of interesting things, including:

• cache invalidation

• replication to RDBS / NoSQL / SearchEngine

• event sourcing

• logging and auditing

• realtime analytics

• notifications pushing

Intro

meepo is event sourcing and event broadcasting for databases, it follows database table events from sqlalchemy and mysql binlog, and publish them to subscribers.

MySQL Pub

mysql_pub event flow:

                                                   +---------------------+
                                                   |                     |
                                               +--->  table_write event  |
                                               |   |                     |
                                               |   +---------------------+
                                               |
+--------------------+      +---------------+  |
|                    |      |               |  |   +---------------------+
|        mysql       |      |   meepo.pub   |  |   |                     |
|                    +------>               +--+--->  table_update event |
|  row-based binlog  |      |   mysql_pub   |  |   |                     |
|                    |      |               |  |   +---------------------+
+--------------------+      +---------------+  |
                                               |
                                               |   +---------------------+
                                               |   |                     |
                                               +--->  table_delete event |
                                                   |                     |
                                                   +---------------------+

MySQL Pub use row-based mysql binlog as events source, and transfer them into table_action events. mysql_pub generates an accurate event stream with one pk per event.

For example, the following sql:

INSERT INTO test (data) VALUES ('a');
INSERT INTO test (data) VALUES ('b'), ('c'), ('d');
UPDATE test SET data = 'aa' WHERE id = 1;
UPDATE test SET data = 'bb' WHERE id = 2;
UPDATE test SET data = 'cc' WHERE id != 1;
DELETE FROM test WHERE id != 1;
DELETE FROM test WHERE id = 1;

Generates the following events:

test_write 1
test_write 2
test_write 3
test_write 4
test_update 1
test_update 2
test_update 2
test_update 3
test_update 4
test_delete 2
test_delete 3
test_delete 4
test_delete 1

SQLAlchemy Pub

sqlalchemy_pub event flow:

+------------------+
|                  |
|    meepo.pub     |
|                  |
|  sqlalchemy_pub  |                                       +---------------------+
|                  |     +-----------------------+         |                     |
+---------+--------+     |                       |     +--->  table_write event  |
          |              |      before_flush     |     |   |                     |
    hook  |           +-->                       |     |   +---------------------+
          |           |  |  record model states  |     |
+---------v--------+  |  |                       |     |
|                  |  |  +-----------+-----------+     |   +---------------------+
|    sqlalchemy    |  |              |                 |   |                     |
|                  +--+              |              +------>  table_update event |
|  session events  |                 |              |  |   |                     |
|                  |     +-----------v-----------+  |  |   +---------------------+
+------------------+     |                       |  |  |
                         |     after_commit      |  |  |
                         |                       +--+  |   +---------------------+
                         |  record model states  |     |   |                     |
                         |                       |     +--->  table_delete event |
                         +-----------------------+         |                     |
                                                           +---------------------+

SQLAlchemy is a ORM layer above database, it uses session to maintain model instances states before the data flush to database, and flush them to database in commit.

So meepo will hook into the event system, record all the instances in session.new, session.dirty, session.deleted in before_flush event, then publish the table_action event after commit issued.

For example, the following code:

class Test(Base):
    __tablename__ = "test"
    id = Column(Integer, primary_key=True)
    data = Column(String)

t_1 = Test(id=1, data='a')
session.add(t_1)
session.commit()

t_2 = Test(id=2, data='b')
t_3 = Test(id=3, data='c')
session.add(t_2)
session.add(t_3)
session.add(t_4)
session.commit()

t_2.data = "x"
session.commit()

session.delete(t_3)
session.commit()

Generates the following events:

test_write 1
test_write 2
test_write 3
test_update 2
test_delete 3

Examples

Refer to meepo/examples/ for more examples.

Contribute

1. Fork the repo and make changes.

2. Write a test which shows a bug was fixed or the feature works as expected.

3. Make sure travis-ci test succeed.

4. Send pull request.