Skip to main content

Realtime sync data from MySQL/PostgreSQL/MongoDB to meilisearch

Project description

meilisync

image image image image PyPI - Python Version

Introduction

Realtime sync data from MySQL/PostgreSQL/MongoDB to Meilisearch.

There is also a web admin dashboard for meilisync meilisync-admin.

Install

Just install from pypi:

pip install meilisync

Use docker (Recommended)

You can use docker to run meilisync:

version: "3"
services:
  meilisync:
    image: long2ice/meilisync
    volumes:
      - ./config.yml:/meilisync/config.yml
    restart: always

Prerequisites

  • MySQL: binlog_format = ROW, use binary log.
  • PostgreSQL: wal_level = logical and install wal2json extension, use logical replication.
  • MongoDB: enable replica set mode, use change stream.

Quick Start

If you run meilisync without any arguments, it will try to load the configuration from config.yml in the current directory.

 meilisync --help

 Usage: meilisync [OPTIONS] COMMAND [ARGS]...

╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --config              -c      TEXT  Config file path [default: config.yml]                                                                                                         │
│ --install-completion                Install completion for the current shell.                                                                                                      │
│ --show-completion                   Show completion for the current shell, to copy it or customize the installation.                                                               │
│ --help                              Show this message and exit.                                                                                                                    │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ check            Check whether the data in the database is consistent with the data in Meilisearch                                                                                 │
│ refresh          Refresh all data by swap index                                                                                                                                    │
│ start            Start meilisync                                                                                                                                                   │
│ version          Show meilisync version                                                                                                                                            │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Start sync

Start sync data from MySQL to Meilisearch:

 meilisync start
2023-03-07 08:37:25.656 | INFO     | meilisync.main:_:86 - Start increment sync data from "mysql" to Meilisearch...

Refresh sync

Refresh all data by swap index:

 meilisync refresh -t test

Before refresh, you need stop the sync process first to avoid data inconsistency.

Check sync

Check whether the data count in the database is consistent with the data in Meilisearch:

 meilisync check -t test

Configuration

Here is an example configuration file:

debug: true
plugins:
  - meilisync.plugin.Plugin
progress:
  type: file
source:
  type: mysql
  host: 192.168.123.205
  port: 3306
  user: root
  password: "123456"
  database: beauty
meilisearch:
  api_url: http://192.168.123.205:7700
  api_key:
  insert_size: 1000
  insert_interval: 10
sync:
  - table: collection
    index: beauty-collections
    plugins:
      - meilisync.plugin.Plugin
    full: true
    fields:
      id:
      title:
      description:
      category:
  - table: picture
    index: beauty-pictures
    full: true
    fields:
      id:
      description:
      category:
sentry:
  dsn: ""
  environment: "production"

debug (optional)

Enable debug mode, default is false, if you want to see more logs, you can set it to true.

plugins (optional)

The plugins are used to customize the data before or after insert to Meilisearch and the plugins is a list of python modules.

Which is a python class with pre_event and post_event methods, the pre_event method is called before insert to Meilisearch, the post_event method is called after insert to Meilisearch.

class Plugin:
    is_global = False

    async def pre_event(self, event: Event):
        logger.debug(f"pre_event: {event}, is_global: {self.is_global}")
        return event

    async def post_event(self, event: Event):
        logger.debug(f"post_event: {event}, is_global: {self.is_global}")
        return event

The is_global is used to indicate whether the plugin instance is global, if set to True, the plugin instance will be created only once, otherwise, the plugin instance will be created for each event.

progress

The progress is used to record the last sync position, such as binlog position for MySQL.

  • type: file or redis, if set to file, another option path is required.
  • path: the file path to store the progress, default is progress.json.
  • key: the redis key to store the progress, default is meilisync:progress.
  • dsn: the redis dsn, default is redis://localhost:6379/0.

source

Source database configuration, currently only support MySQL and PostgreSQL and MongoDB.

  • type: mysql or postgres or mongo.
  • server_id: the server id for MySQL binlog, default is 1.
  • database: the database name.
  • other keys: the database connection arguments, MySQL see asyncmy, PostgreSQL see psycopg2, MongoDB see motor.

meilisearch

Meilisearch configuration.

  • api_url: the Meilisearch API URL.
  • api_key: the Meilisearch API key.
  • insert_size: insert after collecting this many documents, optional.
  • insert_interval: insert after this many seconds have passed, optional.

If nether insert_size nor insert_interval is set, it will insert each document immediately.

If you prefer performance, just set and increase insert_size and insert_interval. The insert will be made as long as one of the conditions is met.

sync

The sync configuration, you can add multiple sync tasks.

  • table: the database table name or collection name.
  • index: the Meilisearch index name, if not set, it will use the table name.
  • full: whether to do a full sync, default is false.
  • fields: the fields to sync, if not set, it will sync all fields. The key is table field name, the value is the Meilisearch field name, if not set, it will use the table field name.
  • plugins: the table level plugins, optional.

sentry (optional)

Sentry configuration.

  • dsn: the sentry dsn.
  • environment: the sentry environment, default is production.

License

This project is licensed under the Apache-2.0 License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

meilisync-0.1.3.tar.gz (18.0 kB view details)

Uploaded Source

Built Distribution

meilisync-0.1.3-py3-none-any.whl (27.7 kB view details)

Uploaded Python 3

File details

Details for the file meilisync-0.1.3.tar.gz.

File metadata

  • Download URL: meilisync-0.1.3.tar.gz
  • Upload date:
  • Size: 18.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.9.18

File hashes

Hashes for meilisync-0.1.3.tar.gz
Algorithm Hash digest
SHA256 64894f5992e2d672c90f773aefee0bc18e857903815a76317571b7bffdb3318a
MD5 9438bbd65b8cb99e98c1f7d38776b05c
BLAKE2b-256 aae32287613a6aaeb099b7234280c27ccae777a5169313bbf2647b949a0102df

See more details on using hashes here.

File details

Details for the file meilisync-0.1.3-py3-none-any.whl.

File metadata

  • Download URL: meilisync-0.1.3-py3-none-any.whl
  • Upload date:
  • Size: 27.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.9.18

File hashes

Hashes for meilisync-0.1.3-py3-none-any.whl
Algorithm Hash digest
SHA256 7c92e765d435d51423fc15d52ada8feb1a116b06d3131f688b0e299a31eeae3a
MD5 4c0cb6ea80e7af27d06cc281baaef731
BLAKE2b-256 a09c277d8a4f21498a72a674776a119d6d9fdc58e05fc8d71feef5b7f9fb3b70

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page