Realtime sync data from MySQL/PostgreSQL/MongoDB to meilisearch
Project description
meilisync
Introduction
Realtime sync data from MySQL/PostgreSQL/MongoDB to meilisearch.
Install
Just install from pypi:
pip install meilisync
Use docker (Recommended)
You can use docker to run meilisync
:
version: '3'
services:
meilisync:
image: long2ice/meilisync
volumes:
- ./config.yml:/meilisync/config.yml
restart: always
Quick Start
If you run meilisync
without any arguments, it will try to load the configuration from config.yml
in the current
directory.
```shell
> meilisync
2023-03-03 15:31:26.222 | INFO | meilisync.main:cli:44 - Full data sync for table "beauty.collection" done! 803 documents added.
2023-03-03 15:31:26.413 | INFO | meilisync.main:cli:44 - Full data sync for table "beauty.picture" done! 10612 documents added.
2023-03-03 15:31:26.413 | INFO | meilisync.main:cli:53 - Start increment sync data from "mysql" to MeiliSearch...
Configuration
Here is an example configuration file:
debug: true
progress:
type: file
source:
type: mysql
host: 192.168.123.205
port: 3306
user: root
password: '123456'
database: beauty
meilisearch:
api_url: http://192.168.123.205:7700
api_key:
sync:
- table: collection
index: beauty-collections
full: true
fields:
id:
title:
description:
category:
- table: picture
index: beauty-pictures
full: true
fields:
id:
description:
category:
sentry:
dsn: ''
environment: 'production'
debug
Enable debug mode, default is false
, if you want to see more logs, you can set it to true
.
progress
The progress is used to record the last sync position, such as binlog position for MySQL.
type
:file
orredis
, if set to file, another optionpath
is required.path
: the file path to store the progress, default isprogress.json
.key
: the redis key to store the progress, default ismeilisync:progress
.dsn
: the redis dsn, default isredis://localhost:6379/0
.
source
Source database configuration, currently only support MySQL and PostgreSQL and MongoDB.
type
:mysql
orpostgres
ormongo
.server_id
: the server id for MySQL binlog, default is1
.other keys
: the database connection arguments, MySQL see asyncmy, PostgreSQL see psycopg2, MongoDB see motor.
meilisearch
MeiliSearch configuration.
api_url
: the MeiliSearch API URL.api_key
: the MeiliSearch API key.
sync
The sync configuration, you can add multiple sync tasks.
table
: the database table name or collection name.index
: the MeiliSearch index name, if not set, it will use the table name.full
: whether to do a full sync, default isfalse
.fields
: the fields to sync, if not set, it will sync all fields. The key is table field name, the value is the MeiliSearch field name, if not set, it will use the table field name.
sentry
Sentry configuration.
dsn
: the sentry dsn.environment
: the sentry environment, default isproduction
.
License
This project is licensed under the Apache-2.0 License.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for meilisync-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e16636ee927e0378cce3712c7d6fe8982ff5ac1877f7de203e30b65ff3228bb8 |
|
MD5 | 8008ffa61cfb8c4ab9969fa83b41dcf3 |
|
BLAKE2b-256 | 5771612b33bd9addca2b2debd2836ae2a2e35b68aee56098995e2290fad8844b |