MySync
同步MySQL数据到ElasticSearch
支持全量同步
必须有自增id
安装
$ pip install mysync
$ mysync
# run config test
$ mysync -t config.yml
# run sync
$ mysync -c config.yml
默认的配置文件default_config.yml
# 输入配置
input:
# 数据库配置
mysql:
db_url: ~
# 或者
host: '127.0.0.1'
port: 3306
database: data
username: root
password: ''
# 主键列名
primary_key: id
# 主键类型
primary_type: int
# 同步的表名,必填
# table: ''
# 需要同步字段
fields: '*'
# 分页大小
size: 10
# 生产者,会按照路径导入
producer: 'mysync.producer.mysql_producer@producer'
# 启用同步点
sync_point: true
# 同步点文件
sync_file: ~
# 数据处理管道
pipeline:
handlers: ~
# 输出配置
output:
# ES配置
elasticsearch:
hosts:
- 'http://127.0.0.1:9200'
# 索引名,必填
# index: ''
# 操作:index, update
action: 'index'
# 文档type
document_type: 'doc'
# 文档id
document_id: 'id'
# 是否在控制台输出json
stdout: false
# 数据消费者
consumer: 'mysync.consumer.es_consumer@consumer'
默认的处理器
# 生产者,会按照路径导入
producer: 'mysync.producer.mysql_producer@producer'
# 数据消费者
consumer: 'mysync.consumer.es_consumer@consumer'
# 可选的数据处理器
handlers:
- mysync.pipeline.field_mapping_pipeline@pipeline
接口说明
生产者
def producer(config):
处理器
def pipeline(config, rows)
消费者
def consumer(config, rows):
TODO
1. 配置文件继承
extends2. 配置文件运行前检测 -t
Metadata
Release files for MySync 0.0.13
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| MySync-0.0.13.tar.gz | 7.4 kB | Without provenance | Details for MySync-0.0.13.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| MySync-0.0.13-py3-none-any.whl | Python 3 | none | any | Details for MySync-0.0.13-py3-none-any.whl |
Total release size: 19.0 kB
Release files / MySync-0.0.13.tar.gz
| Download URL | MySync-0.0.13.tar.gz |
|---|---|
| Size | 7.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a6c9303b297b731754c0bc2a907f7b3a54c73833b49152a7e7b5372621eb688f
|
|
BLAKE2b-256 checksum How to use checksums |
9dbe6646432bd5b07bbf7b091979f40d59f4ef75058ba01c7d52c6a46e942969
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.7.0
|
Release files / MySync-0.0.13-py3-none-any.whl
| Download URL | MySync-0.0.13-py3-none-any.whl |
|---|---|
| Size | 11.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
85b26d3c26cdcc35b7da461fc94aaa698085bf17962b7084713c2875a55cc732
|
|
BLAKE2b-256 checksum How to use checksums |
06556e24e65ffd1715a5c409f54fd28facffa14233dcd74c085e6ba95053070e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.7.0
|
Security for MySync 0.0.13
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.