Python package for simple migration elasticsearch indexes between servers.
Project description
Elasticsearch Reindex
Description
This tool can be used to transfer Elasticsearch indexes between different servers.
Installing
pip install elasticsearch-reindex
How to use
Whitelist source_host
Elasticsearch in dest_host
Elasticsearch.
You should edit Elasticsearch YML config:
Config path:
$ /etc/elasticsearch/elasticsearch.yml
Add setting to the end of the file:
$ reindex.remote.whitelist: <es-source-host>:<es-source-port>
Use CLI for run migration data:
$ elasticsearch_reindex \
--source_host http(s)://es-source-host:es-source-port \
--dest_host http(s)://es-dest-host:es-dest-port \
--check_interval 5 \
--concurrent_tasks 3 \
-i test_index_1 -i test_index_2
CLI Params description (example):
Required fields:
-
source_host
- Elasticsearch endpoint where data will be extracted. -
dest_host
- Elasticsearch endpoint where data will be transfered.
Optional fields:
-
check_interval
- Time period (in second) to check task success status.Default value
-10
(seconds) -
concurrent_tasks
- How many parallel task Elasticsearch will process.Default value
-1
(sync mode) -
indexes
- List of user ES indexes to migrate instead of all source indexes.
Run library from Python script:
from elasticsearch_reindex import Manager
def main() -> None:
"""
Example reindex function.
"""
config = {
"source_host": "http://localhost:9201",
"dest_host": "http://localhost:9202",
"check_interval": 20,
"concurrent_tasks": 5,
}
manager = Manager.from_dict(data=config)
manager.start_reindex()
if __name__ == "__main__":
main()
With custom user indexes:
from elasticsearch_reindex import Manager
def main() -> None:
"""
Example reindex function.
"""
config = {
"source_host": "http://localhost:9201",
"dest_host": "http://localhost:9202",
"check_interval": 20,
"concurrent_tasks": 5,
"indexes": ["es-index-1", "es-index-2", "es-index-n"]
}
manager = Manager.from_dict(data=config)
manager.start_reindex()
if __name__ == "__main__":
main()
Local install
Setup and activate a python3 virtualenv via your preferred method. e.g. and install production requirements:
$ make ve
To remove virtualenv:
$ make clean
To install github hooks:
$ make install_hooks
Create .env file and fill the data:
$ cp .env.example .env
Export env variables:
$ export $(xargs < .env)
Env variables description:
Variable for enable testing:
ENV
- variable for enable testing mode. For activate test mode set to value -test
.
Elasticsearch docker settings:
-
ES_SOURCE_PORT
- Source Elasticsearch port -
ES_DEST_PORT
- Destination Elasticsearch port -
ES_VERSION
- Elasticsearch version -
LOCAL_IP
- Address of you local host machine in LAN like192.168.4.106
.
You can find it in:
-
Mac OS:
$ ifconfig
-
Linux (find it in response):
$ ip r
Tests
Firstly up docker-compose services with 2 nodes of ElasticSearch:
$ docker-compose up -d
Ensure that Elasticsearch nodes started correctly.
Env variables set from .env
file.
For Source Elasticsearch:
$ curl -X GET $LOCAL_IP:$ES_SOURCE_PORT
For destination Elasticsearch:
$ curl -X GET $LOCAL_IP:$ES_DEST_PORT
Export to PYTHONPATH
env variable:
$ export PYTHONPATH="."
For run tests with pytest
use:
$ pytest ./tests
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for elasticsearch-reindex-1.2.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | f926df56ae90e155bb34f08bbdfc639c9fff3acec664c585d42a6d75a0d8587d |
|
MD5 | db4d10b64601a7ba3edeea81c5aea38a |
|
BLAKE2b-256 | 03061d39750cb14e9519c06ab2b3a58fbb4ca7420b890deba3fa76638e1b740b |