Collections of ElasticSearch pyscripts for human
Project description
ElasticSearchCollections
Collections of ElasticSearch pyscripts for human
Example
from elasticSearch_collections.elasticSearch_collections import ElasticSearchCollections
from typing import Dict, List
# Connect to es hosts
es = ElasticSearchCollections(["https://127.0.0.1"],
http_auth=("username", "password"),
ca_certs=False,
verify_certs=False,
timeout=600)
def fn_data_update(_data: List[Dict]):
actions = []
for i in _data:
# do something
action = {
"_op_type": "update",
"_index": i["_index"],
"_type": "_doc",
"_id": i["_id"],
"doc": {
# update logic
}
}
actions.append(action)
if actions:
es.bulk(actions)
es_search_info = {
"index": '_index',
"scroll": '1m',
"size": 1000,
"body": """{
"query": {
"match_all": {}
}
}""",
"timeout": "10m"
}
_res_list = es.scroll_source_generator(es_search_info)
# run multiple threads with threado
from threado.simple_thread_runner import SimpleThreadsRunner
# Define the num_workers and batch_size
# refer to https://github.com/AlexNg9527/threado
sr = SimpleThreadsRunner(num_workers=10,fn=fn_data_update)
sr.run_threads(iter_data=_res_list, batch_size=10)
# clear scroll_id
es.delete_scroll_id()
Installing and Supported Versions
elasticSearch_collections is available on PyPI:
$ python -m pip install elasticSearch_collections
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for elasticSearch_collections-1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | d61928ed1b2b139f93e2b3049b3cf3e657fac93569092b93e9a0fb0fd828667c |
|
MD5 | c2efad472797d098c3f464e2c67be9dd |
|
BLAKE2b-256 | 41efb05eccb02a0a3e78db99c77098a7573105b7e2e4b7ab68d39d122dc64825 |
Close
Hashes for elasticSearch_collections-1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 83a20b80b80bff57c433e4e968a4b717b8b9673ff23111d6b4ea368db719cbd0 |
|
MD5 | 22988965c3fa4c0345acb6c9eb045e27 |
|
BLAKE2b-256 | ed3144fe62718815bb39a3f277e1c18ce0c31e8b0edaf72fcfdba86fc7191eae |