Collections of ElasticSearch pyscripts for human
Project description
ElasticSearchCollections
Collections of ElasticSearch pyscripts for human
Example
from elasticSearch_collections.elasticSearch_collections import ElasticSearchCollections
from typing import Dict, List
# Connect to es hosts
es = ElasticSearchCollections(["https://127.0.0.1"],
http_auth=("username", "password"),
ca_certs=False,
verify_certs=False,
timeout=600)
def fn_data_update(_data: List[Dict]):
actions = []
for i in _data:
# do something
action = {
"_op_type": "update",
"_index": i["_index"],
"_type": "_doc",
"_id": i["_id"],
"doc": {
# update logic
}
}
actions.append(action)
if actions:
es.bulk(actions)
es_search_info = {
"index": '_index',
"scroll": '1m',
"size": 1000,
"body": """{
"query": {
"match_all": {}
}
}""",
"timeout": "10m"
}
_res_list = es.scroll_source_generator(es_search_info)
# run multiple threads with threado
from threado.simple_thread_runner import SimpleThreadsRunner
# Define the num_workers and batch_size
# refer to https://github.com/AlexNg9527/threado
sr = SimpleThreadsRunner(fn=fn_data_update)
sr.run_threads(num_workers=10, iter_data=_res_list, batch_size=10)
# clear scroll_id
es.delete_scroll_id()
Installing and Supported Versions
elasticSearch_collections is available on PyPI:
$ python -m pip install elasticSearch_collections
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for elasticSearch_collections-1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 686312af4fd62da5b51c6970917a21604d78e6e4647e7d1f27415758375e4afa |
|
MD5 | fd14bd10e3f04c85a9342fbb39369300 |
|
BLAKE2b-256 | 2bb57c6638577a8af2fd08705dbbe3c10ce58a1091f157d9e451d3c8ac0dd886 |
Close
Hashes for elasticSearch_collections-1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 522ecf5b1a63e72be27754371a4ce89bb37f464fdd9a0190feef1fc45715a34e |
|
MD5 | 1119cbf30e0e8a3d13950f5d01f1cc36 |
|
BLAKE2b-256 | 3410f834f5d709813b04323754e40473863d545340602f4ffd244861a3043d52 |