Read, write and update large scale pandas DataFrame with ElasticSearch
Project description
es_pandas
Read, write and update large scale pandas DataFrame with ElasticSearch.
Requirements
This package should work on python3(>=3.4) and ElasticSearch should be version 6.x or 7.x(>=6.8).
Installation The package is hosted on PyPi and can be installed with pip:
pip install es_pandas
Usage
import time
import pandas as pd
from es_pandas import to_pandas, to_es
# Information of es cluseter
es_host = 'localhost:9200'
index = 'demo'
# Example data frame
df = pd.DataFrame({'Alpha': [chr(i) for i in range(97, 128)],
'Num': [x for x in range(31)],
'Date': pd.date_range(start='2019/01/01', end='2019/01/31')})
# Example of write data to es, auto create and put template to es if template does not exits
to_es(df, es_host, index)
time.sleep(10)
# Example of read data from es
df = to_pandas(es_host, index)
print(df.head())
df2 = pd.DataFrame({'Alpha': [chr(i) for i in range(97, 129)],
'Num': [x for x in range(32)],
'Date': pd.date_range(start='2019/01/01', end='2019/02/01')})
df2.loc[df2['Num']==10, ['Alpha']] = 'change'
# Example of update data in es
to_es_dev(df2, es_host, index, 'Num')
License
(c) 2019 Frank
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
es_pandas-0.0.3.tar.gz
(4.7 kB
view hashes)
Built Distribution
Close
Hashes for es_pandas-0.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0b74f12aa8465f742374e2736c1980cee6cdede3b61f5ac8f5420ccc26993abc |
|
MD5 | 98b8a4ed42788598f7a48c71ff98ac12 |
|
BLAKE2b-256 | 7d8b884b09580d463525ee7ba870b00b9a62e9c170d48622a1ab0e779f50a55f |