es_pandas
Read, write and update large scale pandas DataFrame with ElasticSearch.
Requirements
This package should work on Python3(>=3.4) and ElasticSearch should be version 5.x, 6.x or 7.x.
Installation The package is hosted on PyPi and can be installed with pip:
pip install es_pandas
Deprecation Notice
Supporting of ElasticSearch 5.x will by deprecated in future version.
Usage
import time
import pandas as pd
from es_pandas import es_pandas
# Information of es cluseter
es_host = 'localhost:9200'
index = 'demo'
# crete es_pandas instance
ep = es_pandas(es_host)
# Example data frame
df = pd.DataFrame({'Num': [x for x in range(100000)]})
df['Alpha'] = 'Hello'
df['Date'] = pd.datetime.now()
# init template if you want
doc_type = 'demo'
ep.init_es_tmpl(df, doc_type)
# Example of write data to es, use the template you create
ep.to_es(df, index, doc_type=doc_type, thread_count=2, chunk_size=10000)
# set use_index=True if you want to use DataFrame index as records' _id
ep.to_es(df, index, doc_type=doc_type, use_index=True, thread_count=2, chunk_size=10000)
# delete records from es
ep.to_es(df.iloc[5000:], index, doc_type=doc_type, _op_type='delete', thread_count=2, chunk_size=10000)
# Update doc by doc _id
df.iloc[:1000, 1] = 'Bye'
df.iloc[:1000, 2] = pd.datetime.now()
ep.to_es(df.iloc[:1000, 1:], index, doc_type=doc_type, _op_type='update')
# Example of read data from es
df = ep.to_pandas(index)
print(df.head())
# return certain fields in es
heads = ['Num', 'Date']
df = ep.to_pandas(index, heads=heads)
print(df.head())
# set certain columns dtype
dtype = {'Num': 'float', 'Alpha': object}
df = ep.to_pandas(index, dtype=dtype)
print(df.dtypes)
# infer dtype from es template
df = ep.to_pandas(index, infer_dtype=True)
print(df.dtypes)
# use query_sql parameter if you want to do query in sql
# Example of write data to es with pandas.io.json
ep.to_es(df, index, doc_type=doc_type, use_pandas_json=True, thread_count=2, chunk_size=10000)
print('write es doc with pandas.io.json finished')
1 maintainer
Release files for es-pandas 0.0.23
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| es_pandas-0.0.23.tar.gz | 5.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| es_pandas-0.0.23-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 12.3 kB
Release files / es_pandas-0.0.23.tar.gz
| Download URL | es_pandas-0.0.23.tar.gz |
|---|---|
| Size | 5.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b868060f2185260594e2e5e400ce58dfb5c57af00dd8603301b7070a3d45403d
|
|
BLAKE2b-256 checksum How to use checksums |
f4083e9ea2907a6b069bd2136b226dd2589e7ffe927473d9ac845f09a3c218a8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.9.7
|
Release files / es_pandas-0.0.23-py3-none-any.whl
| Download URL | es_pandas-0.0.23-py3-none-any.whl |
|---|---|
| Size | 6.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
933b2bad7874c011e22ee41c7535fcd7a5614b413dec74b892de02c841928795
|
|
BLAKE2b-256 checksum How to use checksums |
5dc9041a7d721f6e76a0097d7dcb08277cd7007059c71f5f428a7312001951cc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.9.7
|
