Import JSON raw data to ElasticSearch using Python in one line of commands
Project description
json-py-es
==========
Alexander Liu
* To import raw JSON data files to ElasticSearch using Python in one line of commands
### Installation
* ```pip install jsonpyes```
> In the past we import data in this way. Too many jobs by hands...
> ![Alt](static/snapshot106.jpg)
>
> Using ElasticSearch Bulk API to import this data, sometimes ES only recognizes data in its API way.
>
> But now let `json-py-es` glue them all.
jsonpyes
--------
![Alt](static/snapshot104.jpg)
#### Instructions:
There are 3 proccesses of importing raw JSON data to ElasticSearch
1. Only validating raw JSON data
2. Without validating ,just import data to ElasticSearch
3. After validating successfully, then import data to ElasticSearch
##### 1. Only validating
* ```python jsonpyes.py --data raw_data.json --check```
* If the json data file is valid:
![Alt](static/snapshot98.jpg)
* If the json data file is invalid:
![Alt](static/snapshot99.jpg)
##### 2. Only importing without validating
* Notice: If the raw JSON data file is invalid, ElasticSearch will not import it.
* ```python jsonpyes.py --data raw_data.json --bulk http://localhost:9200 --import --index myindex2 --type mytype2```
* Or enable multi-threads ```python jsonpyes.py --data raw_data.json --bulk http://localhost:9200 --import --index myindex2 --type mytype2 --thread 8```
![Alt](static/snapshot102.jpg)
** jsonpyes supports multi-threads when importing data to elasticsearch
![Alt](static/snapshot132.png)
> As you can see these two containers have same docs loaded, if we use **_--thread 8_** it could be slightly faster.
That really depends on your computer/server resources.
This was tested on a 4GB RAM / 2.4Ghz intel i5 Linux x64 laptop system.
![Alt](static/snapshot133.png)
* And it works.
![Alt](static/snapshot105.jpg)
##### 3. Both validating and importing
* ```python jsonpyes.py --data raw_data.json --bulk http://localhost:9200 --import --index myindex1 --type mytype1 --check```
![Alt](static/snapshot135.png)
* And it works.
![Alt](static/snapshot101.jpg)
##### Happy hacking!
==========
Alexander Liu
* To import raw JSON data files to ElasticSearch using Python in one line of commands
### Installation
* ```pip install jsonpyes```
> In the past we import data in this way. Too many jobs by hands...
> ![Alt](static/snapshot106.jpg)
>
> Using ElasticSearch Bulk API to import this data, sometimes ES only recognizes data in its API way.
>
> But now let `json-py-es` glue them all.
jsonpyes
--------
![Alt](static/snapshot104.jpg)
#### Instructions:
There are 3 proccesses of importing raw JSON data to ElasticSearch
1. Only validating raw JSON data
2. Without validating ,just import data to ElasticSearch
3. After validating successfully, then import data to ElasticSearch
##### 1. Only validating
* ```python jsonpyes.py --data raw_data.json --check```
* If the json data file is valid:
![Alt](static/snapshot98.jpg)
* If the json data file is invalid:
![Alt](static/snapshot99.jpg)
##### 2. Only importing without validating
* Notice: If the raw JSON data file is invalid, ElasticSearch will not import it.
* ```python jsonpyes.py --data raw_data.json --bulk http://localhost:9200 --import --index myindex2 --type mytype2```
* Or enable multi-threads ```python jsonpyes.py --data raw_data.json --bulk http://localhost:9200 --import --index myindex2 --type mytype2 --thread 8```
![Alt](static/snapshot102.jpg)
** jsonpyes supports multi-threads when importing data to elasticsearch
![Alt](static/snapshot132.png)
> As you can see these two containers have same docs loaded, if we use **_--thread 8_** it could be slightly faster.
That really depends on your computer/server resources.
This was tested on a 4GB RAM / 2.4Ghz intel i5 Linux x64 laptop system.
![Alt](static/snapshot133.png)
* And it works.
![Alt](static/snapshot105.jpg)
##### 3. Both validating and importing
* ```python jsonpyes.py --data raw_data.json --bulk http://localhost:9200 --import --index myindex1 --type mytype1 --check```
![Alt](static/snapshot135.png)
* And it works.
![Alt](static/snapshot101.jpg)
##### Happy hacking!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
jsonpyes-1.2.0.tar.gz
(5.7 kB
view details)
Built Distribution
File details
Details for the file jsonpyes-1.2.0.tar.gz
.
File metadata
- Download URL: jsonpyes-1.2.0.tar.gz
- Upload date:
- Size: 5.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | f481d4a1812da701978b1506d43340208e206e8c3e5255820b42b461b74cdfcc |
|
MD5 | 5b5baa008074fc270df55dccc6b0e650 |
|
BLAKE2b-256 | 0da8e26eb89a7e5c240e5bb4a776c8327f6a929f58e4f593047facb69446a93a |
File details
Details for the file jsonpyes-1.2.0.linux-x86_64.tar.gz
.
File metadata
- Download URL: jsonpyes-1.2.0.linux-x86_64.tar.gz
- Upload date:
- Size: 5.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0672a807855095c332e5079e9929fca01c596db4a9788a8a64333f1b45c8970c |
|
MD5 | 5c9cbe8e3cdb146d87757efc94825f92 |
|
BLAKE2b-256 | bdecf58577e275f00f40ac1c07c5656b7ee2b533b1403e3bc22b7f538fed298a |