Skip to main content

Import JSON raw data to ElasticSearch using Python in one line of commands

Project description

Build Status GitHub release PyPI GitHub license

Alexander Liu

  • To import raw JSON data files to ElasticSearch using Python in one line of commands

Installation

  • pip install jsonpyes

    In the past we import data in this way. Too many jobs by hands… before image

    Using ElasticSearch Bulk API to import this data, sometimes ES only recognizes data in its API way.

    But now let json-py-es glue them all.

jsonpyes

  • user interface

Instructions:

There are 3 proccesses of importing raw JSON data to ElasticSearch
1. Only validating raw JSON data
2. Without validating ,just import data to ElasticSearch
3. After validating successfully, then import data to ElasticSearch
1. Only validating
  • jsonpyes --data raw_data.json --check

  • If the json data file is valid: json valid

  • If the json data file is invalid: json invalid

2. Only importing without validating
  • Notice: If the raw JSON data file is invalid, ElasticSearch will not import it.

  • jsonpyes --data raw_data.json --bulk http://localhost:9200 --import --index myindex2 --type mytype2

  • Or enable multi-threads jsonpyes --data raw_data.json --bulk http://localhost:9200 --import --index myindex2 --type mytype2 --thread 8 no threads

  • jsonpyes supports multi-threads when importing data to elasticsearch muti-threads enabled

    As you can see these two containers have same docs loaded, if we use *–thread 8* it could be slightly faster. That really depends on your computer/server resources. This was tested on a 4GB RAM / 2.4Ghz intel i5 Linux x64 laptop system.

  • benmarks

    benmarks

  • And it works. it works

3. Both validating and importing
  • jsonpyes --data raw_data.json --bulk http://localhost:9200 --import --index myindex1 --type mytype1 --check validating and importing

  • And it works. the results

Happy hacking!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

jsonpyes-1.2.22.tar.gz (19.2 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page