es2csv 5.5.2

A CLI tool for exporting data from Elasticsearch into a CSV file.

A CLI tool for exporting data from Elasticsearch into a CSV file

Command line utility, written in Python, for querying Elasticsearch in Lucene query syntax or Query DSL syntax and exporting result as documents into a CSV file. This tool can query bulk docs in multiple indices and get only selected fields, this reduces query execution time.

Quick Look Demo

Requirements

This tool should be used with Elasticsearch 5.x version, for older version please check 2.x release.

You also need Python 2.7.x and pip.

Installation

>From source:

$ pip install git+https://github.com/taraslayshchuk/es2csv.git

>From pip:

$ pip install es2csv

Usage

$ es2csv [-h] -q QUERY [-u URL] [-a AUTH] [-i INDEX [INDEX ...]]
         [-D DOC_TYPE [DOC_TYPE ...]] [-t TAGS [TAGS ...]] -o FILE
         [-f FIELDS [FIELDS ...]] [-S FIELDS [FIELDS ...]] [-d DELIMITER]
         [-m INTEGER] [-s INTEGER] [-k] [-r] [-e] [--verify-certs]
         [--ca-certs CA_CERTS] [--client-cert CLIENT_CERT]
         [--client-key CLIENT_KEY] [-v] [--debug]

Arguments:
 -q, --query QUERY                        Query string in Lucene syntax.               [required]
 -o, --output-file FILE                   CSV file location.                           [required]
 -u, --url URL                            Elasticsearch host URL. Default is http://localhost:9200.
 -a, --auth                               Elasticsearch basic authentication in the form of username:password.
 -i, --index-prefixes INDEX [INDEX ...]   Index name prefix(es). Default is ['logstash-*'].
 -D, --doc-types DOC_TYPE [DOC_TYPE ...]  Document type(s).
 -t, --tags TAGS [TAGS ...]               Query tags.
 -f, --fields FIELDS [FIELDS ...]         List of selected fields in output. Default is ['_all'].
 -S, --sort FIELDS [FIELDS ...]           List of <field>:<direction> pairs to sort on. Default is [].
 -d, --delimiter DELIMITER                Delimiter to use in CSV file. Default is ",".
 -m, --max INTEGER                        Maximum number of results to return. Default is 0.
 -s, --scroll-size INTEGER                Scroll size for each batch of results. Default is 100.
 -k, --kibana-nested                      Format nested fields in Kibana style.
 -r, --raw-query                          Switch query format in the Query DSL.
 -e, --meta-fields                        Add meta-fields in output.
 --verify-certs                           Verify SSL certificates. Default is False.
 --ca-certs CA_CERTS                      Location of CA bundle.
 --client-cert CLIENT_CERT                Location of Client Auth cert.
 --client-key CLIENT_KEY                  Location of Client Cert Key.
 -v, --version                            Show version and exit.
 --debug                                  Debug mode on.
 -h, --help                               show this help message and exit

[ Usage Examples | Release Changelog ]

Release Changelog

5.5.2 (2018-03-21)

• Fixed encoding in field name to UTF-8. (Issue #35)

• Added –sort(-S) argument for sorting data by selected field. (Issue #41)

• Added requirement for version of python 2.7.*. (Issue #8, #12, #20, #29, #33 and #38)

• Update documentation with examples.

• Updating version elasticsearch-py to 5.5.*.

5.2.1 (2017-04-02)

• Added –verify-certs, –ca-certs, –client-cert, –client-key arguments for SSL configuration. (Issue #11 and #24, Pull #22)

• Added –scroll_size(-s) argument to specify the scroll size of requests. (Pull #27)

5.2.0 (2017-02-16)

• Updating version elasticsearch-py to 5.2.* and added support of Elasticsearch 5. (Issue #19)

2.4.3 (2017-02-15)

• Update doc according to wildcard support in fields naming.

• Added support of old version pip. (Issue #16)

2.4.2 (2017-02-14)

• Added wildcard support in fields naming.

• Removed column sorting. (Issue #21)

2.4.1 (2016-11-10)

• Added –auth(-a) argument for Elasticsearch basic authentication. (Pull #17)

• Added –doc_types(-D) argument for specifying document type. (Pull #13)

2.4.0 (2016-10-26)

• Added JSON validation for raw query. (Issue #7)

• Added checks to exclude hangs during connection issues. (Issue #9)

• Updating version elasticsearch-py to 2.4.0 and freeze this dependence according to mask 2.4.*. (Issue #14)

• Updating version progressbar2 to fix issue with visibility.

1.0.3 (2016-06-12)

• Added option to read query string from file –query(-q) @’~/filename.json’. (Issue #5)

• Added –meta_fields(-e) argument for selecting meta-fields: _id, _index, _score, _type. (Issue #6)

• Updating version elasticsearch-py to 2.3.0.

1.0.2 (2016-04-12)

• Added –raw_query(-r) argument for using the native Query DSL format.

1.0.1 (2016-01-22)

• Fixed support elasticsearch-1.4.0.

• Added –version argument.

• Added history changelog.

1.0.0.dev1 (2016-01-04)

• Fixed encoding in CSV to UTF-8. (Issue #3, Pull #1)

• Added better progressbar unit names. (Pull #2)

• Added pip installation instruction.

1.0.0.dev0 (2015-12-25)

• Initial registration.

• Added first dev-release on github.

• Added first release on PyPI.