Elasctic load CSV utility
Project description
Elastic CSV Loader
This command line utility loads csv file into an elasticsearch index, using a provided yaml config file.
load-csv
considerations:
- CSV files MUST include a header with field names
- Header field names will be used as elastic index fields
- A
@timestamp
anddate
fields will be added to all indexed docs- A
date
logic date could be forced through command parameter.
- A
- Depending on
elastic_index.data_format.parent_data_object
value, all original csv header fields will be arranged under adata
parent object.
Indexed data will use the same field names that
download-index
considerations:
- If csv file is an existing file the download process will append data including headers
- You have to rename or delete previous csv file if you want to start fresh.
Install
Dependencies
Python
3.8 or higherpip
package manager
pip install --upgrade elasticcsv
Run
Elastic Connection Config
Connection configuration is based in a YAML text file (connection.yaml
) that must be present in
command directory.
Sample connection.yaml
elastic_connection:
proxies:
http: "http://user:pass@proxy.url:8080"
https: "http://user:pass@proxy.url:8080"
user: myuser
password: mypassword
node: my.elastic.node
port: 9200
elastic_index:
data_format:
parent_data_object: true
Run command
❯ python elasticcsv/csv2es.py load-csv --help
Usage: csv2es.py load-csv [OPTIONS]
Loads csv to elastic index
Options:
--csv PATH CSV File [required]
--csv_offset INT CSV File offset
--sep TEXT CSV field sepator [required]
--index TEXT Elastic Index [required]
--csv-date-format TEXT date format for *_date columns as for ex:
'%Y-%m-%d'
--logic_date [%Y-%m-%d] Date reference for interfaces
-d, --delete-if-exists Flag for deleting index before running load
--help Show this message and exit.
Python date formats references: String Format Time
❯ python elasticcsv/csv2es.py download-index --help
Usage: csv2es.py download-index [OPTIONS]
Download index to csv file
Options:
--csv PATH CSV File [required]
--sep TEXT CSV field sepator [required]
--index TEXT Elastic Index [required]
-d, --delete-if-exists Flag for deleting csv file before download
--help Show this message and exit.
Example:
csv2es load-csv --csv ./pathtomyfile/file.csv --index myindex --sep ";"
csv2es download-index --csv ./pathtomyfile/file.csv --index myindex --sep ";" -d
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
elasticcsv-0.2.2.tar.gz
(8.8 kB
view hashes)
Built Distribution
Close
Hashes for elasticcsv-0.2.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7524924c128467d371333bc946029055afe08be44e1c18aaa7075fbfc7f5003b |
|
MD5 | e47850e300968f4c402a055c1328dcaa |
|
BLAKE2b-256 | 5528e53f715aea4466e6f4f9d45551912ddb30de0ea4215e2346214ccee81b33 |