fs2elastic 0.1.2

Python based CLI for Syncing File System based Data (CSV file etc.) to Elasticsearch.

FS2Elastic

Python based CLI for Syncing File System based Data (CSV file etc.) to Elasticsearch.

Features

• Real time csv file to Elasticsearch dataset syncing
• Configurable with custom config file

Installation

pip install fs2elastic

Configuration

configuration file present at ~/.fs2elastic/fs2elastic.conf.

• please create configuration file if not present.
• replace values with correct values
• remove configs that does not require

default configuration:

[AppConfig]
app_home = "/home/john/.fs2elastic"
app_config_file_path = "/home/john/.fs2elastic/fs2elastic.conf"

[DatasetConfig]
dataset_source_dir = "/home/john/csv_data_set"
dataset_supported_file_extensions = [ "csv", "xlsx", "xls", "json"]
dataset_max_workers = 1
dataset_threads_per_worker = 10
dataset_chunk_size = 200

[ESConfig]
es_hosts = [ "https://localhost:9200",]
es_username = "elastic"
es_password = ""
es_timeout = 300
es_index_prefix = "fs2es-"
es_ssl_ca = ""
es_verify_certs = false

[LogConfig]
log_file_path = "/home/john/.fs2elastic/fs2elastic.log"
log_max_size = 10485760
log_backup_count = 5

Usage

with Default config file

fs2elastic

with Custom config file

fs2elastic -c <config file path>
eg: fs2elastic -c ~/Documents/fs2elastic_custom_config.conf

Help

fs2elastic -h

Version

fs2elastic -v

Use as a system service

NOTE: Required sudo permission

# install Package
sudo pip install fs2elastic

# Create fs2elastic home directory somewhere. for eg /tmp/fs2elastic/
mkdir /tmp/fs2elastic/

# create config file at /tmp/fs2elastic/fs2elastic.conf inside fs2elastic home directory created above
[AppConfig]
app_home = "/home/john/.fs2elastic"
app_config_file_path = "/home/john/.fs2elastic/fs2elastic.conf"

[DatasetConfig]
dataset_source_dir = "/home/john/csv_data_set"
dataset_supported_file_extensions = [ "csv", "xlsx", "xls", "json"]
dataset_max_workers = 1
dataset_threads_per_worker = 10
dataset_chunk_size = 200

[ESConfig]
es_hosts = [ "https://localhost:9200",]
es_username = "elastic"
es_password = ""
es_timeout = 300
es_index_prefix = "fs2es-"
es_ssl_ca = ""
es_verify_certs = false

[LogConfig]
log_file_path = "/home/john/.fs2elastic/fs2elastic.log"
log_max_size = 10485760
log_backup_count = 5

# replace with correct values

# create service file at /etc/systemd/system/fs2elastic.service  with following content

[Unit]
Description=FS2Elastic Service
After=network.target

[Service]
User=root
Group=root
ExecStart=/usr/bin/fs2elastic -c /tmp/fs2elastic/fs2elastic.conf
Restart=always

[Install]
WantedBy=multi-user.target

# Reloading Change to Systemd Unit Files
sudo systemctl daemon-reload

# Start service
sudo systemctl start fs2elastic

# Enable service to start on boot
sudo systemctl enable fs2elastic

# check status
sudo systemctl status fs2elastic

# check logs
sudo tail -f /var/log/fs2elastic.log

# To stop service
sudo systemctl stop fs2elastic

# To remove service
sudo systemctl disable fs2elastic
sudo rm /etc/systemd/system/fs2elastic.service

Uninstall

pip uninstall fs2elastic

Who do I talk to?

• Repo owner or admin
• Other community or team contact