FS2Elastic

Python based CLI for Syncing File System based Data (CSV file etc.) to Elasticsearch.

Features

• Real time csv file to Elasticsearch dataset syncing
• Configurable with custom config file

Installation

pip install fs2elastic

Configuration

configuration file present at ~/.fs2elastic/fs2elastic.conf.

• please create configuration file if not present.
• replace values with correct values
• remove configs that does not require

default configuration:

[AppConfig]
app_home = "/home/john/.fs2elastic"
app_config_file_path = "/home/john/.fs2elastic/fs2elastic.conf"

[DatasetConfig]
dataset_source_dir = "/home/john/csv_data_set"
dataset_supported_file_extensions = [ "csv", "xlsx", "xls", "json"]
dataset_max_workers = 1
dataset_threads_per_worker = 10
dataset_chunk_size = 200

[ESConfig]
es_hosts = [ "https://localhost:9200",]
es_username = "elastic"
es_password = ""
es_timeout = 300
es_index_prefix = "fs2es-"
es_ssl_ca = ""
es_verify_certs = false

[LogConfig]
log_file_path = "/home/john/.fs2elastic/fs2elastic.log"
log_max_size = 10485760
log_backup_count = 5

Usage

with Default config file

fs2elastic

with Custom config file

fs2elastic -c <config file path>
eg: fs2elastic -c ~/Documents/fs2elastic_custom_config.conf

Help

fs2elastic -h

Version

fs2elastic -v

Use as a system service

NOTE: Required sudo permission

# install Package
sudo pip install fs2elastic

# Create fs2elastic home directory somewhere. for eg /tmp/fs2elastic/
mkdir /tmp/fs2elastic/

# create config file at /tmp/fs2elastic/fs2elastic.conf inside fs2elastic home directory created above
[AppConfig]
app_home = "/home/john/.fs2elastic"
app_config_file_path = "/home/john/.fs2elastic/fs2elastic.conf"

[DatasetConfig]
dataset_source_dir = "/home/john/csv_data_set"
dataset_supported_file_extensions = [ "csv", "xlsx", "xls", "json"]
dataset_max_workers = 1
dataset_threads_per_worker = 10
dataset_chunk_size = 200

[ESConfig]
es_hosts = [ "https://localhost:9200",]
es_username = "elastic"
es_password = ""
es_timeout = 300
es_index_prefix = "fs2es-"
es_ssl_ca = ""
es_verify_certs = false

[LogConfig]
log_file_path = "/home/john/.fs2elastic/fs2elastic.log"
log_max_size = 10485760
log_backup_count = 5

# replace with correct values

# create service file at /etc/systemd/system/fs2elastic.service  with following content

[Unit]
Description=FS2Elastic Service
After=network.target

[Service]
User=root
Group=root
ExecStart=/usr/bin/fs2elastic -c /tmp/fs2elastic/fs2elastic.conf
Restart=always

[Install]
WantedBy=multi-user.target

# Reloading Change to Systemd Unit Files
sudo systemctl daemon-reload

# Start service
sudo systemctl start fs2elastic

# Enable service to start on boot
sudo systemctl enable fs2elastic

# check status
sudo systemctl status fs2elastic

# check logs
sudo tail -f /var/log/fs2elastic.log

# To stop service
sudo systemctl stop fs2elastic

# To remove service
sudo systemctl disable fs2elastic
sudo rm /etc/systemd/system/fs2elastic.service

Uninstall

pip uninstall fs2elastic

Who do I talk to?

• Repo owner or admin
• Other community or team contact