Skip to main content

A CLI client for exporting elasticsearch data to csv

Project description

Build Status Latest Version Python versions Package status Package coverage

This project is to just have a simple cli command to export data from ES, postgres, etc using the CPU’s, It’s intended to be used in Data workflow for extracting data out.

Note

This is still early in the development and a bit rough around the edges. Any bug reports, feature suggestions, etc are greatly appreciated. :)

Installation and usage

Installation Since this is a Python package available on PyPi you can install it like any other Python package.

# on modern systems with Python you can install with pip
$ pip install bq-sqoop
# on older systems you can install using easy_install
$ easy_install bq-sqoop

Usage The commands should be mostly self-documenting in how they are defined, which is made available through the help command.

$ bq-sqoop
usage: bq-sqoop -h

arguments:
    -h, --help            show this help message and exit
    -v, --version         Show version and exit.
    -c CONFIG_FILE,       --config_file CONFIG_FILE
                            Toml Config file for the bq-sqoop job.Can be a local
                            file path or a public http link or a GCS fileeg,
                            https://storage.googleapis.com/sample_config.toml or
                            gs://gcs_bucket/sample_config.toml or
                            /tmp/sample_config.toml
    -d, --debug           Debug mode on.

Configuration files

You can find an example repository at https://github.com/therako/bqsqoop-examples.git

Configuration objects

  1. Bigquery
  2. Extractor

Bigquery

[bigquery]
project_id="destination-google-project-id"
dataset_name="destination-dataset"
table_name="destination-table-name"
gcs_tmp_path="gs://gcs-tmp-bucket/bqsqoop/"

Extractor

Elasticsearch

[extractor.elasticsearch]
url="localhost:9200,localhost:9201"
index="source-es-index-name"
timeout="60s"
scroll_size=500
fields=["_all"]

SQL

[extractor.sql]
sql_bind="postgresql+psycopg2://username:password@127.0.0.1:5432/database"
query="select * from table_name"

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for bq-sqoop, version 0.0.13
Filename, size File type Python version Upload date Hashes
Filename, size bq_sqoop-0.0.13-py3-none-any.whl (20.9 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size bq-sqoop-0.0.13.tar.gz (16.1 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page