elasticsearch-fabric 0.1.5

This package provides a unified command line interface to Elasticsearch in Fabric.

This package provides a unified command line interface to Elasticsearch in Fabric.

Installation

The current release, published on PyPI, can be installed using the following command:

$ pip install elasticsearch-fabric

Configuration

Tasks

If you plan to use the built-in tasks, include the module in your fabfile module (e.g. fabfile.py). Most likely you might want to assign an alias for the task namespace:

from esfabric import tasks as es

Environment

• elasticsearch_clients: Customize elasticsearch client configurations.

• elasticsearch_alias: Default Elasticsearch client alias in elasticsearch_clients. default “default”

• elasticsearch_dest_alias: Reindex dest Elasticsearch client alias in elasticsearch_clients. default elasticsearch_alias

Examples

# cat fabfile.py
from fabric.api import env
from elasticsearch import Elasticsearch
from esfabric import tasks as es


env.elasticsearch_clients = {
    "default": Elasticsearch(**{
        "host": "localhost",
        "port": 9200,
        "send_get_body_as": "POST"
    }),
    "example": Elasticsearch(**{
        "host": "search.example.org",
        "port": 443,
        "send_get_body_as": "POST",
        "use_ssl": True,
        "verify_certs": True
    })
}

Elasticsearch with Shield

You can configure the client to use basic authentication:

# cat fabfile.py
from fabric.api import env
from elasticsearch import Elasticsearch
from esfabric import tasks as es


env.elasticsearch_clients = {
    "default": Elasticsearch(**{
      "host": "localhost",
      "port": 9200,
      "send_get_body_as": "POST",
      "http_auth": ('user', 'secret')
    })
}

Running on AWS with IAM

# cat fabfile.py
from fabric.api import env
from elasticsearch import Elasticsearch
from elasticsearch import RequestsHttpConnection
from requests_aws4auth import AWS4Auth
from esfabric import tasks as es

awsauth = AWS4Auth(YOUR_ACCESS_KEY, YOUR_SECRET_KEY, REGION, 'es')

env.elasticsearch_clients = {
    "default": Elasticsearch(**{
        "host": "YOURHOST.us-east-1.es.amazonaws.com",
        "port": 443,
        "send_get_body_as": "POST",
        "http_auth": awsauth,
        "use_ssl": True,
        "verify_certs": True,
        "connection_class": RequestsHttpConnection
    })
}

Checking the setup

For checking if everything is set up properly, you can run the included task info. For example, running

$ fab es.info

you can show a similar result:

{
  "cluster_name": "elasticsearch",
  "tagline": "You Know, for Search",
  "version": {
    "lucene_version": "5.5.0",
    "build_hash": "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
    "number": "2.3.3",
    "build_timestamp": "2016-05-17T15:40:04Z",
    "build_snapshot": false
  },
  "name": "Ares"
}


Done.

Example use

You can do this for example with the following command:

# -------------------------------------------------------------
# by Default connect to localhost:9200
$ fab es.info
{
  "cluster_name": "elasticsearch",
  "tagline": "You Know, for Search",
  "version": {
    "lucene_version": "5.5.0",
    "build_hash": "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
    "number": "2.3.3",
    "build_timestamp": "2016-05-17T15:40:04Z",
    "build_snapshot": false
  },
  "name": "Ares"
}


Done.

# -------------------------------------------------------------
# index a document
#
# $ cat doc.json
# {
#   "title": "Hello Elasticsearch",
#   "description": "elasticsearch fabric test"
# }
$ cat doc.json | fab es.index:index=blog,doc_type=posts,id=1
{
  "_type": "posts",
  "created": true,
  "_shards": {
    "successful": 1,
    "failed": 0,
    "total": 2
  },
  "_version": 1,
  "_index": "blog",
  "_id": "1"
}


Done.

# -------------------------------------------------------------
# get the document.
$ fab es.get:index=blog,doc_type=posts,id=1
{
  "_type": "posts",
  "_source": {
    "description": "elasticsearch fabric test",
    "title": "Hello Elasticsearch"
  },
  "_index": "blog",
  "_version": 1,
  "found": true,
  "_id": "1"
}


Done.

# -------------------------------------------------------------
# simple query search.
$ fab es.search:q="title:hello"
{
  "hits": {
    "hits": [
      {
        "_score": 0.19178301,
        "_type": "posts",
        "_id": "1",
        "_source": {
          "description": "elasticsearch fabric test",
          "title": "Hello Elasticsearch"
        },
        "_index": "blog"
      }
    ],
    "total": 1,
    "max_score": 0.19178301
  },
  "_shards": {
    "successful": 26,
    "failed": 0,
    "total": 26
  },
  "took": 4,
  "timed_out": false
}

Done.

# -------------------------------------------------------------
# request body search.
#
# $ cat query.json
# {
#   "query": {
#     "match": {
#       "title": "hello"
#     }
#   }
# }
$ cat query.json | fab es.search
{
  "hits": {
    "hits": [
      {
        "_score": 0.19178301,
        "_type": "posts",
        "_id": "1",
        "_source": {
          "description": "elasticsearch fabric test",
          "title": "Hello Elasticsearch"
        },
        "_index": "blog"
      }
    ],
    "total": 1,
    "max_score": 0.19178301
  },
  "_shards": {
    "successful": 26,
    "failed": 0,
    "total": 26
  },
  "took": 8,
  "timed_out": false
}


Done.

# -------------------------------------------------------------
# change number of replicas
#
# cat indices
$ fab es.cat.indices
health status index                  pri rep docs.count docs.deleted store.size pri.store.size
yellow open   blog                     5   1          1            0      3.9kb          3.9kb
# change number of replicas
$ fab es.helpers.change_replicas:index=blog,number_of_replicas=0
{
  "acknowledged": true
}
# cat indices
$ fab es.cat.indices:v=1
health status index                  pri rep docs.count docs.deleted store.size pri.store.size
green  open   blog                     5   0          1            0      3.9kb          3.9kb


# -------------------------------------------------------------
# reindex blog to blog2
$ fab es.helpers.reindex:source_index=blog,dest_index=blog2
{
  "dest": {
    "index": "blog2",
    "host": "http://localhost:9200"
  },
  "source": {
    "index": "blog",
    "host": "http://localhost:9200"
  },
  "errors": 0,
  "success": 1
}


Done.
# cat indices
$ fab es.cat.indices:v=1
health status index                  pri rep docs.count docs.deleted store.size pri.store.size
yellow open   blog2                    5   1          1            0      3.7kb          3.7kb
green  open   blog                     5   0          1            0      3.9kb          3.9kb

Client selection

# fabfile.py
from esfabric import tasks as es
from esfabric.tasks import client_selection

env.elasticsearch_clients = {
    "client1": Elasticsearch(**{
      ...
    }),
    "client2": Elasticsearch(**{
      ...
    })
}

$ fab c:client2 es.info

c is client_selection alias

Logging

you can enable the elasticsearch.trace logger and have it log a shell transcript of your session using curl:

# fabfile.py
import logging
tracer = logging.getLogger('elasticsearch.trace')
tracer.setLevel(logging.DEBUG)
tracer.addHandler(logging.FileHandler('/tmp/elasticsearch-py.sh'))

Custom Task

from esfabric import tasks as es
from fabric.api import execute, task


@task
def change_replicas(number_of_replicas=1):
    execute(es.cat.indices, v=1)
    execute(es.helpers.change_replicas, number_of_replicas=number_of_replicas)
    execute(es.cat.indices, v=1)

run task:

$ fab change_replicas:10

Available commands

The following command will show a list of avaliable commands.

$ fab -l

• Available commands

• es.bulk

• es.c

• es.clear_scroll

• es.client_selection

• es.count

• es.count_percolate

• es.create

• es.delete

• es.delete_by_query

• es.delete_script

• es.delete_template

• es.exists

• es.explain

• es.field_stats

• es.get

• es.get_script

• es.get_source

• es.get_template

• es.index

• es.info

• es.mget

• es.mpercolate

• es.msearch

• es.msearch_template

• es.mtermvectors

• es.percolate

• es.ping

• es.put_script

• es.put_template

• es.reindex

• es.reindex_rethrottle

• es.render_search_template

• es.scroll

• es.search

• es.search_shards

• es.search_template

• es.suggest

• es.termvectors

• es.update

• es.update_by_query

• es.cat.aliases

• es.cat.allocation

• es.cat.count

• es.cat.fielddata

• es.cat.health

• es.cat.indices

• es.cat.master

• es.cat.nodeattrs

• es.cat.nodes

• es.cat.pending_tasks

• es.cat.plugins

• es.cat.recovery

• es.cat.repositories

• es.cat.segments

• es.cat.shards

• es.cat.snapshots

• es.cat.thread_pool

• es.cluster.allocation_explain

• es.cluster.get_settings

• es.cluster.health

• es.cluster.pending_tasks

• es.cluster.put_settings

• es.cluster.reroute

• es.cluster.state

• es.cluster.stats

• es.helpers.bulk

• es.helpers.change_replicas

• es.helpers.reindex

• es.helpers.scan

• es.indices.analyze

• es.indices.clear_cache

• es.indices.close

• es.indices.create

• es.indices.delete

• es.indices.delete_alias

• es.indices.delete_template

• es.indices.exists

• es.indices.exists_alias

• es.indices.exists_template

• es.indices.exists_type

• es.indices.flush

• es.indices.flush_synced

• es.indices.forcemerge

• es.indices.get

• es.indices.get_alias

• es.indices.get_field_mapping

• es.indices.get_mapping

• es.indices.get_settings

• es.indices.get_template

• es.indices.get_upgrade

• es.indices.open

• es.indices.put_alias

• es.indices.put_mapping

• es.indices.put_settings

• es.indices.put_template

• es.indices.recovery

• es.indices.refresh

• es.indices.rollover

• es.indices.segments

• es.indices.shard_stores

• es.indices.shrink

• es.indices.stats

• es.indices.update_aliases

• es.indices.upgrade

• es.indices.validate_query

• es.nodes.hot_threads

• es.nodes.info

• es.nodes.stats

• es.snapshot.create

• es.snapshot.create_repository

• es.snapshot.delete

• es.snapshot.delete_repository

• es.snapshot.get

• es.snapshot.get_repository

• es.snapshot.restore

• es.snapshot.status

• es.snapshot.verify_repository