es-peek 0.4.0

Peek into elasticsearch clusters

Peek is an interactive CLI tool for working with Elasticsearch clusters. It is like Kibana Console running in terminal with additional features for tinkers.

Installation

Please note the project requires Python 3.8+. It is recommended to install from PyPI with:

pip install es-peek

The tool is now available as the peek command. Peek will ask permission to access system keyring for storing credentials. This can be disabled temporarily by invoking the command as peek -e use_keyring=False or permanently by setting use_keyring = False in peekrc file

Besides HTTP calls to Elasticsearch clusters, type help to see a list of builtin functions. To enable auto-completions for APIs, run the _download_api_specs builtin function (note the leading underscore) to download API spec file from the Elasticsearch Specification project.

Alternatively, peek can also be installed from source with:

python setup.py install

Features

Peek supports most editing features offered by Kibana Console, e.g. auto-completion, syntax highlighting, auto-formatting, auto-indent, par-editing, triple-quotes, etc. It also offers following additional features:

• Lightweight CLI tool

• Multiplex a single terminal session to multiple Elasticsearch clusters or multiple connections to a single cluster

• Flexible usages of quotes, comma, comments for the JSON payload, case-insensitive http method names

• Multiple authentication schemes, including UserPass, API key, Token, SAML, OIDC, Kerberos, PKI

• Support run-as, x-opaque-id and arbitrary request headers

• Load JSON payload from external files

• Run file input in batch mode

• Readline editing features, e.g. Ctrl-_ for undo, Ctrl-r for reverse search, etc.

• History management

• Capture terminal input and output into file

• Connect to Elastic Cloud with Cloud ID

• Shell out for system commands

• Minimal scripting support

• Extensible via external scripts

Sample Usages

Assuming a locally running Elasticsearch cluster, start a Peek session with:

peek --hosts localhost:9200 --username elastic

The following sample is a quick guide on Peek usages:

// Download the API schema from the elasticsearch-specification project to enable auto-completion
// If the output says the schema file already exists, you can safely proceed to the next command.
_download_api_specs

// NOTE a blank line is needed to trigger API execution, or type "ESC + Enter" to execute regardlessly
// Exit the interactive session any time by pressing Ctrl-d or type exit
GET /_cluster/health  // comment is allowed almost anywhere

// Index a single document
POST /my-index/_doc
{'foo': "bar"}  // both single and double quotes are acceptable

// Bulk indexing
// Press <F3> to switch between pretty and compact formatting for the JSON payload
PUT _bulk
{"index":{"_index":"test","_id":"1"}}
{"value":"1","category":"click"}
{"index":{"_index":"test","_id":"2"}}
{"value":"2","category":"click"}

// Auto encoding for date math expression
// The following is encoded automatically into "PUT /%3Cmy-index-%7Bnow%2Fd%7D%3E" on the background
PUT /<my-index-{now/d}>

// Shell out to download the EQL threat hunting demo file
!curl -o normalized-T1117-AtomicRed-regsvr32.json https://raw.githubusercontent.com/elastic/elasticsearch/master/docs/src/test/resources/normalized-T1117-AtomicRed-regsvr32.json

// Bulk indexing the above downloaded file
POST my-index-000001/_bulk
@normalized-T1117-AtomicRed-regsvr32.json

// Execute an EQL query (triple quotes can be either ''' or """)
GET /my-index-000001/_eql/search?filter_path=-hits.events
{
  "query": """
    any where process.name == "regsvr32.exe"
  """,
  "size": 200
}

// Create an API key
PUT _security/api_key
{
  "name": "key-1",  // extra comma is OK, and this comment is ok as well
}

// Connect using the above generated API key
// The dot notation is used to index into JSON object and array
connect api_key=_.'id' + ":" + _.'api_key' hosts='localhost:9200'  // Quotes are necessary for strings

// Connect to Elastic Cloud with Cloud ID
connect cloud_id='YOUR_CLOUD_ID' username='elastic'

// Issue a call to the cloud cluster
get /  // HTTP method is case-insensitive
get / conn=0  // send the request to the first connection (zero-based index) with the conn option

// Check configuration location and values
config

// List available connections
connection
connection @info  // check details
connection rename='my-cloud-cluster'  // give a friendly name to the current connection
connection 0  // switch to the first connection
connection remove=0  // remove the first connection
connection 'my-cloud-cluster'  // switch to the cloud cluster connection

// Save the connections we have so far. Session is also auto-saved on exit.
session @save  // it can be loaded later with "session @load"

// Session auto-load on start up can be enabled by set "auto_load_session = True" in peekrc file.
// This helps preserving connections across restart.

// Builtin help
help  // list available functions
help session  // a bit more detailed info about the "session" builtin function

// Capture the terminal I/O
capture @start
capture  // show capture status

// Run-AS and other headers
GET _security/_authenticate runas='foo' xoid='my-x-opaque-id' headers={'some-other-header': 'blah'}

// Show only the first role from previous response
echo _."roles".0

// If the cluster has SAML integration configured, authenticate with saml
// Note this opens a web browser to perform the front-channel flow
saml_authenticate

// Load and run an external script
run 'my-setup.es'

// Stop the capture
capture @stop

// Minimal scripting for populating an index
let tags = range(0, 100)
for i in tags {
  PUT ("my-index/_doc/" + i)
  { 'tag': i, "value": i * i }
}

// Or with bulk index
for i in range(1, 100) {  // first prepare the payload file
  echo {"index":{"_index":"test","_id":"" + i}} file='payload.json'
  echo {"value":i,"category":"click"} file='payload.json'
}
// Now bulk indexing with the above generated file
PUT _bulk
@payload.json

The tool can also run in batch mode. Assuming above commands are saved in a file called script.es, it can be executed as:

# Positional argument
peek script.es

# Pipe
cat script.es | peek

# Or with heredoc
peek << EOF
GET /_cluster/health
// etc
EOF

External scripts can used to provide extra functions. They are simple Python scripts that define and expose callabes under a variable named EXPORTS. Please refer natives.py for examples.

Please also read Usages for more details.

Credits

Pgcli has been a great reference of learning how to use prompt-toolkit, which is a critical dependency of this project.