Skip to main content

Helping you determine field usage by index

Project description

es-fieldusage

PyPI - Version PyPI - Python Version


Table of Contents

Installation

pip install es-fieldusage

Description

Determine which fields are being used, how much, for a given index.

Top-level help output

$ es-fieldusage --help

Usage: es-fieldusage [OPTIONS] COMMAND [ARGS]...

  Elasticsearch Index Field Usage Reporting Tool

  Sum all field query/request access for one or more indices using the Elastic Field Usage API (https://ela.st/usagestats)

  Generate a report at the command-line with the stdout command for all indices in INDEX_PATTERN:

  $ es-fieldusage stdout INDEX_PATTERN

  To avoid errors, be sure to encapsulate wildcards in single-quotes:

  $ es-fieldusage stdout 'index-*'

Options:
  --config PATH                   Path to configuration file.
  --hosts TEXT                    Elasticsearch URL to connect to.
  --cloud_id TEXT                 Elastic Cloud instance id
  --api_token TEXT                The base64 encoded API Key token
  --id TEXT                       API Key "id" value
  --api_key TEXT                  API Key "api_key" value
  --username TEXT                 Elasticsearch username
  --password TEXT                 Elasticsearch password
  --request_timeout FLOAT         Request timeout in seconds
  --verify_certs / --no-verify_certs
                                  Verify SSL/TLS certificate(s)
  --ca_certs TEXT                 Path to CA certificate file or directory
  --client_cert TEXT              Path to client certificate file
  --client_key TEXT               Path to client key file
  --loglevel [DEBUG|INFO|WARNING|ERROR|CRITICAL]
                                  Log level
  --logfile TEXT                  Log file
  --logformat [default|ecs]       Log output format
  --blacklist TEXT                Named entities will not be logged
  -v, --version                   Show the version and exit.
  -h, --help                      Show this message and exit.

Commands:
  show-all-options  Show all configuration options
  stdout            Output field usage information to the console

  Learn more at https://github.com/untergeek/es-fieldusage

Command: stdout help output

$ es-fieldusage stdout --help
Usage: es-fieldusage stdout [OPTIONS] SEARCH_PATTERN

  Display field usage information on the console for SEARCH_PATTERN

  $ es-fieldusage stdout [OPTIONS] SEARCH_PATTERN

Options:
  --show-report / --hide-report   Show a summary report  [default: show-report]
  --show-headers / --hide-headers
                                  Show block headers for un|accessed fields  [default: show-headers]
  --show-accessed / --hide-accessed
                                  Show accessed fields  [default: hide-accessed]
  --show-unaccessed / --hide-unaccessed
                                  Show unaccessed fields  [default: hide-unaccessed]
  --show-counts / --hide-counts   Show field access counts  [default: hide-counts]
  --delimiter TEXT                Value delimiter if access counts are shown  [default: ,]
  -h, --help                      Show this message and exit.

  Learn more at https://github.com/untergeek/es-fieldusage

Command file help output

$ es-fieldusage file --help
Usage: es-fieldusage file [OPTIONS] SEARCH_PATTERN

  Write field usage information to file for SEARCH_PATTERN

  $ es-fieldusage file [OPTIONS] SEARCH_PATTERN

Options:
  --show-report / --hide-report   Show a summary report  [default: show-report]
  --show-headers / --hide-headers
                                  Show block headers for un|accessed fields  [default: show-headers]
  --show-accessed / --hide-accessed
                                  Show accessed fields  [default: hide-accessed]
  --show-unaccessed / --hide-unaccessed
                                  Show unaccessed fields  [default: hide-unaccessed]
  --show-counts / --hide-counts   Show field access counts  [default: hide-counts]
  --per_index                     Create one file per index found
  --delimiter TEXT                Value delimiter if access counts are shown  [default: ,]
  -h, --help                      Show this message and exit.

  Learn more at https://github.com/untergeek/es-fieldusage

Command show-indices help output

$ es-fieldusage show-indices --help
Usage: es-fieldusage show-indices SEARCH_PATTERN

  Show indices on the console matching SEARCH_PATTERN

  $ es-fieldusage show_indices SEARCH_PATTERN

  This is included as a way to ensure you are seeing the indices you expect before using the file or stdout commands.

Options:
  -h, --help  Show this message and exit.

  Learn more at https://github.com/untergeek/es-fieldusage

Docker usage

Docker build

From the path which contains Dockerfile:

$ ./docker build . -t reponame/es-fieldusage:x.y.z

e.g.

docker build . -t untergeek/es-fieldusage:1.0.0

You can also build for x86_64 and arm64 if you have the appropriate buildx image setup:

docker buildx build --platform linux/amd64,linux/arm64 -t untergeek/es-fieldusage:1.0.0 --push .

Docker run

Example:

docker run -t --rm --name es-fieldusage -v /path/to/configfile/:/.esfieldusage -v $(pwd)/:/fileoutput untergeek/es-fieldusage:1.0.0 --config /.esfieldusage/config.yml show-indices 'index-*'

Explanation:

  • The -t flag indicates that you are interacting with a terminal application
  • --rm deletes the created Docker image after the run. Omitting this will result in a lot of created images that run once. Using --name will prevent this collision if --rm is omitted by reminding you that there is already an image with the same name.
  • --name is the name of the image to create. This is optional.
  • -v sets up volumes. /path/to/configfile/ is the local file path to where you have a YAML configuration file, if you choose to use one. The :/.esfieldusage portion of the volume map is the directory where that configuration file is expected in the Docker image. The second volume mapping is $(pwd):/fileoutput. /fileoutput is the dedicated file path on the Docker image where output from the file command will be written. By using $(pwd) it will map your present working directory so the files will appear there. Otherwise you can map another path here.
  • untergeek/es-fieldusage:1.0.0 is the repository/image:version to run.
  • --config /.esfieldusage/config.yml, as stated previously, if you intend to use a YAML configuration file, the path needs to be mapped as a volume, and then accessed this way. The filename should match whatever you actually have, and not necessarily config.yml
  • show-indices 'index-*' Everything after here is available as regular options and commands for es-fieldusage.

License

es-fieldusage is distributed under the terms of the Apache 2.0 license.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

es_fieldusage-1.1.2.tar.gz (14.4 kB view details)

Uploaded Source

Built Distribution

es_fieldusage-1.1.2-py3-none-any.whl (14.1 kB view details)

Uploaded Python 3

File details

Details for the file es_fieldusage-1.1.2.tar.gz.

File metadata

  • Download URL: es_fieldusage-1.1.2.tar.gz
  • Upload date:
  • Size: 14.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.5

File hashes

Hashes for es_fieldusage-1.1.2.tar.gz
Algorithm Hash digest
SHA256 834812102231b52e4d94fb6620c7605678fdd5060b1edad2f0ef1e9b7d4329b6
MD5 45e7de3e3a392483532199bd8d5bf0d5
BLAKE2b-256 501718be5f21ef794ad7d7acf72511f50d82d62415f07a3719a50f889f107665

See more details on using hashes here.

File details

Details for the file es_fieldusage-1.1.2-py3-none-any.whl.

File metadata

File hashes

Hashes for es_fieldusage-1.1.2-py3-none-any.whl
Algorithm Hash digest
SHA256 3f1078dcd4a6348335e073fa2b8d41446fb2ea7f477995205dbb33b85bb83758
MD5 7e176c69f9c8baf824dde200ed283304
BLAKE2b-256 82cf14d1b38921288d018429d6daa0f2c7ca7ce1d6a50cd6de428a1d98b8793e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page