great-expectations-cloud 0.0.47.dev0

Great Expectations Cloud

cloud

Quick Start

Python

Install

pip install great_expectations_cloud

Optional Dependencies

pip install 'great_expectations_cloud[sql]'

$ gx-agent --help
usage: gx-agent [-h] [--log-level LOG_LEVEL] [--skip-log-file SKIP_LOG_FILE] [--log-cfg-file LOG_CFG_FILE] [--version]

optional arguments:
  -h, --help            show this help message and exit
  --log-level LOG_LEVEL
                        Level of logging to use. Defaults to WARNING.
  --skip-log-file SKIP_LOG_FILE
                        Skip writing debug logs to a file. Defaults to False. Does not affect logging to stdout/stderr.
  --log-cfg-file LOG_CFG_FILE
                        Path to a logging configuration json file. Supersedes --log-level and --skip-log-file.
  --version             Show the gx agent version.

Set env variables

GX_CLOUD_ACCESS_TOKEN GX_CLOUD_ORGANIZATION_ID

Start the Agent

If you intend to run the agent against local services (Cloud backend or datasources) run the agent outside of the container.

gx-agent

Docker

Building and running the Agent with Docker

Dev Setup

See also CONTRIBUTING.md

1. Install poetry
   • pipx install poetry
2. Set up virtual environment and install dependencies.
   • poetry install --sync
3. Activate your virtual environment.
   • poetry shell
4. Set up precommit hooks
   • pre-commit install

Developer Tasks

Common developer tasks are available via invoke (defined in tasks.py)

invoke --list to see available tasks.

Synchronize Dependencies

To ensure you are using the latest version of the core and development dependencies run poetry install --sync. Also available as an invoke task.

invoke deps

Updating poetry.lock dependencies

The dependencies installed in our CI and the docker build step are determined by the poetry.lock file.

To update only a specific dependency (such as great_expectations) ...

poetry update great_expectations

To resolve and update all dependencies ...

poetry lock

In either case, the updated poetry.lock file must be committed and merged to main.

Release to PyPI and Docker

To release a new version to PyPI the version must be incremented. New versions are automatically published to PyPI when merging to main.

invoke version-bump

A new docker tag will also be generated and pushed to Docker Hub.

Building and Running the GX Agent Image

To build the GX Agent Docker image, run the following in the root dir:

invoke docker

Running the GX Agent:

invoke docker --run

or

docker run --env GX_CLOUD_ACCESS_TOKEN="<GX_TOKEN>" --env GX_CLOUD_ORGANIZATION_ID="<GX_ORG_ID>" gx/agent

Now go into GX Cloud and issue commands for the GX Agent to run, such as generating an Expectation Suite for a Data Source.

Note if you are pushing out a new image update the image tag version in containerize-agent.yaml. The image will be built and pushed out via GitHub Actions.

Example Data

The contents from /examples/agent/data will be copied to /data for the docker container.

Adding an action to the agent

1. Make a new action in great_expectations_cloud/agent/actions/ in a separate file.
2. Register your action in the file it was created in using great_expectations_cloud.agent.event_handler.register_event_action(). Register for the major version of GX Core that the action applies to, e.g. register_event_action("1", RunCheckpointEvent, RunCheckpointAction) registers the action for major version 1 of GX Core (e.g. 1.0.0).
3. Import your action in great_expectations_cloud/agent/actions/__init__.py

Note: The agent is core-version specific but this registration mechanism allows us to preemptively work on actions for future versions of GX Core while still supporting the existing latest major version.