montecarlodata 0.51.0

Monte Carlo's CLI

Monte Carlo CLI

Monte Carlo's Alpha CLI!

Installation

Requires Python 3.7 or greater. Normally you can install and update using pip. For instance:

pip install virtualenv
virtualenv venv
. venv/bin/activate

pip install -U montecarlodata

Developers of the CLI can use:

pip install virtualenv
make install
. venv/bin/activate
pre-commit install

Either way confirm the installation by running:

montecarlo --version

If the Python requirement does not work for you please reach out to [e-mail]. Docker is an option.

Quick start

First time users can configure the tool by following the onscreen prompts:

montecarlo configure

MCD tokens can be generated from the dashboard.

Any AWS profiles or regions should be for the account the Data Collector (DC) is deployed to. If you are not using the CLI for onboarding or managing a DC, you may leave the AWS configuration blank.

Use the --help flag for details on any advanced options (e.g. creating multiple montecarlo profiles) or see docs here.

That's it! You can always validate your connection with:

montecarlo validate

User settings

Any configuration set by montecarlo configure can be found in ~/.mcd/ by default.

The MCD ID and Token can be overwritten, or even set, by the environment:

• MCD_DEFAULT_API_ID
• MCD_DEFAULT_API_TOKEN

These two are required either as part of configure or as environment variables. For AWS, system defaults are used if not set as part of configure.

The following values can also be set by the environment:

• MCD_API_ENDPOINT - Overwrite the default API endpoint
• MCD_VERBOSE_ERRORS - Enable verbose logging on errors (default=false)

Help

Documentation for commands, options, and arguments can be found here.

You can also use montecarlo help to echo all help text or use the --help flag on any command.

Examples

Using Docker from a local installation

docker build -t montecarlo .
docker run -v ${HOME}/.aws/credentials:/root/.aws/credentials:ro \
            -e MCD_DEFAULT_API_ID='<ID>' \
            -e MCD_DEFAULT_API_TOKEN='<TOKEN>' \
            -e AWS_DEFAULT_PROFILE='<PROFILE>' \
            -e AWS_DEFAULT_REGION='us-east-1' \
            montecarlo --version

Replace --version with any sub-commands or options. If interacting with files those directories will probably need to be mounted too.

Configure a named profile with custom config-path

$ montecarlo configure --profile-name zeus --config-path .
Key ID: 1234
Secret:
AWS profile name []: shiva
AWS region [us-east-1]:

$ cat ./profiles.ini
[zeus]
mcd_id = 1234
mcd_token = 5678
aws_profile = shiva
aws_region = us-east-1

List active integrations

$ montecarlo integrations list
╒══════════════════╤══════════════════════════════════════╤══════════════════════════════════╕
│ Integration      │ ID                                   │ Created on (UTC)                 │
╞══════════════════╪══════════════════════════════════════╪══════════════════════════════════╡
│ Odin             │ 58005657-2914-4701-9a11-260ac425b14e │ 2021-01-02T01:30:52.806602+00:00 │
├──────────────────┼──────────────────────────────────────┼──────────────────────────────────┤
│ Thor             │ 926816bd-ab17-4f95-a953-fa14482c59de │ 2021-01-02T01:31:19.892205+00:00 │
├──────────────────┼──────────────────────────────────────┼──────────────────────────────────┤
│ Loki             │ 1cf1dc0d-d8ec-4c85-8e64-57ab2ad8e023 │ 2021-01-02T01:32:37.709747+00:00 │
╘══════════════════╧══════════════════════════════════════╧══════════════════════════════════╛

Apply monitors configuration

$ montecarlo monitors apply --namespace my-monitors

Gathering monitor configuration files.
- models/customer_success/schema.yml - Embedded monitor configuration found.
- models/customer_success/schema.yml - Monitor configuration found.
- models/lineage/schema.yml - Embedded monitor configuration found.

Modifications:
- ResourceModificationType.UPDATE - Monitor: type=stats, table=analytics:prod.customer_360
- ResourceModificationType.UPDATE - Monitor: type=categories, table=analytics:prod.customer_360
- ResourceModificationType.UPDATE - Monitor: type=stats, table=analytics:prod_lineage.lineage_nodes
- ResourceModificationType.UPDATE - Freshness SLI: table=analytics:prod.customer_360, freshness_threshold=30

Import DBT manifest

$ montecarlo import dbt-manifest --dbt-manifest-file target/manifest.json

Importing DBT objects into Monte Carlo catalog. please wait...

Imported a total of 51 DBT objects into Monte Carlo catalog.

Tests and Releases

Locally make test will run all tests. CircleCI manages all testing for deployment.

To publish a new release, simply add a new version tag, e.g. v1.0.0, and push that tag to GitHub. CircleCI will take care of publishing a new package to PyPI and generating documentation.

License

Apache 2.0 - See the LICENSE for more information.