arize_toolkit 1.0.18

A library to interact with Arize AI APIs

📚 Documentation

Quick Links

• Home - Main documentation page
• Quickstart Guide - Get started quickly with Arize Toolkit
• CLI Reference - Command-line interface documentation

Tools Documentation

Tool Category	Description
Project/Model Tools	Access and manage projects/models, retrieve performance metrics and traces/inference volumes
Monitor Tools	Create, copy, and manage monitors and alerting
Custom Metrics Tools	Create and manage custom metrics
Language Model Tools	Work with prompts, annotations, evaluations, and LLM features
Space & Organization Tools	Navigate and manage spaces, organizations, and projects
Data Import Tools	Import data from cloud storage and databases
Trace Tools	Query traces and spans, discover columns, and export trace data
Dashboard Tools	Create and manage dashboards
Utility Tools	Client configuration and utility functions

Extensions

Extension	Description
Prompt Optimization	Automated prompt improvement based on historical performance

Claude Code Plugin Marketplace

This repository is a Claude Code plugin marketplace with AI-assisted tools for the Arize platform.

# Add the marketplace
claude plugin marketplace add duncankmckinnon/arize_toolkit

# Install plugins
claude plugin install arize-toolkit@arize-toolkit
claude plugin install arize-toolkit-dev@arize-toolkit

Plugin	Skills	Description
arize-toolkit	arize-toolkit-cli, arize-traces	Manage Arize resources via CLI, debug traces
arize-toolkit-dev	arize-graphql-analytics, new-query-workflow	Developer workflows for adding new functionality

See the Plugin Marketplace docs for full details.

Arize Claude Code Tracing

For automatic tracing of Claude Code sessions to Arize AX or Phoenix, install the tracing plugin from the Arize Claude Code Plugin marketplace:

# Add the Arize plugin marketplace
claude plugin marketplace add Arize-ai/arize-claude-code-plugin

# Install the tracing plugin
claude plugin install claude-code-tracing@arize-claude-plugin

Plugin	Skills	Description
claude-code-tracing	setup-claude-code-tracing	Automatic tracing of Claude Code sessions with OpenInference spans (9 lifecycle hooks)

For Developers

• Development Guide - Information about extending the toolkit
• Integration Tests - Running integration tests
• Claude Code Plugins - Plugin development and skill authoring

Disclaimer

Although this package is used for development work with and within the Arize platform, it is not an Arize product. It is a open source project developed and maintained by an Arize Engineer. Feel free to add issues or reach out for help in the Arize community Slack channel.

Overview

Arize Toolkit is a set of tools packaged as a Python client that lets you easily interact with Arize AI APIs. Here's a quick overview of the main features in the current release:

• Access and manage models
• Retrieve performance metrics over a time period
• Retrieve inference volume over a time period
• Create, copy, and manage custom metrics
• Create, copy, and manage monitors and alerting
• Work with LLM features like prompts and annotations
• Import data from cloud storage (S3, GCS, Azure) and databases (BigQuery, Snowflake, Databricks)
• Create, update, and delete data import jobs with full lifecycle management
• Prompt Optimization Extension (optional): Automatically optimize prompts using meta-prompt techniques with feedback from evaluators
• Trace Inspection: List traces, retrieve spans with full attributes, discover available columns, and export to DataFrames or CSV
• Command-Line Interface (optional): Manage all Arize resources directly from the terminal with Rich table output and JSON mode

Installation

pip install arize_toolkit

Optional Dependencies

Command-Line Interface

To use the arize_toolkit CLI, install with the cli extras:

pip install arize_toolkit[cli]

Prompt Optimization Extension

For automated prompt optimization using meta-prompt techniques, install with the prompt_optimizer extras:

pip install arize_toolkit[prompt_optimizer]

Command-Line Interface

The CLI wraps all Client functionality so you can manage models, monitors, prompts, and more directly from the terminal.

Quick Start

# One-time setup — saves credentials to ~/.arize_toolkit/config.toml
arize_toolkit config init

# List models (Rich table output by default)
arize_toolkit models list

# Same thing, using the "projects" alias
arize_toolkit projects list

# JSON output, pipe to jq
arize_toolkit --json models list | jq '.[].name'

# Get monitor details for a specific model
arize_toolkit monitors get "accuracy-alert" --model "fraud-detection-v3"

# Create a performance monitor
arize_toolkit monitors create-performance "accuracy-alert" \
    --model "fraud-detection-v3" \
    --environment production \
    --performance-metric accuracy \
    --threshold 0.95

# Use a named profile for a different environment
arize_toolkit --profile staging spaces list

Global Options

Flag	Description
--version	Show the installed version
--profile NAME	Use a named configuration profile
--json	Output raw JSON instead of Rich tables
--api-key KEY	Override the API key
--org NAME	Override the organization
--space NAME	Override the space
--app-url URL	Override the Arize app URL

Command Groups

Group	Description
config	Manage configuration profiles (init, list, show, use)
spaces	List, create, and switch spaces or orgs
orgs	List and create organizations
users	Search users and manage space membership
models / projects	List models, check volume, pull performance metrics
monitors	Create, list, copy, and delete monitors
prompts	Manage prompt templates and versions
custom-metrics	Create and manage custom metrics
evaluators	Manage LLM and code evaluators
dashboards	Create and view dashboards
traces	List traces, get spans, and discover available columns
imports	Manage file and table import jobs

See the full CLI documentation for detailed usage and examples.

Client Setup

The Client class is the entrypoint for interacting with the toolkit. It provides maintains the connection information for making requests to the Arize APIs, and offers a wide range of operations for interacting with models, monitors, dashboards, and more.

API Key

To create a client, you need to provide your Arize API key. Use this reference to get your API key from the Arize UI.

Organization and Space

You will also need to provide an organization name and space name. To give some context, models are scoped to a space, and the space is scoped to an organization. These can be found by navigating to the Arize UI and looking at the upper left dropdown from the landing page. You will first select an organization from the dropdown and then pick the space from the available spaces in that organization.

For the example below, the organization is Demo Org and the space is Test Space.

For On Prem deployments

For SaaS users, the default API endpoint is always going to be https://api.arize.com. If you are using an on prem deployment of Arize, you will need to provide the api_url parameter. This parameters should just be the base url of your Arize instance.