Skip to main content

DataOps TestGen

apache 2.0 license Badge PRs Badge Latest Version Docker Pulls Documentation Static Badge

DataOps TestGen is an open-source data quality tool that profiles tables, generates tests from the profiling results, and runs them on a schedule with freshness, volume, and schema monitoring.

The open-source edition includes the full web UI, not just a library. You also get a REST API and an MCP server, so you can drive TestGen from your pipelines or an AI assistant. It is Apache 2.0 licensed and runs the same core engine and UI as TestGen Enterprise.

Interactive Product Tour | Product page | Documentation

What it does

  • Profiles every column in a table group and records type-specific statistics.
  • Flags hygiene issues in new data, so you can review a dataset before it reaches production and send problems back to the supplier.
  • Generates data quality validation tests from the profile. You can add your own SQL tests too.
  • Runs tests on a schedule, as SQL inside your database. Your data stays where it is.
  • Watches each table between test runs with freshness, volume, and schema monitors.
  • Gives you one place to manage data quality across datasets, locations, and teams.

DataOps TestGen: hygiene review of new data

DataOps TestGen: continuous monitoring and quality testing

DataOps TestGen: one place to manage data quality

Supported databases

As of TestGen 5.134.0, TestGen connects to 14 databases:

  • Snowflake
  • Databricks SQL
  • DuckDB
  • Azure Synapse Analytics
  • Azure SQL Database
  • Microsoft SQL Server
  • Microsoft OneLake
  • Google BigQuery
  • Amazon Redshift
  • Amazon Aurora PostgreSQL
  • Oracle Database 12c and later
  • SAP HANA
  • PostgreSQL
  • Salesforce Data 360

TestGen also reads CSV, Parquet, and JSON files on a local path, in Amazon S3, or in Azure Blob Storage, with no database in front. Every source works in both the open-source and Enterprise editions.

To set up a connection, see Connect your data.

Quick start

The dk-installer program installs TestGen in either Docker or pip mode.

You need 2 CPUs, 8 GB memory, 20 GB disk space. pip mode needs Python 3.9+ to run the installer. Docker mode also needs Docker 27+ and Docker Compose 5.0+.

Download the installer into a new, empty directory:

curl -o dk-installer.py 'https://raw.githubusercontent.com/DataKitchen/data-observability-installer/main/dk-installer.py'

On Windows, you can instead download dk-installer.exe and double-click it.

Install TestGen:

python3 dk-installer.py tg install

The installer checks for Docker and asks you to pick Docker or pip. Pass --docker or --pip to skip the prompt. When it finishes, it writes credentials to dk-tg-credentials.txt, generates demo data, and opens the TestGen UI in your browser. Use --no-demo to skip the demo data.

Upgrade later with python3 dk-installer.py tg upgrade. Restart with python3 dk-installer.py tg start.

For complete instructions, see the documentation:

Then walk through the Data Observability Overview Demo, and connect your own database.

Use it from an AI assistant (MCP)

TestGen ships a Model Context Protocol (MCP) server in the open-source edition. Connect Claude, Claude Code, Cursor, GitHub Copilot, or any MCP client, and ask about your data in plain English. The assistant can profile tables, generate and run tests, and triage failures. Every tool runs under your own TestGen role, so the assistant can do only what you can do.

The installer serves the API and MCP server on port 8530 by default. Change it with --api-port.

Setup instructions: docs.datakitchen.io/testgen/mcp

Open source vs Enterprise

The core profiling, testing, and monitoring engine is free and open source, with unlimited tables and unlimited data volume. Enterprise adds team access, security, Commercial Pharma tests, source-data controls for AI, and support.

Open source Enterprise
License Apache 2.0 Commercial
Users 1 Unlimited, $100 per user per month
Database connections 1 Unlimited, $100 per connection per month
Projects 1 Unlimited
Enterprise security Included
Commercial Pharma test types (12) Included
Hide source data from AI clients (MCP) Included
Support Community Slack DataKitchen support

See datakitchen.io/pricing for current pricing.

Documentation

Community

Talk with other data practitioners who build with DataKitchen. Share what you know, get help, and contribute.

Contributing

For details on contributing or running the project for development, see the contributing guide.

License

DataOps TestGen is Apache 2.0 licensed. See LICENSE.

Metadata

Release files for dataops-testgen 5.189.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dataops-testgen 5.189.4
File Size Uploaded Provenance Actions
dataops_testgen-5.189.4.tar.gz 6.0 MB Without provenance Details for dataops_testgen-5.189.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dataops-testgen 5.189.4
File Interpreter ABI Platform Actions
dataops_testgen-5.189.4-py3-none-any.whl Python 3 none any Details for dataops_testgen-5.189.4-py3-none-any.whl

Total release size: 12.4 MB

Release files / dataops_testgen-5.189.4.tar.gz

Download URL dataops_testgen-5.189.4.tar.gz
Size 6.0 MB
Tags Source
SHA-256 checksum
How to use checksums
6915ce649095339f56cef8651201ff27d6514aefa1fcab502dc8997dfc7ea4cf
BLAKE2b-256 checksum
How to use checksums
6f74dd1b0ebd2ffe5f6a2845b659b527ef73616715d109f532a9d5345356b3c4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.18

Release files / dataops_testgen-5.189.4-py3-none-any.whl

Download URL dataops_testgen-5.189.4-py3-none-any.whl
Size 6.5 MB
Tags Python 3
SHA-256 checksum
How to use checksums
9208093a80d7e9fb99daefaf6aa8ab13449f009fd33dca89099fac1e8c82858d
BLAKE2b-256 checksum
How to use checksums
2bd358d4a767f2dfc7d2e9d4cc1c829c5a76f73232422823dc2ced0d1bbe27d7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.18

Security for dataops-testgen 5.189.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

5.189.4 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page