AutoDQ Analytics

AutoDQ is an end-to-end analytics workflow framework for tabular data. It profiles datasets, diagnoses quality problems, recommends and reviews cleaning actions, engineers features, trains explainable models, generates visualizations and dashboards, and runs complete workflows from Python, Jupyter, the command line, or standalone .adql notebooks.

Highlights

• CSV, XLSX, and XLS dataset loading
• Dataset profiling, semantic inference, and quality scoring
• Missing-value, duplicate, outlier, datatype, and leakage diagnosis
• Knowledge-aware cleaning recommendations and approval workflows
• Audited missing-value fills/removals, exact-duplicate inspection/removal, manual row editing, domain validation, and outlier treatment
• Descriptive statistics, distribution analysis, and correlations
• Feature engineering and ML-readiness analysis
• Regression and classification with prediction uncertainty
• SHAP explanations and publication-ready SHAP plots
• BLUE regression diagnostics, visual interpretation, and prescriptions
• Reusable visualization objects, galleries, HTML reports, and dashboards
• Multi-workspace project isolation and model persistence
• project.auto() for an automated workflow
• A headless pipeline runner with run specifications, stable exit codes, artifact discovery, bounded logs, and replaceable source/result connectors
• ADQL files with executable notebook cells, direct named-dataset workflows, ASSERT quality gates and reusable test suites, session inspection, versioned schema contracts, statistical drift detection, explicit datatype formatting, and rich VS Code output

Requirements

• Python 3.10 or newer
• macOS, Linux, or Windows

Installation

Install the released package from PyPI:

python -m pip install autodq

Verify the active installation:

autodq --version
python -c "import autodq; print(autodq.__version__)"

Version 0.1.23 is available on PyPI. To work on AutoDQ itself, install directly from the project source:

git clone https://github.com/josephubani/autodq-analytics.git
cd autodq-analytics
python -m venv .venv
source .venv/bin/activate
python -m pip install .

On Windows PowerShell, activate the environment with:

.venv\Scripts\Activate.ps1

For editable development and release tools:

python -m pip install -e ".[dev]"

New users can follow the quickstart for a safe review workflow in Python and ADQL.

Python quick start

from autodq import AutoDQ

project = AutoDQ("datasets/sample/sales.csv", target="Revenue")

profile = project.profile()
diagnosis = project.diagnose()
recommendations = project.recommend()
review = project.review_cleaning()

review.approve_all()
project.clean()
validation = project.validate_cleaning()

chart = project.visualize(
    chart="bar",
    x="Region",
    y="Revenue",
    title="Revenue by Region",
    x_label="Region",
    y_label="Average revenue",
    theme="journal",
)
chart.show()

ADQL notebooks

ADQL is AutoDQ's standalone analytics language. A .adql file can contain named executable cells and markdown cells, while retaining project state between executions. In the AutoDQ ADQL 0.3.16 VS Code extension, REVIEW opens a theme-aware interactive panel for approvals, rejections, previews, manual row edits, audit inspection, and applying reviewed work to CLEANED.

# %% [Dataset]
DATASET "sales.csv" TARGET Revenue;

# %% [Data quality]
PROFILE;
DIAGNOSE;
RECOMMEND;

# %% [Complete missing values]
MISSING SUMMARY;
MISSING FILL ALL STRATEGY auto;
DUPLICATES SUMMARY;
DUPLICATES DROP KEEP first;
CLEANING APPLY;
LET cleaned_sales = CLEANED;

# %% [Quality gate]
ASSERT SUITE ADD sales_gate Transaction_ID UNIQUE;
ASSERT SUITE ADD sales_gate Revenue MIN 0;
ASSERT SUITE RUN sales_gate FAIL_ON error;

# %% [Schema and drift gates]
CONTRACT sales_v1 FROM cleaned_sales
    VERSION 1.0.0 EXTRA_COLUMNS warning;
CONTRACT sales_v1 REQUIRE Revenue TYPE numeric NOT NULL MIN 0;
CHECK CONTRACT sales_v1 ON cleaned_sales FAIL ON error;

BASELINE sales_baseline FROM cleaned_sales;
CHECK DRIFT sales_baseline ON cleaned_sales
    CONTRACT sales_v1 SENSITIVITY normal FAIL ON warning;

# %% [Regional analysis]
SELECT Region,
       SUM(Revenue) AS total_revenue,
       COUNT(*) AS transactions
FROM CURRENT
GROUP BY Region
ORDER BY total_revenue DESC;

# %% [Visualization]
VISUALIZE bar X Region Y Revenue
    TITLE "Revenue by Region"
    THEME journal;

# %% [Session]
SESSION;

Customer datasets can use the built-in email format without maintaining a regular expression:

ASSERT Email FORMAT email SEVERITY warning;

Create a reusable dataset snapshot with LET, then query, profile, activate, or export its assigned name:

CLEAN;
LET cleaned_sales = CLEANED;
EXPORT cleaned_sales TO "exports/cleaned-sales.csv" OVERWRITE;

LET regional_summary = SELECT Region, SUM(Revenue) AS total_revenue
                       FROM CURRENT GROUP BY Region;
SELECT * FROM regional_summary ORDER BY total_revenue DESC;

Convert string dates with an explicit format and control numeric precision:

SET TYPE Created_At datetime FORMAT "DD/MM/YYYY HH:mm:ss";
SET TYPE Imported_At datetime FORMAT MIXED DAYFIRST true UTC true;
SET TYPE Revenue decimal DECIMALS 2;

Date patterns may use familiar tokens such as YYYY, MM, DD, HH, mm, and ss, or Python strftime directives such as %Y-%m-%d.

Run the same automatic workflow available as project.auto() directly from an ADQL cell:

# %% [Automatic workflow]
AUTO MODE full
    VISUALIZE true
    APPLY_FEATURES true
    ALGORITHM random_forest_regressor
    REPORT "reports/auto-report.html"
    CONTINUE_ON_ERROR true;

review mode analyzes data and prepares cleaning actions without applying them. clean applies approved cleaning actions. full continues through modeling, prediction, and explainability when a target is available. The ADQL notebook renders the automatic stages, status, timing, and next actions as a collapsible rich result.

Run the file from a terminal:

autodq run analysis.adql

Inspect or validate cells without executing the workflow:

autodq cells analysis.adql
autodq validate analysis.adql

Run through a particular cell:

autodq run analysis.adql --through-cell 3

VS Code support

The Python distribution bundles the AutoDQ ADQL extension for local or offline installation. Install it with:

autodq vscode install

Reload VS Code after installation. .adql files then receive syntax highlighting, named notebook cells, rich tables and charts, cell-by-cell execution, saved output restoration after reopening, and an AutoDQ file icon. Press Save after execution to persist the displayed notebook outputs.

For a normal VS Code installation, download the latest autodq-adql-VERSION.vsix from GitHub Releases, then choose Extensions → … → Install from VSIX. From a terminal:

code --install-extension autodq-adql-VERSION.vsix --force

The VSIX is packaged automatically and attached to an adql-vVERSION GitHub Release. Manually installed VSIX extensions do not receive automatic updates, so install the new VSIX when a release is announced. See the VS Code extension distribution guide.

Command-line interface

autodq --version
autodq run workflow.adql
autodq pipeline --workflow workflow.adql --result run-result.json
autodq pipeline --spec pipeline-run.json
autodq validate workflow.adql
autodq cells workflow.adql
autodq vscode path
autodq vscode install

The package can also be executed as a Python module:

python -m autodq --version

Development

Run the test suite:

python -m unittest discover -s tests

References: Python API, ADQL user guide, ADQL 2.5 specification, pipeline runner, troubleshooting, release guide, and changelog.

Build and verify release artifacts:

python -m build
python -m twine check dist/*
python scripts/check_distribution.py dist

For the complete release process, see the AutoDQ release guide.

Documentation

• ADQL language reference
• Formal ADQL 2.5 specification
• Machine-readable ADQL grammar
• Quickstart
• Pipeline runner
• Troubleshooting
• VS Code Marketplace publishing
• System architecture
• Plugin development
• Project roadmap
• Package and release procedure
• AutoDQ 0.1.23 release notes

License

AutoDQ is released under the MIT License.

Author

Joseph Ubani
Master of Data Analytics, University of Niagara Falls Canada