DataAdvisor (data_advisor)

Autonomous Data Science Diagnostic Engine & Exam Code Generator
Engineered for students, practitioners, and data scientists to audit unknown datasets, formulate statistical hypotheses, and generate copy-pasteable, production-ready Python pipelines during timed data analysis examinations.

---

🚀 Key Highlights

1. Deterministic Auditing Engine: Deeply profiles any pd.DataFrame (or CSV, TSV, Excel, Parquet file) in seconds.
2. Automated Hypothesis Testing:
   • Numeric vs. Numeric: Pearson & Spearman correlations with $p$-values.
   • Categorical vs. Categorical: Chi-Square Test of Independence with Cramér's V effect size.
   • Numeric vs. Binary Target: Welch's Two-Sample t-test & Mann-Whitney U test.
   • Numeric vs. Multi-class Target: One-Way ANOVA ($F$-test) & Kruskal-Wallis $H$-test.
3. Outlier & Skewness Diagnostics: Combines IQR fences (1.5× and 3.0×) and Z-score criteria ($|z| > 3$); automatically recommends np.log1p, Yeo-Johnson power transformations, or winsorization.
4. Target & Model Strategy Selector: Automatically identifies whether the task is Regression or Classification (Binary or Multiclass), detects critical class imbalance, and selects baseline models (LogisticRegression, Ridge, RandomForest).
5. Copy-Pasteable Code Generator: Emits clean, PEP 8 compliant, runnable Python scripts containing complete Pandas, Scipy, and Scikit-Learn code tailored to the audited dataset.

---

📦 Installation

To install data_advisor in editable mode directly from your local repository:

# Navigate to the project root directory
cd "path/to/Custom Library"

# Install in editable mode with dependencies
pip install -e .

Verified Dependencies

• Python >= 3.9
• pandas >= 2.0.0
• numpy >= 1.24.0
• scipy >= 1.10.0
• scikit-learn >= 1.2.0
• matplotlib >= 3.7.0
• seaborn >= 0.12.0

---

⚡ Quickstart: The 1-Line Exam Workflow

When handed an unfamiliar exam dataset (exam_dataset.csv), run the top-level data_advisor function in your Jupyter Notebook or Python terminal:

from data_advisor import data_advisor

# 1-Line Diagnostic & Code Generation:
# Outputs:
# 1. Full diagnostic report stating what all analysis to perform
# 2. Executable, copy-pasteable Python code to execute all steps
data_advisor("exam_dataset.csv")

Or with an optional target column or output script:

data_advisor("exam_dataset.csv", target="churn", output_script="solution.py")

You can also use the object-oriented API:

from data_advisor import DataAdvisor

advisor = DataAdvisor("exam_dataset.csv", target="churn", id_cols=["customer_id"])
advisor.report()                          # Diagnostic report
advisor.generate_script("solution.py")    # Full Python script
clean_code = advisor.get_code_block("cleaning")  # Specific stage code

Or directly from your command-line terminal:

# Diagnostic report + code prompt
data-advisor exam_dataset.csv

# Print complete executable Python code directly to the terminal
data-advisor exam_dataset.csv --code

# Save complete pipeline to a Python file
data-advisor exam_dataset.csv --output solution.py

---

🔍 Instant Code Snippet Search: search(key)

data_advisor includes a search engine that searches any keyword across your own local code folder (code_bank/).

How It Works:

1. Your Folder: Simply paste your custom Python scripts (.py), Jupyter Notebooks (.ipynb), Arduino/ESP32 sketches (.ino, .cpp), or text files into code_bank/.
2. Dynamic Scanning: The search engine scans code_bank/ in real time on every search call—no indexing or restarts required. You can add, edit, or delete files anytime.
3. Any Keyword: Search for single terms, phrases, or keyword lists.

from data_advisor import search

# 1. List all available snippets in your code_bank
search()

# 2. Search for any keyword (e.g., machine learning, IoT, algorithms)
search("linear regression")
search("dht22")
search("kmeans")
search("wokwi")

# 3. Search with multiple keywords or a list
search(["esp32", "temperature"])
search("decision tree, entropy")

# 4. Search within a custom directory
search("apriori", folder="my_other_codes")

Or search directly via the CLI:

# List all files currently in code_bank
data-advisor --search "list"

# Search for any keyword
data-advisor --search "linear regression"
data-advisor --search "wokwi"

---

📄 Complete File Viewing, Retrieval & Execution: get_file(), read_file(), run_file()

To retrieve or execute the entire complete code file from code_bank/ (not just search snippets):

1. View / Print Complete File

from data_advisor import get_file, view_file

# By question number:
get_file("q1")                     # Complete code for Question 1
get_file(24)                       # Complete code for Question 24
get_file("q32", line_numbers=True) # With line numbers

# By filename or keyword:
get_file("sketch.ino")             # Arduino / ESP32 sketch
get_file("subscriber.py")

2. Retrieve Complete Raw Code String (No Printing)

from data_advisor import read_file

# Return complete raw code as a Python string
code_string = read_file("q24")
print(f"Loaded {len(code_string)} characters")

3. Run Any File Directly

from data_advisor import run_file

# Execute a Python file directly via subprocess
run_file("q24")
run_file("iss")

4. Direct CLI Commands

# Print complete file to console
data-advisor --get q1
data-advisor -g q24 --line-numbers
data-advisor --get sketch.ino

# Execute script directly
data-advisor --run q24
data-advisor -r 14

# List all available code files
data-advisor --list

---

🧠 Decision Rules Matrix

1. Data Cleaning & Integrity Rules

Condition	Severity	Recommended Action	Generated Code
Duplicates Detected (> 0)	WARNING / CRITICAL	Drop duplicate rows	df = df.drop_duplicates().reset_index(drop=True)
High Missingness (>= 60%)	CRITICAL	Drop column entirely	df = df.drop(columns=[col])
Low Missingness (< 5%)	INFO	Impute median (num) / mode (cat)	df[col] = df[col].fillna(df[col].median())
Moderate Missingness (5% - 60%)	WARNING	Skewed: Median; Symmetric: Mean	SimpleImputer(strategy='median')
Constant Column (variance = 0)	CRITICAL	Drop column	df = df.drop(columns=[col])
Identifier Column (IDs, hashes)	WARNING	Exclude from feature matrix $X$	X = df.drop(columns=[id_col])
Date String Column	RECOMMENDED	Parse date & extract temporal features	.dt.year, .dt.month, .dt.dayofweek

2. Distribution & Outlier Rules

Metric	Condition	Transformation Strategy	Rationale
Skewness	$> 1.2$ (Non-negative)	np.log1p(df[col])	Stabilizes variance, compresses long positive tail
Skewness	$> 1.2$ (Contains negatives)	PowerTransformer(method='yeo-johnson')	Normalizes non-Gaussian data with non-positive values
Skewness	$< -1.2$ (Left-skewed)	PowerTransformer(method='yeo-johnson')	Corrects severe left tail
IQR Outliers	$> 5%$ of samples	IQR clipping / RobustScaler()	Caps extreme leverage points

3. Categorical Processing Rules

Cardinality	Encoding Strategy	Implementation
Binary ($2$ levels)	Direct Binary Mapping ($0/1$)	df[col].map({'No': 0, 'Yes': 1})
Low-to-Medium ($3$ to $10$ levels)	One-Hot Encoding (drop_first=True)	pd.get_dummies(..., drop_first=True, dtype=int)
High ($> 10$ levels)	Frequency Encoding / Target Encoding	df[col].map(df[col].value_counts(normalize=True))

4. Hypothesis Testing Decision Engine

Variable Types	Primary Test	Non-Parametric Alternative	Effect Size Metric
2 Numeric Features	Pearson Correlation ($r, p$)	Spearman Rank Correlation ($\rho, p$)	Correlation Coefficient ($r$)
2 Categorical Features	Chi-Square Test of Independence ($\chi^2$)	Fisher's Exact Test (for $2\times 2$)	Cramér's $V$
1 Numeric + 1 Binary Cat	Welch's Independent $t$-test	Mann-Whitney $U$ test	Cohen's $d$ / Rank-biserial
1 Numeric + 1 Multi-class Cat	One-Way ANOVA ($F, p$)	Kruskal-Wallis $H$-test	Eta-squared ($\eta^2$)

5. Target Variable & Model Baseline Matrix

Target Type	Detected Problem	Imbalance Threshold	Recommended Baselines	Evaluation Metrics
Continuous Numeric	Regression	N/A	Linear Regression, Ridge ($L_2$), Random Forest Regressor	RMSE, MAE, $R^2$ Score
Binary / Discrete	Classification	Minority $< 20%$ (Ratio $\ge 3:1$)	Logistic Regression (class_weight='balanced'), Random Forest Classifier	F1-Score, PR-AUC, ROC-AUC, Confusion Matrix

---

📂 Project Architecture

Custom Library/
├── pyproject.toml                     # Modern setuptools packaging & dependency metadata
├── README.md                          # Comprehensive library documentation & exam cheat sheet
├── src/
│   └── data_advisor/
│       ├── __init__.py                # Package exports (DataAdvisor, DataScanner, RuleEngine)
│       ├── advisor.py                 # Primary facade class & CLI reporter
│       ├── scanner.py                 # Core profiling engine & statistical test runner
│       ├── rules.py                   # Heuristic recommendation engine & severity prioritizer
│       ├── codegen.py                 # Sequential Python script & code block generator
│       ├── visualizer.py              # Publication-ready Matplotlib & Seaborn code generator
│       └── cli.py                     # Command-line entry point (`data-advisor`)
├── tests/
│   └── test_advisor.py                # Automated pytest suite covering regression & classification
└── examples/
    ├── demo_exam_workflow.py          # End-to-end exam simulation walkthrough
    ├── student_exam_data.csv          # Sample dataset with outliers, nulls, and duplicates
    └── generated_analysis_pipeline.py # Fully generated, executable production script

---

🧪 Running Tests

Run the test suite using pytest:

python -m pytest -v

All tests inspect and validate:

• Missing value audit & column dropping (>60%).
• Identifier and constant column detection.
• Skewness and IQR/Z-score outlier calculations.
• Parametric and non-parametric hypothesis testing.
• Target classification and regression auto-detection.
• Full end-to-end Python script code generation and execution.

---

📜 License

MIT License. Built for rigorous data science education, competitive coding, and exam excellence.