DeepFeature

AI Data Scientist – Automate End‑to‑End Machine Learning Pipelines

DeepFeature is an open‑source Python framework that acts as an autonomous AI Data Scientist. Just give it a dataset and a target column, and it will:

• Infer the problem type (classification/regression)
• Perform exploratory data analysis (EDA)
• Preprocess data (impute, scale, encode)
• Suggest and train multiple ML models (with optional LLM support)
• Evaluate and select the best model
• Export the entire pipeline as a Jupyter notebook or Python script
• Save trained model artifacts for later use

All decisions are transparent, and the generated code is editable – so you can run, modify, and reuse it.

---

Features

• Automatic Planning – detects problem type and suggests a workflow.
• EDA – profiles columns, finds issues (missing values, constants, high cardinality).
• Preprocessing – handles numeric (median imputation + scaling) and categorical (most‑frequent imputation + one‑hot encoding) features.
• LLM‑Powered Code Generation – uses OpenAI, Anthropic, Groq, Ollama, etc. to write all pipeline code.
• Any Model – supports scikit‑learn, XGBoost, LightGBM, CatBoost, and more, dynamically loaded from import paths.
• Artifact Saving – automatically saves the best model, metrics, and metadata to disk.
• Export – export the generated code as a .ipynb notebook or .py script.
• Optional Module Installation – if a model library is missing, asks the user to install it.
• Provider‑Agnostic – works with any LLM provider via litellm.

---

Installation

pip install deepfeature

Optional Dependencies

For specific LLM providers:

pip install "deepfeature[openai]"
pip install "deepfeature[anthropic]"
pip install "deepfeature[groq]"

For notebook visualisations (when exporting):

pip install "deepfeature[notebook]"

To install everything:

pip install "deepfeature[all]"

---

Quickstart

from deepfeature import DeepFeature

# Initialize the agent (using Groq as the LLM provider)
agent = DeepFeature(
    llm_provider="groq",
    llm_config={"model": "mixtral-8x7b-32768"},
    api_key="gsk_...",          # or set GROQ_API_KEY env var
    verbose=True,
)

# Run the full pipeline
result = agent.run(
    dataset="path/to/titanic.csv",
    target="Survived",
)

# Check the results
print(result["report"]["summary"])
print(result["evaluation"])

# Export the generated code as a Python script
agent.export_file(result, "output/pipeline.py", format="script")

---

Configuration

LLM Providers

Provider	llm_provider	Required env var
OpenAI	"openai"	OPENAI_API_KEY
Anthropic	"anthropic"	ANTHROPIC_API_KEY
Groq	"groq"	GROQ_API_KEY
Ollama (local)	"ollama"	none (uses http://localhost:11434)
Together AI	"together"	TOGETHER_API_KEY

You can also pass an existing LLMClient instance:

from deepfeature.llm import LLMClient
client = LLMClient(provider="openai", model="gpt-4")
agent = DeepFeature(llm_client=client)

Saving Artifacts

By default, the best model, metrics, and metadata are saved under artifacts/run_<timestamp>/.
You can change the directory or disable saving:

agent = DeepFeature(artifacts_dir="my_models", save_artifacts=False)

Exporting Code

# Export as Jupyter notebook (default)
agent.export_file(result, "output/analysis.ipynb", format="notebook")

# Export as Python script
agent.export_file(result, "output/pipeline.py", format="script")

---

How It Works

The pipeline is orchestrated with LangGraph and consists of these steps:

1. Loader – reads the CSV.
2. Planner – infers problem type (LLM‑assisted if available).
3. EDA – profiles columns, detects issues.
4. Preprocessor – cleans and transforms data.
5. Training – suggests models via LLM (or fallback), cross‑validates, selects best.
6. Evaluation – computes metrics on the full dataset (or a hold‑out set if you add it).
7. Reporting – compiles a summary report.
8. Export – generates code blocks and saves them as a notebook/script.

All generated code is produced by the LLM and cleaned for execution.

---

Code Structure

deepfeature/
├── agents/            # modular pipeline steps
├── exporters/         # notebook/script export
├── llm/               # unified LLM client (litellm)
├── models/            # Pydantic models for state
├── nodes/             # LangGraph nodes
├── utils/             # helpers (permission, module install)
├── agent.py           # main DeepFeature class
├── graph.py           # LangGraph workflow
├── saver.py           # artifact saving/loading
└── prompts.py         # (coming soon) centralised prompt templates

---

Example Notebook

You can also explore the exported notebook interactively:

agent.export_file(result, "analysis.ipynb", format="notebook")

Then open it in Jupyter and run it step by step – the code is fully editable.

---

Development

Install the package in development mode:

git clone https://github.com/Mindlord-rex/deepfeature.git
cd deepfeature
pip install -e ".[all]"

Run tests (coming soon):

pytest

---

License

This project is licensed under the MIT License – see the LICENSE file for details.

---

Contributing

Contributions are welcome! Please open an issue or pull request on GitHub.

---

Roadmap (Future Ideas)

• Train/test split for more realistic evaluation.
• Hyperparameter tuning.
• SHAP / LIME explanations.
• Web UI.
• Support for time‑series and NLP tasks.
• Memory and human‑in‑the‑loop with LangGraph interrupt().

---

Acknowledgements

• Built with LangGraph and LiteLLM.
• Inspired by AutoML and agentic AI workflows.

---

Made with ❤️ by Mindlord-rex