active-vision 0.0.4

Active learning for edge vision.

Active learning at the edge for computer vision.

The goal of this project is to create a framework for the active learning loop for computer vision deployed on edge devices.

Supported tasks:

• Image classification
• Object detection
• Segmentation

Installation

Get a release from PyPI

pip install active-vision

Install from source

git clone https://github.com/dnth/active-vision.git
cd active-vision
pip install -e .

I recommend using uv to set up a virtual environment and install the package. You can also use other virtual env of your choice.

If you're using uv:

uv venv
uv sync

Once the virtual environment is created, you can install the package using pip.

[!TIP] If you're using uv add a uv before the pip install command to install into your virtual environment. Eg:

uv pip install active-vision

Usage

See the notebook for a complete example.

Be sure to prepared 3 datasets:

• initial_samples: A dataframe of an existing labeled training dataset to seed the training set.
• unlabeled: A dataframe of unlabeled data which we will sample from using active learning.
• eval: A dataframe of labeled data which we will use to evaluate the performance of the model.

As a toy example I created the above 3 datasets from the imagenette dataset.

from active_vision import ActiveLearner
import pandas as pd

# Create an active learner instance with a model
al = ActiveLearner("resnet18")

# Load dataset 
train_df = pd.read_parquet("training_samples.parquet")
al.load_dataset(df, filepath_col="filepath", label_col="label")

# Train model
al.train(epochs=3, lr=1e-3)

# Evaluate the model on a *labeled* evaluation set
accuracy = al.evaluate(eval_df, filepath_col="filepath", label_col="label")

# Get predictions from an *unlabeled* set
pred_df = al.predict(filepaths)

# Sample low confidence predictions from unlabeled set
uncertain_df = al.sample_uncertain(pred_df, num_samples=10)

# Launch a Gradio UI to label the low confidence samples
al.label(uncertain_df, output_filename="uncertain")

Once complete, the labeled samples will be save into a new df. We can now add the newly labeled data to the training set.

# Add newly labeled data to training set and save as a new file active_labeled
al.add_to_train_set(labeled_df, output_filename="active_labeled")

Repeat the process until the model is good enough. Use the dataset to train a larger model and deploy.

[!TIP] For the toy dataset, I got to about 93% accuracy on the evaluation set with 200+ labeled images. The best performing model on the leaderboard got 95.11% accuracy training on all 9469 labeled images.

This took me about 6 iterations of relabeling. Each iteration took about 5 minutes to complete including labeling and model training (resnet18). See the notebook for more details.

But using the dataset of 200+ images, I trained a more capable model (convnext_small_in22k) and got 99.3% accuracy on the evaluation set. See the notebook for more details.

Workflow

There are two workflows for active learning at the edge that we can use depending on the availability of labeled data.

With unlabeled data

If we have no labeled data, we can use active learning to iteratively improve the model and build a labeled dataset.

1. Load a small proxy model.
2. Label an initial dataset. If there is none, you'll have to label some images.
3. Train the proxy model on the labeled dataset.
4. Run inference on the unlabeled dataset.
5. Evaluate the performance of the proxy model.
6. Is model good enough?
   • Yes: Save the proxy model and the dataset.
   • No: Select the most informative images to label using active learning.
7. Label the most informative images and add them to the dataset.
8. Repeat steps 3-6.
9. Save the proxy model and the dataset.
10. Train a larger model on the saved dataset.

graph TD
    A[Load a small proxy model] --> B[Label an initial dataset]
    B --> C[Train proxy model on labeled dataset]
    C --> D[Run inference on unlabeled dataset]
    D --> E[Evaluate proxy model performance]
    E --> F{Model good enough?}
    F -->|Yes| G[Save proxy model and dataset]
    G --> H[Train and deploy a larger model]
    F -->|No| I[Select informative images using active learning]
    I --> J[Label selected images]
    J --> C

With labeled data

If we have a labeled dataset, we can use active learning to iteratively improve the dataset and the model by fixing the most important label errors.

1. Load a small proxy model.
2. Train the proxy model on the labeled dataset.
3. Run inference on the entire labeled dataset.
4. Get the most important label errors with active learning.
5. Fix the label errors.
6. Repeat steps 2-5 until the dataset is good enough.
7. Save the labeled dataset.
8. Train a larger model on the saved labeled dataset.

graph TD
    A[Load a small proxy model] --> B[Train proxy model on labeled dataset]
    B --> C[Run inference on labeled dataset]
    C --> D[Get important label errors using active learning]
    D --> E[Fix label errors]
    E --> F{Dataset good enough?}
    F -->|No| B
    F -->|Yes| G[Save cleaned dataset]
    G --> H[Train and deploy larger model]