scikit-ai

Category	Tools
Development
Testing
Security
Automation
Package
Documentation
Communication

Introduction

scikit-ai provides task-oriented components for text, images, audio and video. Each component does one job, such as describing an image, summarizing a text, transcribing speech or making a portrait speak, and follows the scikit-learn estimator API. The same component runs on open-source models or on cloud APIs, chosen with one backend parameter, and fits into Pipeline, GridSearchCV and the rest of the scikit-learn ecosystem.

Installation

scikit-ai is available on PyPI. The base install includes the offline dummy backend, and every other backend is an optional extra:

pip install scikit-ai              # components and the dummy backend
pip install "scikit-ai[hf]"        # local open-source models with Hugging Face
pip install "scikit-ai[openai]"    # the OpenAI API
pip install "scikit-ai[litellm]"   # Anthropic, Gemini, Mistral, Bedrock, Ollama and others through LiteLLM
pip install "scikit-ai[replicate]" # open-source models hosted by Replicate
pip install "scikit-ai[video]"     # encoding and decoding video
pip install "scikit-ai[all]"       # everything

Development installation requires to clone the repository and then use PDM to install the project as well as the main and development dependencies:

git clone https://github.com/georgedouzas/scikit-ai.git
cd scikit-ai
pdm install

Components

Components live in modules named after their task family, like scikit-learn's estimators. Every class name starts with its input type, except generators, which are named for what they generate.

What fit learns

Every component learns from the data given to fit, without training a model unless you ask for fine-tuning.

With adaptation='finetune', classifiers, TextSummarizer, TextTranslator and ImageDescriptor train a model on all the training data on backends that support fine-tuning, and fit raises UnsupportedBackendError on the others. In a pipeline, intermediate steps are fitted through fit_transform, which ignores y, so a transcriber in front of a classifier never reads the class labels as transcripts.

Usage

Every component is a scikit-learn estimator. The examples use the offline dummy backend, so they run without models or credentials. Replace backend='dummy' with a backend from the table below to get real results.

Classify texts with a few labeled examples:

from skai.classification import TextClassifier

X = ['I loved this movie.', 'The product broke after a day.', 'Great service!', 'Never again.']
y = ['positive', 'negative', 'positive', 'negative']

classifier = TextClassifier(k_shot=2, random_state=0, backend='dummy').fit(X, y)
classifier.predict(['What a wonderful day.'])

Describe images, then classify the descriptions in one pipeline:

from PIL import Image
from sklearn.pipeline import make_pipeline
from skai.description import ImageDescriptor

images = [Image.new('RGB', (64, 64), color) for color in ('red', 'green', 'blue', 'white')]
pipeline = make_pipeline(ImageDescriptor(backend='dummy'), TextClassifier(backend='dummy'))
pipeline.fit(images, ['warm', 'cold', 'cold', 'warm']).predict(images)

Make a portrait speak a text:

from skai.generation import TalkingAvatarGenerator
from skai.generation import SpeechSynthesizer

portrait = Image.new('RGB', (64, 64), 'white')
pipeline = make_pipeline(
    SpeechSynthesizer(backend='dummy'),
    TalkingAvatarGenerator(avatar=portrait, backend='dummy'),
)
videos = pipeline.fit(None).predict(['Hello, I am a talking portrait.'])
videos[0].duration

Backends

The backend parameter takes '<provider>', '<provider>:<model>' or a backend object from skai.backends. Without it, components run an open-source model.

The default model of every component and provider:

Cloud backends read credentials from environment variables, such as OPENAI_API_KEY or REPLICATE_API_TOKEN, and never take a secret as an argument. Generated speech and talking avatars can imitate real people. Use them only with the consent of the people they depict, and within the policies of the providers.

Spec-driven development

The project's engineering constitution lives in openspec/constitution.md and binds all work. The project uses OpenSpec for spec-driven development, and openspec/config.yaml names the constitution as binding.