scikit-ai

Category	Tools
Development
Testing
Security
Automation
Package
Documentation
Communication

Introduction

scikit-ai provides components for text, images, audio, video and multimodal samples that follow the scikit-learn estimator API. Components are named by their input and output, such as ImageClassifier, AudioSegmenter or TextToVideoGenerator, so the name says what every sample is and what every output is. The same component runs on open-source models or on cloud APIs, chosen with one backend parameter, and fits into Pipeline, GridSearchCV and the rest of the scikit-learn ecosystem. Presets and recipes configure the components and pipelines of common tasks, from a summary to a video dubbed in the original voice.

Installation

scikit-ai is available on PyPI. The base install includes the offline dummy backend, and every other backend is an optional extra:

pip install scikit-ai              # components and the dummy backend
pip install "scikit-ai[hf]"        # local open-source models with Hugging Face
pip install "scikit-ai[openai]"    # the OpenAI API
pip install "scikit-ai[litellm]"   # Anthropic, Gemini, Mistral, Bedrock, Ollama and others through LiteLLM
pip install "scikit-ai[replicate]" # open-source models hosted by Replicate
pip install "scikit-ai[diarize]"   # speaker diarization with pyannote
pip install "scikit-ai[video]"     # encoding, decoding and assembling video
pip install "scikit-ai[all]"       # everything

Development installation requires to clone the repository and then use PDM to install the project as well as the main and development dependencies:

git clone https://github.com/georgedouzas/scikit-ai.git
cd scikit-ai
pdm install

Components

Components live in modules named after their role, like scikit-learn's estimators. Analyzers are named {Input}{Role} and generators {Input}To{Output}Generator, where the input is Text, Image, Audio, Video or Multimodal, a dict of named parts.

operation selects the procedure of a component, such as 'transcribe' or 'upscale', and instructions holds the prompt of the components that follow one, such as 'Summarize the text.'.

What fit learns

Every component learns from the data given to fit, without training a model unless you ask for fine-tuning.

With adaptation='finetune', classifiers and text generators with operation='instruct' train a model on all the training data on backends that support fine-tuning, and fit raises UnsupportedBackendError on the others. In a pipeline, intermediate steps are fitted through fit_transform, which ignores y, so a transcriber in front of a classifier never reads the class labels as transcripts.

Usage

Every component is a scikit-learn estimator. The examples use the offline dummy backend, so they run without models or credentials. Replace backend='dummy' with a backend from the table below to get real results.

Classify texts with a few labeled examples:

from skai.classification import TextClassifier

X = ['I loved this movie.', 'The product broke after a day.', 'Great service!', 'Never again.']
y = ['positive', 'negative', 'positive', 'negative']

classifier = TextClassifier(k_shot=2, random_state=0, backend='dummy').fit(X, y)
classifier.predict(['What a wonderful day.'])

Describe images, then classify the descriptions in one pipeline:

from PIL import Image
from sklearn.pipeline import make_pipeline
from skai.generation import ImageToTextGenerator

images = [Image.new('RGB', (64, 64), color) for color in ('red', 'green', 'blue', 'white')]
pipeline = make_pipeline(
    ImageToTextGenerator(instructions='Describe the image.', backend='dummy'),
    TextClassifier(backend='dummy'),
)
pipeline.fit(images, ['warm', 'cold', 'cold', 'warm']).predict(images)

Make a portrait speak a text:

from skai.generation import AudioToVideoGenerator, TextToAudioGenerator

portrait = Image.new('RGB', (64, 64), 'white')
pipeline = make_pipeline(
    TextToAudioGenerator(backend='dummy'),
    AudioToVideoGenerator(avatar=portrait, backend='dummy'),
)
videos = pipeline.fit(None).predict(['Hello, I am a talking portrait.'])
videos[0].duration

Turn a brief into a narrated, subtitled video with a recipe:

from skai import recipes

explainer = recipes.explainer_video(size=(64, 48), scene_seconds=1, backend='dummy')
video = explainer.fit(None).predict(['How solar panels work'])[0]
video.duration, video.audio is not None

Backends

The backend parameter takes '<provider>', '<provider>:<model>' or a backend object from skai.backends. Without it, components run an open-source model, locally with hf for most operations.

Local operations, such as joining clips, mixing tracks, muxing, subtitles, collages, frames, shot cuts, silences and text splitting, run without a backend. The backends guide lists the providers and the default model of every class and operation.

Cloud backends read credentials from environment variables, such as OPENAI_API_KEY or REPLICATE_API_TOKEN, and never take a secret as an argument. Generated speech, converted voices, lip sync and talking heads can imitate real people. Use them only with the consent of the people they depict, and within the policies of the providers.

Spec-driven development

The project's engineering constitution lives in openspec/constitution.md and binds all work. The project uses OpenSpec for spec-driven development, and openspec/config.yaml names the constitution as binding.