bitskitten

A small toolbox of things I kept re-typing in every EDA notebook: quick dataset reports, a few plots, and sklearn-compatible encoders that can be dropped straight into a Pipeline / GridSearchCV.

Nothing here is a framework. Each helper is one function or one class you can read in a minute.

Install

pip install bitskitten

The latest commit: pip install git+https://github.com/Caerno/bitskit.git. On Kaggle / Colab the same lines work in a cell (!pip install ...). For a kernel with no internet, add the archive GitHub builds for every commit as a dataset and install it from disk:

pip install --no-deps /kaggle/input/<your-dataset>/bitskit-main.zip

Requires Python ≥ 3.9, numpy, pandas, scikit-learn, matplotlib, seaborn.

Quick start

import pandas as pd
from bitskitten import informer, informer_print, imperfection, text_imperfection, NanFixer, Digitalize

df = pd.concat([train, test])          # test rows have no target -> NaN

informer_print(informer(df, "Survived"))
#   Features:       12
#   Observations:   891/418
#   train dataset:  68.0%
#   classes:        0: 61.6%, 1: 38.4%

imperfection(df)                       # what is missing and how varied it is

imperfection returns a frame sorted by NA share — share of missing values, number of unique values, dtype and the values themselves. By default it shows non-numeric columns only; numeric=True adds the rest, no_nan=True keeps only columns that actually have gaps. informer switches to describe() of the target when it has more than max_classes distinct values (regression).

text_imperfection(texts) is the same look at a corpus: one row per defect — empty, too short, control characters, not NFKC-normalized, URL, HTML tag, base64-like run, repeated character or word, Cyrillic and Latin mixed inside a word, few letters, exact duplicate — with share, count and a few offending examples. Rows with zero hits are hidden (no_clean=False shows them).

Encoders

Both are TransformerMixin + BaseEstimator and pass check_estimator: statistics are learned at fit and applied at transform, so the test set never leaks into its own filling, a batch that lacks a category still comes out with the same columns, get_feature_names_out, set_output(transform="pandas"), check_is_fitted, clone and grid search all work. A frame in gives a frame out; an array in gives an array out (or a frame with set_output).

from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

pipe = Pipeline([
    ("nan", NanFixer()),               # fill the gaps
    ("dig", Digitalize()),             # make everything numeric
    ("clf", RandomForestClassifier()),
])

grid = GridSearchCV(pipe, {
    "nan__lim":    [0.1, 0.225, 0.5],
    "nan__method": ["mean", "median"],
    "dig__mode":   ["raw", "cut", "qcut"],
    "dig__ncut":   [5, 10],
})

NanFixer(lim=0.225, method="mean", filler="U", random_state=None) — picks a filling strategy per column at fit time, measures it on the fit data, applies it at transform:

Columns of any other dtype pass through untouched. The plan is in plan_ after fit.

Digitalize(alim=10, mode="raw", ncut=5) — turns a mixed frame into a numeric one:

Output column names are get_feature_names_out(); up to 0.2 they were pd.get_dummies prefixes (col[:5]_value) and depended on the batch.

FeatureTrans(series) is the shared kit both encoders are built from, and it is iterable: it yields every transformation that makes sense for that column's dtype — handy when you want to throw candidate features at a selector.

candidates = pd.concat(list(FeatureTrans(df["Fare"])), axis=1)
# Fare_count, Fare_freqn, Fare_top5..., Fare_cut5..., Fare_qcut5..., Fare_sqrt, Fare_log2, Fare_pow2 ...

Its pieces are static methods you can also call directly: count, freq, top, cut, qcut, math, bilabel, fuzzy_mean, robust, fill.

Metric

from bitskitten import harmonic_score, harmonic_scorer

harmonic_score(y_true, y_pred)                      # 1.0 perfect, 0.0 if any class is lost
cross_val_score(clf, X, y, scoring=harmonic_scorer)
GridSearchCV(pipe, grid, scoring=harmonic_scorer)

harmonic_score is the harmonic mean of every per-class precision and every per-class recall — a class-agnostic generalization of F1 for when no class is "the positive one". A class that is never predicted or never recovered pulls it to 0. harmonic_from_confusion(conf) returns (score, precision[], recall[]) from a ready confusion matrix.

Plots

from bitskitten import plot_conf_map, plot_some_scatters

hm, precision, recall = plot_conf_map(confusion_matrix(y_true, y_pred), title="RF")

plot_conf_map draws the confusion matrix as shares, with a precision row, a recall column and — in the corner — harmonic_score. Returns (harmonic_mean, precision, recall); blind=True computes without drawing.

plot_some_scatters(X, y) — pairwise scatter plots of every column combination, coloured by y.

multicolumn(series, cols=5) folds a long series into several columns so a wide df.sample().T fits on one screen.

Image compression demo

img_squeeze reduces the dimensionality of an RGBA image by unrolling channels into a flat 2D matrix, running any sklearn decomposition on it and rebuilding the picture:

from sklearn.decomposition import PCA
from bitskitten import img_framaker

img_framaker(img, 20, PCA)   # image restored from 20 components

Estimator from scratch

SlowPolyLinearReg(gener=[0, 1], norm=None, alpha=None) — linear regression written on numpy/pandas, with polynomial and non-linear feature generation (0 = intercept, integers = powers, "sqrt"/"cbrt"/"log2"/"ln"/"log10"), optional normalization ("minmax", "std", "max", "mean") and L2 regularization. Closed-form solution, so it warns you about singular and ill-conditioned matrices instead of hiding them (and falls back to the least-norm solution rather than giving up). It exists to show the mechanics, not to compete with sklearn.linear_model.

The default gener=[0, 1] is the plain intercept + x. Drop the 0 only if you really want the line forced through the origin.

model = SlowPolyLinearReg(gener=[0, 1, 2], norm="std", alpha=0.01)
model.fit(X, y)
model.score(X, y)   # R²

Input

number("How many? ", t=int, op=">=", limit=0) — keeps asking until the answer parses and satisfies the condition.

Development

pip install -e .[test]
pytest

The repo root is the package (flat layout), so the clone has to be in a folder named bitskitten for pytest to import it; CI installs the wheel first for the same reason.

The drop-in zip in download/ is a plain stdlib call, no notebook needed:

python -m zipfile -c download/bitskitten.zip __init__.py classes.py input.py metrics.py output.py text.py