scikit-learn instrumentation tooling
Project description
Generalized instrumentation tooling for scikit-learn models. sklearn_instrumentation allows instrumenting the sklearn package and any scikit-learn compatible packages with estimators and transformers inheriting from sklearn.base.BaseEstimator.
Instrumentation applies decorators to methods of BaseEstimator-derived classes or instances. By default the instrumentor applies instrumentation to the following methods (except when they are properties):
fit
predict
predict_proba
transform
_fit
_predict
_predict_proba
_transform
sklearn-instrumentation supports instrumentation of full sklearn-compatible packages, as well as recursive instrumentation of models (metaestimators like Pipeline, or even single estimators like RandomForestClassifier)
Package instrumentation
Instrument any sklearn compatible package that has BaseEstimator-derived classes.
from sklearn_instrumentation import SklearnInstrumentor
instrumentor = SklearnInstrumentor(decorator=my_instrumentation)
instrumentor.instrument_packages(["sklearn", "xgboost", "lightgbm"])
Full example:
import logging
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import FeatureUnion
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn_instrumentation import SklearnInstrumentor
from sklearn_instrumentation.instrumentation.logging import time_elapsed_logger
logging.basicConfig(level=logging.INFO)
# Create an instrumentor and instrument sklearn
instrumentor = SklearnInstrumentor(decorator=time_elapsed_logger)
instrumentor.instrument_packages(["sklearn"])
# Create a toy model for classification
ss = StandardScaler()
pca = PCA(n_components=3)
rf = RandomForestClassifier()
classification_model = Pipeline(
steps=[
(
"fu",
FeatureUnion(
transformer_list=[
("ss", ss),
("pca", pca),
]
),
),
("rf", rf),
]
)
X, y = load_iris(return_X_y=True)
# Observe logging
classification_model.fit(X, y)
# INFO:root:Pipeline.fit starting.
# INFO:root:Pipeline._fit starting.
# INFO:root:StandardScaler.fit starting.
# INFO:root:StandardScaler.fit elapsed time: 0.0006406307220458984 seconds
# INFO:root:StandardScaler.transform starting.
# INFO:root:StandardScaler.transform elapsed time: 0.0001430511474609375 seconds
# INFO:root:PCA._fit starting.
# INFO:root:PCA._fit elapsed time: 0.0006711483001708984 seconds
# INFO:root:Pipeline._fit elapsed time: 0.0026731491088867188 seconds
# INFO:root:BaseForest.fit starting.
# INFO:root:BaseForest.fit elapsed time: 0.1768970489501953 seconds
# INFO:root:Pipeline.fit elapsed time: 0.17983102798461914 seconds
# Observe logging
classification_model.predict(X)
# INFO:root:Pipeline.predict starting.
# INFO:root:FeatureUnion.transform starting.
# INFO:root:StandardScaler.transform starting.
# INFO:root:StandardScaler.transform elapsed time: 0.00024509429931640625 seconds
# INFO:root:_BasePCA.transform starting.
# INFO:root:_BasePCA.transform elapsed time: 0.0002181529998779297 seconds
# INFO:root:FeatureUnion.transform elapsed time: 0.0012080669403076172 seconds
# INFO:root:ForestClassifier.predict starting.
# INFO:root:ForestClassifier.predict_proba starting.
# INFO:root:ForestClassifier.predict_proba elapsed time: 0.013531208038330078 seconds
# INFO:root:ForestClassifier.predict elapsed time: 0.013692140579223633 seconds
# INFO:root:Pipeline.predict elapsed time: 0.015219926834106445 seconds
# Remove instrumentation
instrumentor.uninstrument_packages(["sklearn"])
# Observe no logging
classification_model.predict(X)
Machine learning model instrumentation
Instrument any sklearn compatible trained estimator or metaestimator.
from sklearn_instrumentation import SklearnInstrumentor
instrumentor = SklearnInstrumentor(decorator=my_instrumentation)
instrumentor.instrument_estimator(estimator=my_ml_pipeline)
import logging
from sklearn.datasets import load_iris
from sklearn_instrumentation import SklearnInstrumentor
from sklearn_instrumentation.instrumentation.logging import time_elapsed_logger
from sklearn.ensemble import RandomForestClassifier
logging.basicConfig(level=logging.INFO)
# Train a classifier
X, y = load_iris(return_X_y=True)
rf = RandomForestClassifier()
rf.fit(X, y)
# Create an instrumentor which decorates BaseEstimator methods with
# logging output when entering and exiting methods, with time elapsed logged
# on exit.
instrumentor = SklearnInstrumentor(decorator=time_elapsed_logger)
# Apply the decorator to all BaseEstimators in each of these libraries
instrumentor.instrument_estimator(rf)
# Observe the logging output
rf.predict(X)
# INFO:root:ForestClassifier.predict starting.
# INFO:root:ForestClassifier.predict_proba starting.
# INFO:root:ForestClassifier.predict_proba elapsed time: 0.014165163040161133 seconds
# INFO:root:ForestClassifier.predict elapsed time: 0.014327764511108398 seconds
# Remove the decorator from all BaseEstimators in each of these libraries
instrumentor.uninstrument_estimator(rf)
# No more logging
rf.predict(X)
Instrumentation
The package comes with a handful of decorators which log information about X or timing of execution. You can create your own instrumentation just by creating a decorator, following this pattern
from functools import wraps
def my_instrumentation(func, **dkwargs):
"""Wrap an estimator method with instrumentation.
:param func: The method to be instrumented.
:param dkwargs: Decorator kwargs, which can be passed to the
decorator at decoration time. For estimator instrumentation
this allows different parametrizations for each ml model.
"""
@wraps(func)
def wrapper(*args, **kwargs):
"""Wrapping function.
:param args: The args passed to methods, typically
just ``X`` and/or ``y``
:param kwargs: The kwargs passed to methods, usually
weights or other params
"""
# Code goes here before execution of the estimator method
retval = func(*args, **kwargs)
# Code goes here after execution of the estimator method
return retval
return wrapper
To pass kwargs for different ml models:
instrumentor = SklearnInstrumentor(decorator=my_instrumentation)
instrumentor.instrument_estimator(estimator=ml_model_1, decorator_kwargs={"name": "awesome_model"})
instrumentor.instrument_estimator(estimator=ml_model_2, decorator_kwargs={"name": "better_model"})
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for sklearn-instrumentation-0.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | a5dc78a2603a670f216bbe613380779b4daf8f18ea1e6524ce94c8bb8cdd6107 |
|
MD5 | 2868916e7ec3d60592dea5c26029d2d8 |
|
BLAKE2b-256 | a69a5bd187b1eb27c5891ee7df4fea508bd0eef3b3c0859a1bab2ec1f34eff6f |
Hashes for sklearn_instrumentation-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 26609233509a4baba1ae1d12c2ea15532379fc40b1e0fafc7398d171c57d87df |
|
MD5 | b7da4b8d673287e7814604f19a24dcfd |
|
BLAKE2b-256 | 1620ec4719c53764e3a8ecee4589ed79b3c8c8865140fe5711f96f46fc0490d0 |