Importance Sampling
This python package provides a library that accelerates the training of arbitrary neural networks created with Keras using importance sampling.
# Keras imports
from importance_sampling.training import ImportanceTraining
x_train, y_train, x_val, y_val = load_data()
model = create_keras_model()
model.compile(
optimizer="adam",
loss="categorical_crossentropy",
metrics=["accuracy"]
)
ImportanceTraining(model).fit(
x_train, y_train,
batch_size=32,
epochs=10,
verbose=1,
validation_data=(x_val, y_val)
)
model.evaluate(x_val, y_val)Importance sampling for Deep Learning is an active research field and this library is undergoing development so your mileage may vary.
Relevant Research
Ours
Not All Samples Are Created Equal: Deep Learning with Importance Sampling [preprint]
Biased Importance Sampling for Deep Neural Network Training [preprint]
By others
Dependencies & Installation
Normally if you already have a functional Keras installation you just need to pip install keras-importance-sampling.
Keras > 2
A Keras backend among Tensorflow, Theano and CNTK
blinker
numpy
matplotlib, seaborn, scikit-learn are optional (used by the plot scripts)
Documentation
The module has a dedicated documentation site but you can also read the source code and the examples to get an idea of how the library should be used and extended.
Examples
In the examples folder you can find some Keras examples that have been edited to use importance sampling.
Code examples
In this section we will showcase part of the API that can be used to train neural networks with importance sampling.
# Import what is needed to build the Keras model
from keras import backend as K
from keras.layers import Dense, Activation, Flatten
from keras.models import Sequential
# Import a toy dataset and the importance training
from importance_sampling.datasets import MNIST
from importance_sampling.training import ImportanceTraining
def create_nn():
"""Build a simple fully connected NN"""
model = Sequential([
Flatten(input_shape=(28, 28, 1)),
Dense(40, activation="tanh"),
Dense(40, activation="tanh"),
Dense(10),
Activation("softmax") # Needs to be separate to automatically
# get the preactivation outputs
])
model.compile(
optimizer="adam",
loss="categorical_crossentropy",
metrics=["accuracy"]
)
return model
if __name__ == "__main__":
# Load the data
dataset = MNIST()
x_train, y_train = dataset.train_data[:]
x_test, y_test = dataset.test_data[:]
# Create the NN and keep the initial weights
model = create_nn()
weights = model.get_weights()
# Train with uniform sampling
K.set_value(model.optimizer.lr, 0.01)
model.fit(
x_train, y_train,
batch_size=64, epochs=10,
validation_data=(x_test, y_test)
)
# Train with importance sampling
model.set_weights(weights)
K.set_value(model.optimizer.lr, 0.01)
ImportanceTraining(model).fit(
x_train, y_train,
batch_size=64, epochs=2,
validation_data=(x_test, y_test)
)Using the script
The following terminal commands train a small VGG-like network to ~0.65% error on MNIST (the numbers are from a CPU). .. code:
$ # Train a small cnn with mnist for 500 mini-batches using importance $ # sampling with bias to achieve ~ 0.65% error (on the CPU). $ time ./importance_sampling.py \ > small_cnn \ > oracle-gnorm \ > model \ > predicted \ > mnist \ > /tmp/is \ > --hyperparams 'batch_size=i128;lr=f0.003;lr_reductions=I10000' \ > --train_for 500 --validate_every 500 real 1m41.985s user 8m14.400s sys 0m35.900s $ $ # And with uniform sampling to achieve ~ 0.9% error. $ time ./importance_sampling.py \ > small_cnn \ > oracle-loss \ > uniform \ > unweighted \ > mnist \ > /tmp/uniform \ > --hyperparams 'batch_size=i128;lr=f0.003;lr_reductions=I10000' \ > --train_for 3000 --validate_every 3000 real 9m23.971s user 47m32.600s sys 3m4.188s
1 maintainer
Metadata
1 maintainer
Release files for keras-importance-sampling 0.10
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| keras-importance-sampling-0.10.tar.gz | 40.9 kB | Without provenance | Details |
Release files / keras-importance-sampling-0.10.tar.gz
| Download URL | keras-importance-sampling-0.10.tar.gz |
|---|---|
| Size | 40.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
48db95d95412b0ec7b014a540cf5eb45f777d0203d2de0f993c882a0f740e035
|
|
BLAKE2b-256 checksum How to use checksums |
cc4ce0ca72fe9baf5c7d8f87f5683cce8c5eb05aeeebc0092423bc06cfea09b8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3.post20200330 requests-toolbelt/0.9.1 tqdm/4.44.1 CPython/3.7.3
|
Security for keras-importance-sampling 0.10
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
