Skip to main content

A sparse matrix implementation of Whittaker-Eilers smoothing and interpolation

Project description

Whittaker-Eilers Smoothing and Interpolation

The Whittaker-Eilers smoother is the perfect smoother. It offers extremely quick, efficient smoothing with built-in interpolation via weights on each measurement. This package provides a sparse-matrix implementation for additional speed and memory efficiency and can handle both equally and unequally spaced measurements. This package was originally written in Rust so additional examples, tests, and benchmarks are also available in addition to it being super speedy. The API is almost identical.


pip install whittaker-eilers

Usage

To start smoothing and interpolating data, create a reusable WhittakerSmoother class. You'll only need to recreate this class if the length or sampling rate of your data changes.

Equally spaced data

This is the fastest smoothing option. It smooths equally spaced y measurements using two tunable parameters, lambda (2e4) and the smoother order (2). The larger the lambda, the smoother the data.

from whittaker_eilers import WhittakerSmoother

data_to_smooth = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]

whittaker_smoother = WhittakerSmoother(lmbda=2e4, order=2, data_length = len(data_to_smooth))

smoothed_data = whittaker_smoother.smooth(data_to_smooth)

print("Smoothed data: {}".format(smoothed_data))

Non-equally spaced data

If you wish to smooth unequally spaced data, you need to provide an x_input with the sample times/positions.

from whittaker_eilers import WhittakerSmoother

x_input = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]
data_to_smooth = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]

whittaker_smoother = WhittakerSmoother(
    lmbda=2e4, order=2, data_length=len(data_to_smooth), x_input=x_input
)

smoothed_data = whittaker_smoother.smooth(data_to_smooth)

print("Smoothed non-equally spaced data: {}".format(smoothed_data))

Weighted data & Interpolation

Each measurement can then be weighted to trust some measurements more than others. Setting weights to 0 for measurements will lead to interpolation.

from whittaker_eilers import WhittakerSmoother

x_input = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]
data_to_smooth = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]
weights = [1.0] * len(x_input)
weights[5] = 0.0

whittaker_smoother = WhittakerSmoother(
    lmbda=2e4,
    order=2,
    data_length=len(data_to_smooth),
    x_input=x_input,
    weights=weights,
)

smoothed_data = whittaker_smoother.smooth(data_to_smooth)

print("Smoothed and interpolated weighted data: {}".format(smoothed_data))

Smoothing with cross validation

With this package, you can also calculate the cross validation error alongside the smoothed series. This shouldn't really be used in production where speed is necessary though!

from whittaker_eilers import WhittakerSmoother

x_input = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]
data_to_smooth = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]

whittaker_smoother = WhittakerSmoother(
    lmbda=2e4, order=2, data_length=len(data_to_smooth), x_input=x_input
)

smoothed_data_with_cross_validation = whittaker_smoother.smooth_and_cross_validate(
    data_to_smooth
)

print(
    "Error :{}".format(smoothed_data_with_cross_validation.get_cross_validation_error())
)
print("Smoothed :{}".format(smoothed_data_with_cross_validation.get_smoothed()))

Automatic smoothing

Smoothing data requires a choice of Lambda. This can be done using visual inspection or by finding the lambda which results in the lowest cross validation error. The smooth_optimal function runs the smoother for a variety of lambdas and returns the results with the ability to retrieve the optimal one.

from whittaker_eilers import WhittakerSmoother

x_input = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]
data_to_smooth = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]

whittaker_smoother = WhittakerSmoother(
    lmbda=2e4, order=2, data_length=len(data_to_smooth), x_input=x_input
)

optimal_smooth = whittaker_smoother.smooth_optimal(data_to_smooth)

print("Optimal lambda: {}".format(optimal_smooth.get_optimal().get_lambda()))

print("Optimally smoothed data: {}".format(optimal_smooth.get_optimal().get_smoothed()))

You can use these methods in combination with each other for instance, interpolating measurements without providing an x input. For more advanced examples of usage take a look at the examples, tests, and benches in the Github repository. Here's an image of some smoothed data from an example:

Time-series smoothed by Whittaker-Eilers method

Other methods

from whittaker_eilers import WhittakerSmoother
x_input = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]
data_to_smooth = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]
weights = [1.0] * len(x_input)
weights[5] = 0.0

whittaker_smoother = WhittakerSmoother(
    lmbda=2e4,
    order=2,
    data_length=len(data_to_smooth),
    x_input=x_input,
    weights=weights,
)

whittaker_smoother.get_order()
whittaker_smoother.get_lambda()
whittaker_smoother.get_data_length()
whittaker_smoother.update_weights([0.5] * len(x_input))
whittaker_smoother.update_order(3)
whittaker_smoother.update_lambda(4321.0)

Further Reading

If you'd like to see a more detailed run through of the library, check out this Medium post. Within it, I run through examples and benchmarks against other smoothing methods.

Future Features

  • Scatter plot smoothing
  • Generic typing

References

The algorithm implemented here mirrors a 2003 implementation by Paul H. C. Eilers in Matlab. I've included scripts and data from the original paper in the tests for this package. The original paper and code can be found here:

A Perfect Smoother Paul H. C. Eilers Analytical Chemistry 2003 75 (14), 3631-3636 DOI: 10.1021/ac034173t

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

whittaker_eilers-0.1.3.tar.gz (195.0 kB view hashes)

Uploaded Source

Built Distributions

whittaker_eilers-0.1.3-cp37-abi3-win_amd64.whl (274.3 kB view hashes)

Uploaded CPython 3.7+ Windows x86-64

whittaker_eilers-0.1.3-cp37-abi3-win32.whl (260.6 kB view hashes)

Uploaded CPython 3.7+ Windows x86

whittaker_eilers-0.1.3-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ x86-64

whittaker_eilers-0.1.3-cp37-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.4 MB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ s390x

whittaker_eilers-0.1.3-cp37-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.3 MB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ ppc64le

whittaker_eilers-0.1.3-cp37-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.2 MB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ ARMv7l

whittaker_eilers-0.1.3-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ ARM64

whittaker_eilers-0.1.3-cp37-abi3-manylinux_2_12_i686.manylinux2010_i686.whl (1.2 MB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.12+ i686

whittaker_eilers-0.1.3-cp37-abi3-macosx_11_0_arm64.whl (383.8 kB view hashes)

Uploaded CPython 3.7+ macOS 11.0+ ARM64

whittaker_eilers-0.1.3-cp37-abi3-macosx_10_12_x86_64.whl (404.7 kB view hashes)

Uploaded CPython 3.7+ macOS 10.12+ x86-64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page