Skip to main content

A sparse matrix implementation of Whittaker-Eilers smoothing and interpolation

Project description

Whittaker-Eilers Smoothing and Interpolation

The Whittaker-Eilers smoother is the perfect smoother. It offers extremely quick, efficient smoothing with built-in interpolation via weights on each measurement. This package provides a sparse-matrix implementation for additional speed and memory efficiency and can handle both equally and unequally spaced measurements. This package was originally written in Rust so additional examples, tests, and benchmarks are also available in addition to it being super speedy. The API is almost identical.


pip install whittaker-eilers

Usage

To start smoothing and interpolating data, create a reusable WhittakerSmoother class. You'll only need to recreate this class if the length or sampling rate of your data changes.

Equally spaced data

This is the fastest smoothing option. It smooths equally spaced y measurements using two tunable parameters, lambda (2e4) and the smoother order (2). The larger the lambda, the smoother the data.

from whittaker_eilers import WhittakerSmoother

data_to_smooth = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]

whittaker_smoother = WhittakerSmoother(lmbda=2e4, order=2, data_length = len(data_to_smooth))

smoothed_data = whittaker_smoother.smooth(data_to_smooth)

print("Smoothed data: {}".format(smoothed_data))

Non-equally spaced data

If you wish to smooth unequally spaced data, you need to provide an x_input with the sample times/positions.

from whittaker_eilers import WhittakerSmoother

x_input = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]
data_to_smooth = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]

whittaker_smoother = WhittakerSmoother(
    lmbda=2e4, order=2, data_length=len(data_to_smooth), x_input=x_input
)

smoothed_data = whittaker_smoother.smooth(data_to_smooth)

print("Smoothed non-equally spaced data: {}".format(smoothed_data))

Weighted data & Interpolation

Each measurement can then be weighted to trust some measurements more than others. Setting weights to 0 for measurements will lead to interpolation.

from whittaker_eilers import WhittakerSmoother

x_input = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]
data_to_smooth = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]
weights = [1.0] * len(x_input)
weights[5] = 0.0

whittaker_smoother = WhittakerSmoother(
    lmbda=2e4,
    order=2,
    data_length=len(data_to_smooth),
    x_input=x_input,
    weights=weights,
)

smoothed_data = whittaker_smoother.smooth(data_to_smooth)

print("Smoothed and interpolated weighted data: {}".format(smoothed_data))

Smoothing with cross validation

With this package, you can also calculate the cross validation error alongside the smoothed series. This shouldn't really be used in production where speed is necessary though!

from whittaker_eilers import WhittakerSmoother

x_input = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]
data_to_smooth = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]

whittaker_smoother = WhittakerSmoother(
    lmbda=2e4, order=2, data_length=len(data_to_smooth), x_input=x_input
)

smoothed_data_with_cross_validation = whittaker_smoother.smooth_and_cross_validate(
    data_to_smooth
)

print(
    "Error :{}".format(smoothed_data_with_cross_validation.get_cross_validation_error())
)
print("Smoothed :{}".format(smoothed_data_with_cross_validation.get_smoothed()))

Automatic smoothing

Smoothing data requires a choice of Lambda. This can be done using visual inspection or by finding the lambda which results in the lowest cross validation error. The smooth_optimal function runs the smoother for a variety of lambdas and returns the results with the ability to retrieve the optimal one.

from whittaker_eilers import WhittakerSmoother

x_input = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]
data_to_smooth = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]

whittaker_smoother = WhittakerSmoother(
    lmbda=2e4, order=2, data_length=len(data_to_smooth), x_input=x_input
)

optimal_smooth = whittaker_smoother.smooth_optimal(data_to_smooth)

print("Optimal lambda: {}".format(optimal_smooth.get_optimal().get_lambda()))

print("Optimally smoothed data: {}".format(optimal_smooth.get_optimal().get_smoothed()))

Parallel Smoothing

With this package you're able to smooth multiple data series in parallel. While y, the data to be smoothed can vary, x, weights, and the other smoothing parameters cannot be varied across the different data series.

y = [[1.2, 4.5, 6.5, 3.2, 1.23, 23.23, 231.23]] * 10

whittaker_smoother = WhittakerSmoother(lmbda=20, order=2, data_length=len(y[0]))

result = whittaker_smoother.smooth_parallel(y)

print("Smoothed data: {}".format(result))

You can use these methods in combination with each other for instance, interpolating measurements without providing an x input. For more advanced examples of usage take a look at the examples, tests, and benches in the Github repository. Here's an image of some smoothed data from an example:

Time-series smoothed by Whittaker-Eilers method

Other methods

from whittaker_eilers import WhittakerSmoother
x_input = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]
data_to_smooth = [1.1, 1.9, 3.1, 3.91, 5.0, 6.02, 7.01, 7.7, 9.0, 10.0]
weights = [1.0] * len(x_input)
weights[5] = 0.0

whittaker_smoother = WhittakerSmoother(
    lmbda=2e4,
    order=2,
    data_length=len(data_to_smooth),
    x_input=x_input,
    weights=weights,
)

whittaker_smoother.get_order()
whittaker_smoother.get_lambda()
whittaker_smoother.get_data_length()
whittaker_smoother.update_weights([0.5] * len(x_input))
whittaker_smoother.update_order(3)
whittaker_smoother.update_lambda(4321.0)

Further Reading

If you'd like to see a more detailed run through of the library, check out this Medium post. Within it, I run through examples and benchmarks against other smoothing methods.

Future Features

  • Scatter plot smoothing
  • Generic typing

References

The algorithm implemented here mirrors a 2003 implementation by Paul H. C. Eilers in Matlab. I've included scripts and data from the original paper in the tests for this package. The original paper and code can be found here:

A Perfect Smoother Paul H. C. Eilers Analytical Chemistry 2003 75 (14), 3631-3636 DOI: 10.1021/ac034173t

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

whittaker_eilers-0.2.0.tar.gz (196.6 kB view details)

Uploaded Source

Built Distributions

whittaker_eilers-0.2.0-cp37-abi3-win_amd64.whl (287.3 kB view details)

Uploaded CPython 3.7+Windows x86-64

whittaker_eilers-0.2.0-cp37-abi3-win32.whl (269.6 kB view details)

Uploaded CPython 3.7+Windows x86

whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (462.3 kB view details)

Uploaded CPython 3.7+manylinux: glibc 2.17+ x86-64

whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl (513.3 kB view details)

Uploaded CPython 3.7+manylinux: glibc 2.17+ s390x

whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (521.5 kB view details)

Uploaded CPython 3.7+manylinux: glibc 2.17+ ppc64le

whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_i686.manylinux2014_i686.whl (494.8 kB view details)

Uploaded CPython 3.7+manylinux: glibc 2.17+ i686

whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (445.4 kB view details)

Uploaded CPython 3.7+manylinux: glibc 2.17+ ARMv7l

whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (443.8 kB view details)

Uploaded CPython 3.7+manylinux: glibc 2.17+ ARM64

whittaker_eilers-0.2.0-cp37-abi3-macosx_11_0_arm64.whl (385.0 kB view details)

Uploaded CPython 3.7+macOS 11.0+ ARM64

whittaker_eilers-0.2.0-cp37-abi3-macosx_10_12_x86_64.whl (408.1 kB view details)

Uploaded CPython 3.7+macOS 10.12+ x86-64

File details

Details for the file whittaker_eilers-0.2.0.tar.gz.

File metadata

  • Download URL: whittaker_eilers-0.2.0.tar.gz
  • Upload date:
  • Size: 196.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: maturin/1.7.6

File hashes

Hashes for whittaker_eilers-0.2.0.tar.gz
Algorithm Hash digest
SHA256 65e5e7cd3c9ea0028fe621f3927aa2d61a69688e2df2815010ec35a7c3424879
MD5 f0046030ae9f28df4beb557425d0a2bc
BLAKE2b-256 8c93ca03431eae550202011d5fb4a5ab66545829ae12e82aefa0ff0726c18948

See more details on using hashes here.

File details

Details for the file whittaker_eilers-0.2.0-cp37-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for whittaker_eilers-0.2.0-cp37-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 b39d462d3f5919b7bf5dccc812c838f72d4c0d73a8e3ea5bdb1fc834eeebba0f
MD5 df6b38b5eef2b1b945719ff0aeed0721
BLAKE2b-256 c9643448ed48760f26e9e3fedd33b9030e74e6711cc57b0f1da73253790153f5

See more details on using hashes here.

File details

Details for the file whittaker_eilers-0.2.0-cp37-abi3-win32.whl.

File metadata

File hashes

Hashes for whittaker_eilers-0.2.0-cp37-abi3-win32.whl
Algorithm Hash digest
SHA256 e925e52196e3a0888f9b71730b6d3cbce0ebbccd6553a136680904c09230bf00
MD5 8ae44051c265e6d2f4973e1383f9f8ae
BLAKE2b-256 65726624d48f4856ee0002c5fcd15edc45b92c5bc210946f15e0f970487113d5

See more details on using hashes here.

File details

Details for the file whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 3002697548ccd84ebbe786da1cfebc1ce934c2d49ecd2669ca2351cb0a62d934
MD5 0c47efad54cb5edf2da451ffa03ec067
BLAKE2b-256 51f2fa9fa022c9aba868107874d2a29550c6ce2dbcea647f51f338195192d869

See more details on using hashes here.

File details

Details for the file whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl.

File metadata

File hashes

Hashes for whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 806bfcc6d76bce17c4a64291cc8e76a1c2bafa34103058bd4721fa3bc3c6e268
MD5 b6886ce8722ac85cd1175e39b9997466
BLAKE2b-256 be86c27f3f0b95b877ed802a630af52212d25828cc12932561843547307d9c1b

See more details on using hashes here.

File details

Details for the file whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl.

File metadata

File hashes

Hashes for whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 cc1488b9bd446cd3323c068f331b20d27e8e9383d8c06aa946b3550f75161e8f
MD5 35b42dcb08ed9544d0848c80a5430aca
BLAKE2b-256 32e6e9e853bb620cd5fad2580cb0231ac460199b3ca49cbe1a16eb8a87acd72b

See more details on using hashes here.

File details

Details for the file whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

File hashes

Hashes for whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm Hash digest
SHA256 e02c484264875311321552fee00b39c40723c46867ac6bc17f4c46accd3ba258
MD5 9fb0f5ab65d9f67a87efb3a43f5a447e
BLAKE2b-256 1a1783f4ab6967b76eaa1cbf942e799d198ff293395f39a17a6db4db53fbaa5a

See more details on using hashes here.

File details

Details for the file whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl.

File metadata

File hashes

Hashes for whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm Hash digest
SHA256 d548abdc197bce1c180fe3d33f8d3318143ab16db160eee2eb56b5b12f35fbb7
MD5 bd4e827f35cdb39a2ee9616d1e35bbe9
BLAKE2b-256 f83a5a303dbd6516634d431284ab1d7b24be13a5e5afa1b070d9c7ec08d1b424

See more details on using hashes here.

File details

Details for the file whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for whittaker_eilers-0.2.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 6822bfcbd4a1bb6fae7189f1d6e245988546e96cdc9a794b343c303a06838a6e
MD5 733c8ec367b21a2f582b221a31aa249a
BLAKE2b-256 4a524de88b97bf9b6acbf76d339d52e2ac46f65cb9f63e8ba5049f3c42ce4824

See more details on using hashes here.

File details

Details for the file whittaker_eilers-0.2.0-cp37-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for whittaker_eilers-0.2.0-cp37-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 3c64d2f215736d8c9c86c5425b1cf40809bac413584c6d7d1bb559ba7491cd9e
MD5 86af34846d226ceb3cb947174e299d8f
BLAKE2b-256 ece4bf2f476e9dfc9889dbf1102f37d9ea11ac028d56d5ae2436942c09c43056

See more details on using hashes here.

File details

Details for the file whittaker_eilers-0.2.0-cp37-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for whittaker_eilers-0.2.0-cp37-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 105e3d7c231fabc928481f5adec62397b87e4143f32793c36bbd99352c553e50
MD5 5b7e64e7d227d039c9c525da305f3954
BLAKE2b-256 2464178bd8c1524cf8125eb8d5b20caee631ef2d642f1e787148bfc8a008d931

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page