Skip to main content

Extremely fast time series downsampling in Rust

Project description

tsdownsample

PyPI Latest Release support-version Downloads Testing Testing

Extremely fast time series downsampling 📈 for visualization, written in Rust.

Features ✨

  • Fast: written in rust with PyO3 bindings
    • leverages optimized argminmax - which is SIMD accelerated with runtime feature detection
    • scales linearly with the number of data points
    • multithreaded with Rayon (in Rust)
      Why we do not use Python multiprocessing Citing the PyO3 docs on parallelism:
      CPython has the infamous Global Interpreter Lock, which prevents several threads from executing Python bytecode in parallel. This makes threading in Python a bad fit for CPU-bound tasks and often forces developers to accept the overhead of multiprocessing.
      In Rust - which is a compiled language - there is no GIL, so CPU-bound tasks can be parallelized (with Rayon) with little to no overhead.
  • Efficient: memory efficient
    • works on views of the data (no copies)
    • no intermediate data structures are created
  • Flexible: works on any type of data
    • supported datatypes are
      • for x: f32, f64, i16, i32, i64, u16, u32, u64, datetime64, timedelta64
      • for y: f16, f32, f64, i8, i16, i32, i64, u8, u16, u32, u64, datetime64, timedelta64, bool
    !! 🚀 f16 argminmax is 200-300x faster than numpy In contrast with all other data types above, f16 is *not* hardware supported (i.e., no instructions for f16) by most modern CPUs!!
    🐌 Programming languages facilitate support for this datatype by either (i) upcasting to f32 or (ii) using a software implementation.
    💡 As for argminmax, only comparisons are needed - and thus no arithmetic operations - creating a symmetrical ordinal mapping from f16 to i16 is sufficient. This mapping allows to use the hardware supported scalar and SIMD i16 instructions - while not producing any memory overhead 🎉
    More details are described in argminmax PR #1.
  • Easy to use: simple & flexible API

Install

pip install tsdownsample

Usage

from tsdownsample import MinMaxLTTBDownsampler
import numpy as np

# Create a time series
y = np.random.randn(10_000_000)
x = np.arange(len(y))

# Downsample to 1000 points (assuming constant sampling rate)
s_ds = MinMaxLTTBDownsampler().downsample(y, n_out=1000)

# Downsample to 1000 points using the (possible irregularly spaced) x-data
s_ds = MinMaxLTTBDownsampler().downsample(x, y, n_out=1000)

Downsampling algorithms & API

Downsampling API 📑

Each downsampling algorithm is implemented as a class that implements a downsample method.
The signature of the downsample method:

downsample([x], y, n_out, **kwargs) -> ndarray[uint64]

Arguments:

  • x is optional
  • x and y are both positional arguments
  • n_out is a mandatory keyword argument that defines the number of output values*
  • **kwargs are optional keyword arguments (see table below):
    • parallel: whether to use multi-threading (default: False)**
    • ...

Returns: a ndarray[uint64] of indices that can be used to index the original data.

*When there are gaps in the time series, fewer than n_out indices may be returned.
**parallel is not supported for LTTBDownsampler.

Downsampling algorithms 📈

The following downsampling algorithms (classes) are implemented:

Downsampler Description **kwargs
MinMaxDownsampler selects the min and max value in each bin parallel
M4Downsampler selects the min, max, first and last value in each bin parallel
LTTBDownsampler performs the Largest Triangle Three Buckets algorithm
MinMaxLTTBDownsampler (new two-step algorithm 🎉) first selects n_out * minmax_ratio min and max values, then further reduces these to n_out values using the Largest Triangle Three Buckets algorithm parallel, minmax_ratio*

*Default value for minmax_ratio is 30, which is empirically proven to be a good default. (More details in our upcomming paper)

Limitations & assumptions 🚨

Assumes;

  1. x-data is (non-strictly) monotonic increasing (i.e., sorted)
  2. no NaNs in the data

👤 Jeroen Van Der Donckt

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tsdownsample-0.1.1.tar.gz (37.5 kB view hashes)

Uploaded Source

Built Distributions

tsdownsample-0.1.1-cp311-none-win_amd64.whl (1.0 MB view hashes)

Uploaded CPython 3.11 Windows x86-64

tsdownsample-0.1.1-cp311-none-win32.whl (781.7 kB view hashes)

Uploaded CPython 3.11 Windows x86

tsdownsample-0.1.1-cp311-cp311-musllinux_1_1_x86_64.whl (2.4 MB view hashes)

Uploaded CPython 3.11 musllinux: musl 1.1+ x86-64

tsdownsample-0.1.1-cp311-cp311-musllinux_1_1_aarch64.whl (2.3 MB view hashes)

Uploaded CPython 3.11 musllinux: musl 1.1+ ARM64

tsdownsample-0.1.1-cp311-cp311-manylinux_2_24_s390x.whl (2.7 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.24+ s390x

tsdownsample-0.1.1-cp311-cp311-manylinux_2_24_ppc64le.whl (2.2 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.24+ ppc64le

tsdownsample-0.1.1-cp311-cp311-manylinux_2_24_armv7l.whl (2.2 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.24+ ARMv7l

tsdownsample-0.1.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

tsdownsample-0.1.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.1 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

tsdownsample-0.1.1-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.whl (2.2 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.12+ i686

tsdownsample-0.1.1-cp311-cp311-macosx_11_0_arm64.whl (1.2 MB view hashes)

Uploaded CPython 3.11 macOS 11.0+ ARM64

tsdownsample-0.1.1-cp311-cp311-macosx_10_7_x86_64.whl (1.3 MB view hashes)

Uploaded CPython 3.11 macOS 10.7+ x86-64

tsdownsample-0.1.1-cp310-none-win_amd64.whl (1.0 MB view hashes)

Uploaded CPython 3.10 Windows x86-64

tsdownsample-0.1.1-cp310-none-win32.whl (781.7 kB view hashes)

Uploaded CPython 3.10 Windows x86

tsdownsample-0.1.1-cp310-cp310-musllinux_1_1_x86_64.whl (2.4 MB view hashes)

Uploaded CPython 3.10 musllinux: musl 1.1+ x86-64

tsdownsample-0.1.1-cp310-cp310-musllinux_1_1_aarch64.whl (2.3 MB view hashes)

Uploaded CPython 3.10 musllinux: musl 1.1+ ARM64

tsdownsample-0.1.1-cp310-cp310-manylinux_2_24_s390x.whl (2.7 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.24+ s390x

tsdownsample-0.1.1-cp310-cp310-manylinux_2_24_ppc64le.whl (2.2 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.24+ ppc64le

tsdownsample-0.1.1-cp310-cp310-manylinux_2_24_armv7l.whl (2.2 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.24+ ARMv7l

tsdownsample-0.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

tsdownsample-0.1.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.1 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

tsdownsample-0.1.1-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl (2.2 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.12+ i686

tsdownsample-0.1.1-cp310-cp310-macosx_11_0_arm64.whl (1.2 MB view hashes)

Uploaded CPython 3.10 macOS 11.0+ ARM64

tsdownsample-0.1.1-cp310-cp310-macosx_10_7_x86_64.whl (1.3 MB view hashes)

Uploaded CPython 3.10 macOS 10.7+ x86-64

tsdownsample-0.1.1-cp39-none-win_amd64.whl (1.0 MB view hashes)

Uploaded CPython 3.9 Windows x86-64

tsdownsample-0.1.1-cp39-none-win32.whl (782.1 kB view hashes)

Uploaded CPython 3.9 Windows x86

tsdownsample-0.1.1-cp39-cp39-musllinux_1_1_x86_64.whl (2.4 MB view hashes)

Uploaded CPython 3.9 musllinux: musl 1.1+ x86-64

tsdownsample-0.1.1-cp39-cp39-musllinux_1_1_aarch64.whl (2.3 MB view hashes)

Uploaded CPython 3.9 musllinux: musl 1.1+ ARM64

tsdownsample-0.1.1-cp39-cp39-manylinux_2_24_s390x.whl (2.7 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.24+ s390x

tsdownsample-0.1.1-cp39-cp39-manylinux_2_24_ppc64le.whl (2.2 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.24+ ppc64le

tsdownsample-0.1.1-cp39-cp39-manylinux_2_24_armv7l.whl (2.2 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.24+ ARMv7l

tsdownsample-0.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

tsdownsample-0.1.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.1 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

tsdownsample-0.1.1-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl (2.2 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.12+ i686

tsdownsample-0.1.1-cp39-cp39-macosx_11_0_arm64.whl (1.2 MB view hashes)

Uploaded CPython 3.9 macOS 11.0+ ARM64

tsdownsample-0.1.1-cp39-cp39-macosx_10_7_x86_64.whl (1.3 MB view hashes)

Uploaded CPython 3.9 macOS 10.7+ x86-64

tsdownsample-0.1.1-cp38-none-win_amd64.whl (1.0 MB view hashes)

Uploaded CPython 3.8 Windows x86-64

tsdownsample-0.1.1-cp38-none-win32.whl (781.6 kB view hashes)

Uploaded CPython 3.8 Windows x86

tsdownsample-0.1.1-cp38-cp38-musllinux_1_1_x86_64.whl (2.4 MB view hashes)

Uploaded CPython 3.8 musllinux: musl 1.1+ x86-64

tsdownsample-0.1.1-cp38-cp38-musllinux_1_1_aarch64.whl (2.3 MB view hashes)

Uploaded CPython 3.8 musllinux: musl 1.1+ ARM64

tsdownsample-0.1.1-cp38-cp38-manylinux_2_24_s390x.whl (2.7 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.24+ s390x

tsdownsample-0.1.1-cp38-cp38-manylinux_2_24_ppc64le.whl (2.2 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.24+ ppc64le

tsdownsample-0.1.1-cp38-cp38-manylinux_2_24_armv7l.whl (2.2 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.24+ ARMv7l

tsdownsample-0.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

tsdownsample-0.1.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.1 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

tsdownsample-0.1.1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl (2.2 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.12+ i686

tsdownsample-0.1.1-cp38-cp38-macosx_11_0_arm64.whl (1.2 MB view hashes)

Uploaded CPython 3.8 macOS 11.0+ ARM64

tsdownsample-0.1.1-cp38-cp38-macosx_10_7_x86_64.whl (1.3 MB view hashes)

Uploaded CPython 3.8 macOS 10.7+ x86-64

tsdownsample-0.1.1-cp37-none-win_amd64.whl (1.0 MB view hashes)

Uploaded CPython 3.7 Windows x86-64

tsdownsample-0.1.1-cp37-none-win32.whl (781.6 kB view hashes)

Uploaded CPython 3.7 Windows x86

tsdownsample-0.1.1-cp37-cp37m-musllinux_1_1_x86_64.whl (2.4 MB view hashes)

Uploaded CPython 3.7m musllinux: musl 1.1+ x86-64

tsdownsample-0.1.1-cp37-cp37m-musllinux_1_1_aarch64.whl (2.3 MB view hashes)

Uploaded CPython 3.7m musllinux: musl 1.1+ ARM64

tsdownsample-0.1.1-cp37-cp37m-manylinux_2_24_s390x.whl (2.7 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.24+ s390x

tsdownsample-0.1.1-cp37-cp37m-manylinux_2_24_ppc64le.whl (2.2 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.24+ ppc64le

tsdownsample-0.1.1-cp37-cp37m-manylinux_2_24_armv7l.whl (2.2 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.24+ ARMv7l

tsdownsample-0.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

tsdownsample-0.1.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.1 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

tsdownsample-0.1.1-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl (2.2 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.12+ i686

tsdownsample-0.1.1-cp37-cp37m-macosx_11_0_arm64.whl (1.2 MB view hashes)

Uploaded CPython 3.7m macOS 11.0+ ARM64

tsdownsample-0.1.1-cp37-cp37m-macosx_10_7_x86_64.whl (1.3 MB view hashes)

Uploaded CPython 3.7m macOS 10.7+ x86-64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page