Skip to main content
Donate to the Python Software Foundation or Purchase a PyCharm License to Benefit the PSF! Donate Now

Detrending algorithms

Project description

Logo

pip Documentation Image Build Status

Wōtan...

...offers free and open source algorithms to automagically remove trends from time-series data.

In Germanic mythology, Odin (/ˈoːðinː/ Old High German: Wōtan) is a widely revered god. He gave one of his eyes to Mimir in return for wisdom. Thus, in order to achieve a goal, one sometimes has to turn a blind eye. In Richard Wagner's "Der Ring des Nibelungen", Wotan is the King of the Gods (god of light, air, and wind) and a bass-baritone. According to Wagner, he is the "pinnacle of intelligence".

Example usage

from wotan import flatten
flatten_lc, trend_lc = flatten(time, flux, window_length=0.5, method='biweight', return_trend=True)

For more details, have a look at the interactive playground, the documentation and tutorials.

Available detrending algorithms

  • Time-windowed sliders with location estimates:
  • Splines:
    • rspline Spline with iterative sigma-clipping
    • hspline Spline with a robust Huber estimator (Huber 1981)
    • pspline Penalized spline to automatically select the knot distance (Eilers 1996), with iteratively sigma-clipping
  • Polynomials and others:
  • gp Gaussian Processes offering:
    • squared_exp Squared-exponential kernel, with option for iterative sigma-clipping
    • matern Matern 3/2 kernel, with option for iterative sigma-clipping
    • periodic Periodic kernel informed by a user-specified period
    • periodic_auto Periodic kernel informed by a Lomb-Scargle periodogram pre-search

Available features

  • window_length The length of the filter window in units of time (usually days).
  • break_tolerance If there are large gaps in time, especially with corresponding flux level offsets, the detrending is much improved when splitting the data into several sub-lightcurves and applying the filter to each individually. Comes with an empirical default and is fully adjustable.
  • edge_cutoff Trends near edges are less robust. Depending on the data, it may be beneficial to remove edges.
  • cval Tuning parameter for the robust estimators (see documentation)
  • return_trend If True, the method will return a tuple of two elements (flattened_flux, trend_flux) where trend_flux is the removed trend. Otherwise, it will only return flattened_flux.

What method to choose?

It depends on your data and what you like to achieve (relevant xkcd). If possible, try it out! Use wotan with a selection of methods, iterate over their parameter space, and choose what gives the best results for your research.

If that is too much effort, you should first examine your data.

  • Is it mostly white (Gaussian) noise? Use a time-windowed sliding mean. This is the most efficient method for white noise.
  • With prominent outliers (such as transits or flares), use a robust time-windowed method such as the biweight. This is usually superior to the median or trimmed methods.
  • Are there (semi-) periodic trends? In addition to a time-windowed biweight, try a spline-based method. Experimenting with periodic GPs is worthwhile.

Installation

To install the released version, type

$ pip install wotan

which automatically installs numpy, numba and scipy if not present. Depending on the algorithm, additional dependencies exist:

  • lowess and huber depend on statsmodels
  • hspline and gp depend on sklearn
  • pspline depends on pygam
  • supersmoother depends on supersmoother

To install all additional dependencies, type $ pip install statsmodels sklearn supersmoother pygam.

Originality

As all scientific work, wōtan is standing on the shoulders of giants. Particularly, many detrending methods are wrapped from existing packages. Original contributions include:

  • A time-windowed detrending master module with edge treatments and segmentation options
  • Robust location estimates using Newton-Raphson iteration to a precision threshold for Tukey's biweight, Andrew's sine wave, and the Welsch-Leclerc. This is probably a "first", which reduces jitter in the location estimate by ~10 ppm
  • Robustified (iterative sigma-clipping) penalized splines for automatic knot distance determination and outlier resistance
  • Robustified (iterative sigma-clipping) Gaussian processes
  • GP with a periodic kernel informed by a Lomb-Scargle periodogram pre-search
  • Bringing together many methods in one place in a common interface, with sensible defaults
  • Providing documentation, tutorials, and a paper which compares and benchmarks the methods

Attribution

Please cite Hippke et al. (2019, XXX) if you find this code useful in your research. The BibTeX entry for the paper is:

@ARTICLE{XXX,
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
wotan-1.0.3-py3-none-any.whl (24.9 kB) Copy SHA256 hash SHA256 Wheel py3
wotan-1.0.3.tar.gz (22.3 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page