light-curve 0.8.0

Feature extractor from noisy time series

light-curve processing toolbox for Python

The Python wrapper for Rust light-curve-feature and light-curve-dmdt packages which gives a collection of high-performant time-series feature extractors.

Installation

python3 -mpip install 'light-curve[full]'

full extras would install the package with all optional Python dependencies required by experimental features. We also provide light-curve-python package which is just an "alias" to the main light-curve[full] package.

Minimum supported Python version is 3.7. We provide binary wheels via PyPi for number of platforms and architectures, both for CPython and PyPy. We also provide binary wheels for stable CPython ABI, so the package is guaranteed to work with all future CPython3 versions.

Support matrix

Arch \ OS	Linux glibc	Linux musl	macOS	Windows https://github.com/light-curve/light-curve-python/issues/186
x86-64	wheel (MKL)	wheel (MKL)	wheel	wheel (no Ceres, no GSL)
i686	src	src	—	not tested
aarch64	wheel	wheel	src https://github.com/light-curve/light-curve-python/issues/5	not tested
ppc64le	wheel	not tested (no Rust toolchain)	—	—

• "wheel": binary wheel is available on pypi.org, local building is not required for the platform, the only pre-requirement is a recent pip version. For Linux x86-64 we provide binary wheels built with Intel MKL for better periodogram performance, which is not a default build option. For Windows x86-64 we provide wheel with no Ceres and no GSL support, which is not a default build option.
• "src": the package is confirmed to be built and pass unit tests locally, but testing and package building is not supported by CI. It is required to have the GNU scientific library (GSL) v2.1+ and the Rust toolchain v1.62+ to install it via pip install.
• "not tested": building from the source code is not tested, please report us building status via issue/PR/email.

We build aarch64 macOS 12.0+ Python 3.8+ wheels locally and submit them running this command in light-curve directory:

rm -rf ./wheelhouse
curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain stable -y
brew install gsl
CIBW_BUILD='cp3*-macosx_arm64' CIBW_ENVIRONMENT="MATURIN_PEP517_ARGS='--locked --no-default-features --features ceres-source,fftw-source,gsl'" CIBW_BEFORE_ALL='' python3 -mcibuildwheel --platform macos
CIBW_BUILD='cp310-macosx_arm64' CIBW_ENVIRONMENT="MATURIN_PEP517_ARGS='--locked --no-default-features --features ceres-source,fftw-source,gsl,abi3'" CIBW_BEFORE_ALL='' python3 -mcibuildwheel --platform macos
twine upload wheelhouse/*.whl

Feature evaluators

Most of the classes implement various feature evaluators useful for light-curve based astrophysical source classification and characterisation.

import light_curve as lc
import numpy as np

# Time values can be non-evenly separated but must be an ascending array
n = 101
t = np.linspace(0.0, 1.0, n)
perfect_m = 1e3 * t + 1e2
err = np.sqrt(perfect_m)
m = perfect_m + np.random.normal(0, err)

# Half-amplitude of magnitude
amplitude = lc.Amplitude()
# Fraction of points beyond standard deviations from mean
beyond_std = lc.BeyondNStd(nstd=1)
# Slope, its error and reduced chi^2 of linear fit
linear_fit = lc.LinearFit()
# Feature extractor, it will evaluate all features in more efficient way
extractor = lc.Extractor(amplitude, beyond_std, linear_fit)

# Array with all 5 extracted features
result = extractor(t, m, err, sorted=True, check=False)

print('\n'.join(f"{name} = {value:.2f}" for name, value in zip(extractor.names, result)))

# Run in parallel for multiple light curves:
results = amplitude.many(
    [(t[:i], m[:i], err[:i]) for i in range(n // 2, n)],
    n_jobs=-1,
    sorted=True,
    check=False,
)
print("Amplitude of amplitude is {:.2f}".format(np.ptp(results)))

If you confident in your inputs you could use sorted = True (t is in ascending order) and check = False (no NaNs in inputs, no infs in t or m) for better performance. Note that if your inputs are not valid and are not validated by sorted=None and check=True (default values) then all kind of bad things could happen.

Print feature classes list

import light_curve as lc

print([x for x in dir(lc) if hasattr(getattr(lc, x), "names")])

Read feature docs

import light_curve as lc

help(lc.BazinFit)

Experimental extractors

From the technical point of view the package consists of two parts: a wrapper for light-curve-feature Rust crate (light_curve_ext sub-package) and pure Python sub-package light_curve_py. We use the Python implementation of feature extractors to test Rust implementation and to implement new experimental extractors. Please note, that the Python implementation is much slower for the most of the extractors and doesn't provide the same functionality as the Rust implementation. However, the Python implementation provides some new feature extractors you can find useful.

You can manually use extractors from both implementations:

import numpy as np
from numpy.testing import assert_allclose
from light_curve.light_curve_ext import LinearTrend as RustLinearTrend
from light_curve.light_curve_py import LinearTrend as PythonLinearTrend

rust_fe = RustLinearTrend()
py_fe = PythonLinearTrend()

n = 100
t = np.sort(np.random.normal(size=n))
m = 3.14 * t - 2.16 + np.random.normal(size=n)

assert_allclose(rust_fe(t, m), py_fe(t, m),
                err_msg="Python and Rust implementations must provide the same result")

This should print a warning about experimental status of the Python class

Available features

See the complite list of evailable feature evaluators and documentation in light-curve-feature Rust crate docs.

Feature name	Description	Min data points	Features number
Amplitude	Half amplitude of magnitude: $\displaystyle \frac{\max (m)-\min (m)}{2}$	1	1
AndersonDarlingNormal	Unbiased Anderson–Darling normality test statistic: $\displaystyle \left( 1+\frac{4}{N} -\frac{25}{N^{2}}\right) \times$ $\times \left( -N-\frac{1}{N}\sum\limits_{i=0}^{N-1} (2i+1)\ln \Phi _{i} +(2(N-i)-1)\ln (1-\Phi _{i} )\right) ,$ where $\Phi _{i\ } \equiv \Phi (( m_{i} \ -\ \langle m\rangle ) /\sigma _{m})$ is the commutative distribution function of the standard normal distribution, $N-$ the number of observations, $\langle m\rangle -$ mean magnitude and $\sigma _{m} =\sqrt{\sum\limits_{i=0}^{N-1}( m_{i} -\langle m\rangle )^{2} /( N-1) \ }$ is the magnitude standard deviation	4	1
BazinFit	Five fit parameters and goodness of fit (reduced $\chi ^{2}$ of the Bazin function developed for core-collapsed supernovae: $\displaystyle f(t)=A\frac{\mathrm{e}^{-(t-t_{0} )/\tau _{fall}}}{1+\mathrm{e}^{-(t-t_{0} )/\tau _{rise}}} +B,$ where $f(t)-$ flux observation	6	1
BeyondNStd	Fraction of observations beyond $n\sigma _{m}$ from the mean magnitude $\langle m\rangle $: $\displaystyle \frac{\sum _{i} I_{|m-\langle m\rangle | >n\sigma _{m}} (m_{i} )}{N},$ where $I-$ an indicator function	2	1
Cusum	A range of cumulative sums: $\displaystyle \max(S) -\min(S),$ where $S_{j} \equiv \frac{1}{N\sigma _{m}}\sum\limits _{i=0}^{j} (m_{i} -\langle m\rangle )$, $j\in \{1..N-1\}$	2	1
Eta	Von Neummann $\eta $: $\displaystyle \eta \equiv \frac{1}{(N-1)\sigma _{m}^{2}}\sum\limits _{i=0}^{N-2} (m_{i+1} -m_{i} )^{2}$	2	1
EtaE	Modernisation of Eta for unevenly time series: $\displaystyle \eta ^{e} \equiv \frac{(t_{N-1} -t_{0} )^{2}}{(N-1)^{3}}\frac{\sum\limits_{i=0}^{N-2}\left(\frac{m_{i+1} -m_{i}}{t_{i+1} -t_{i}}\right)^{2}}{\sigma _{m}^{2}}$	2	1
ExcessVariance	Measure of the variability amplitude: $\displaystyle \frac{\sigma _{m}^{2} -\langle \delta ^{2} \rangle }{\langle m\rangle ^{2}},$ where $\langle \delta ^{2} \rangle -$ mean squared error	2	1
InterPercentileRange	$\displaystyle Q(1-p)-Q(p),$ where $Q(n)$ and $Q(d)-$ $n$-th and $d$-th quantile of magnitude sample	1	1
Kurtosis	Excess kurtosis of magnitude: $\displaystyle \frac{N(N+1)}{(N-1)(N-2)(N-3)}\frac{\sum _{i} (m_{i} -\langle m\rangle )^{4}}{\sigma _{m}^{2}} -3\frac{(N+1)^{2}}{(N-2)(N-3)}$	4	1
LinearFit	The slope, its error and reduced $\chi ^{2}$ of the light curve in the linear fit of a magnitude light curve with respect to the observation error $\{\delta _{i}\}$: $\displaystyle m_{i} \ =\ c\ +\ \text{slope} \ t_{i} \ +\ \delta _{i} \varepsilon _{i} ,$ where $c$ is a constant, $\{\varepsilon _{i}\}$ are standard distributed random variables	3	3
LinearTrend	The slope and its error of the light curve in the linear fit of a magnitude light curve without respect to the observation error $\{\delta _{i}\}$: $\displaystyle m_{i} \ =\ c\ +\ \text{slope} \ t_{i} \ +\ \Sigma \varepsilon _{i} ,$ where $c$ and $\Sigma$ are constants, $\{\varepsilon _{i}\}$ are standard distributed random variables.	2	2
MagnitudePercentageRatio	Magnitude percentage ratio: $\displaystyle \frac{Q(1-n)-Q(n)}{Q(1-d)-Q(d)}$	1	1
MaximumSlope	Maximum slope between two sub-sequential observations: $\displaystyle \max_{i=0\dotsc N-2}\left| \frac{m_{i+1} -m_{i}}{t_{i+1} -t_{i}}\right|$	2	1
Mean	Mean magnitude: $\displaystyle \langle m\rangle =\frac{1}{N}\sum\limits _{i} m_{i}$	1	1
MeanVariance	Standard deviation to mean ratio: $\displaystyle \frac{\sigma _{m}}{\langle m\rangle }$	2	1
Median	Median magnitude	1	1
MedianAbsoluteDeviation	Median of the absolute value of the difference between magnitude and its median: $\displaystyle \mathrm{Median} (|m_{i} -\mathrm{Median} (m)|)$	1	1
MedianBufferRangePercentage	$\displaystyle \mathrm{Median} (m)\pm q\times (\max (m)-\min (m))/2$	1	1
OtsuSplit	Difference of subset means, standard deviation of the lower subset, standard deviation of the upper subset and lower-to-all observation count ratio for two subsets of magnitudes obtained by Otsu's method split. Otsu's method is used to perform automatic thresholding. The algorithm returns a single threshold that separate values into two classes. This threshold is determined by minimizing intra-class intensity variance $\sigma^2_{W}=w_0\sigma^2_0+w_1\sigma^2_1$, or equivalently, by maximizing inter-class variance $\sigma^2_{B}=w_0 w_1 (\mu_1-\mu_0)^2$. There can be more than one extremum. In this case, the algorithm returns the minimum threshold.	2	4
PercentAmplitude	Maximum deviation of magnitude from its median: $\displaystyle \max_{i} |m_{i} \ -\ \text{Median}( m) |$	1	1
PercentDifferenceMagnitudePercentile	Ratio of $p$-th inter-percentile range to the median: $\displaystyle \frac{Q( 1-p) -Q( p)}{\text{Median}( m)}$	1	1
ReducedChi2	Reduced $\chi ^{2}$ of magnitude measurements: $\displaystyle \frac{1}{N-1}\sum _{i}\left(\frac{m_{i} -\overline{m}}{\delta _{i}}\right)^{2} ,$ where $\overline{m} -$ weighted mean magnitude	2	1
Skew	Skewness of magnitude: $\displaystyle \frac{N}{(N-1)(N-2)}\frac{\sum _{i} (m_{i} -\langle m\rangle )^{3}}{\sigma _{m}^{3}}$	3	1
StandardDeviation	Standard deviation of magnitude: $\displaystyle \sigma _{m} \equiv \sqrt{\sum _{i} (m_{i} -\langle m\rangle )^{2} /(N-1)}$	2	1
StetsonK	Stetson K coefficient described light curve shape: $\displaystyle \frac{\sum _{i}\left| \frac{m_{i} -\langle m\rangle }{\delta _{i}}\right| }{\sqrt{N\ \chi ^{2}}}$	2	1
VillarFit	Seven fit parameters and goodness of fit (reduced $\chi ^{2}$) of the Villar function developed for supernovae classification: $f(t)=c+\frac{A}{1+\exp\frac{-(t-t_{0} )}{\tau _{rise}}} \times f_{fall}(t),$ $f_{fall}(t) = 1-\frac{\nu (t-t_{0} )}{\gamma }, ~~~ t< t_{0} +\gamma,$ $f_{fall}(t) = (1-\nu )\exp\frac{-(t-t_{0} -\gamma )}{\tau _{fall}}, ~~~ t \geq t_{0} + \gamma.$ where $f(t) -$ flux observation, $A, \gamma , \tau _{rise} , \tau _{fall} >0$, $\nu \in [0;1)$ Here we introduce a new dimensionless parameter $\nu$ instead of the plateau slope $\beta$ from the original paper: $\nu \equiv -\beta \gamma /A$	8	8
WeightedMean	Weighted mean magnitude: $\displaystyle \overline{m} \equiv \frac{\sum _{i} m_{i} /\delta _{i}^{2}}{\sum _{i} 1/\delta _{i}^{2}}$	1	1

Meta-features

Meta-features can accept another feature extractors and apply them to pre-processed data.

Periodogram

A number of features based on Lomb–Scargle periodogram as it was an evenly separeated uncertancy-less lime series. Periodogram $P(\omega)$ is an estimate of spectral density of unevenly time series. Periodogram's peaks argument corresponds to a number of the most significant spectral density peaks to return. For each peak its period and "signal to noise" ratio is returned.

$$ \text{signal to noise of peak} \equiv \frac{P(\omega_\mathrm{peak}) - \langle P(\omega) \rangle}{\sigma_{P(\omega)}} $$

features argument accepts a list of additional feature evaluators.

Bins

Binning time series to bins with width $\mathrm{window}$ with respect to some $\mathrm{offset}$. $j-th$ bin boundaries are $[j \cdot \mathrm{window} + \mathrm{offset}; (j + 1) \cdot \mathrm{window} + \mathrm{offset}]$.

Binned time series is defined by $$t_j^* = (j + \frac12) \cdot \mathrm{window} + \mathrm{offset},$$ $$m_j^* = \frac{\sum{m_i / \delta_i^2}}{\sum{\delta_i^{-2}}},$$ $$\delta_j^* = \frac{N_j}{\sum{\delta_i^{-2}}},$$ where $N_j$ is a number of sampling observations and all sums are over observations inside considering bin.

Benchmarks

You can run all benchmarks from the Python project folder with python3 -mpytest --benchmark-enable tests/test_w_bench.py, or with slow benchmarks disabled python3 -mpytest -m "not (nobs or multi)" --benchmark-enable tests/test_w_bench.py.

Here we benchmark the Rust implementation (rust) versus feets package and our own Python implementation (lc_py) for a light curve having n=1000 observations.

The plot shows that the Rust implementation of the package outperforms other ones by a factor of 1.5—50. This allows to extract a large set of "cheap" features well under one ms for n=1000. The performance of parametric fits (BazinFit and VillarFit) and Periodogram depend on their parameters, but the typical timescale of feature extraction including these features is 20—50 ms for few hundred observations.

Benchmark results of several features for both the pure-Python and Rust implementations of the ”light-curve” package, as a function of the number of observations in a light curve. Both the x-axis and y-axis are on a logarithmic scale.

Processing time per a single light curve for extraction of features subset presented in first benchmark versus the number of CPU cores used. The dataset consists of 10,000 light curves with 1,000 observations in each.

See benchmarks' descriptions in more details in "Performant feature extraction for photometric time series".

dm-dt map

Class DmDt provides dm–dt mapper (based on Mahabal et al. 2011, Soraisam et al. 2020). It is a Python wrapper for light-curve-dmdt Rust crate.

import numpy as np
from light_curve import DmDt
from numpy.testing import assert_array_equal

dmdt = DmDt.from_borders(min_lgdt=0, max_lgdt=np.log10(3), max_abs_dm=3, lgdt_size=2, dm_size=4, norm=[])

t = np.array([0, 1, 2], dtype=np.float32)
m = np.array([0, 1, 2], dtype=np.float32)

desired = np.array(
    [
        [0, 0, 2, 0],
        [0, 0, 0, 1],
    ]
)
actual = dmdt.points(t, m)

assert_array_equal(actual, desired)

Citation

If you found this project useful for your research please cite Malanchev et al., 2021

@ARTICLE{2021MNRAS.502.5147M,
       author = {{Malanchev}, K.~L. and {Pruzhinskaya}, M.~V. and {Korolev}, V.~S. and {Aleo}, P.~D. and {Kornilov}, M.~V. and {Ishida}, E.~E.~O. and {Krushinsky}, V.~V. and {Mondon}, F. and {Sreejith}, S. and {Volnova}, A.~A. and {Belinski}, A.~A. and {Dodin}, A.~V. and {Tatarnikov}, A.~M. and {Zheltoukhov}, S.~G. and {(The SNAD Team)}},
        title = "{Anomaly detection in the Zwicky Transient Facility DR3}",
      journal = {\mnras},
     keywords = {methods: data analysis, astronomical data bases: miscellaneous, stars: variables: general, Astrophysics - Instrumentation and Methods for Astrophysics, Astrophysics - Solar and Stellar Astrophysics},
         year = 2021,
        month = apr,
       volume = {502},
       number = {4},
        pages = {5147-5175},
          doi = {10.1093/mnras/stab316},
archivePrefix = {arXiv},
       eprint = {2012.01419},
 primaryClass = {astro-ph.IM},
       adsurl = {https://ui.adsabs.harvard.edu/abs/2021MNRAS.502.5147M},
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}