dtfit

Differential-transformation fitting: nonlinear smoothing and forecasting on time-series / Big Data, using methods built in the scheme of differential (non-Taylor) transformations. Developed as part of a PhD dissertation on mathematical models of nonlinear smoothing and prediction.

Models can be a SymPy expression string or a plain Python callable f(x, *params); batch fits accept per-point measurement sigma (or sklearn sample_weight); and every fit carries its own quality diagnostics (R^2, AIC/BIC). All are opt-in - left at their defaults, fits are numerically identical to before.

Installation

pip install dtfit                        # core, from PyPI
pip install 'dtfit[viz]'                 # + matplotlib plotting helpers

From a clone of the repository (this package lives at packages/dtfit):

python -m venv .venv        # create an isolated environment
# Windows:  .venv\Scripts\activate
# Linux/mac: source .venv/bin/activate

# from the repo root:
pip install -e packages/dtfit            # core
pip install -e 'packages/dtfit[viz]'     # + matplotlib plotting helpers
pip install -e 'packages/dtfit[dev]'     # + test/lint tooling

# Optional: the experimental adaptations + experiment suite (a separate
# distribution that depends on dtfit; never shipped in the dtfit wheel):
pip install -e packages/dtfit-experimental             # dtfit_experimental
pip install -e 'packages/dtfit-experimental[bench]'    # + the suite's baselines/plotting

A plain venv with PyPI wheels is the reference environment and works identically on Windows and Linux. (Conda is not recommended on Windows: its MKL-linked numpy only loads its LAPACK DLLs when the env is activated, so tools that call the interpreter directly - VS Code, pytest - crash with a delay-load error.)

Quick start

import numpy as np
import dtfit as dt

x = np.linspace(0, 10, 400)
y = ...  # your observations

# Batch fit, numeric (no polynomial pre-fit needed):
result = dt.fit("a*atan(w*x)", dt.Original(x, y), "x", basis="block")
print(result.params)

# ...or through the scikit-learn compatible estimator:
from dtfit.sklearn import NonlineRegressor

reg = NonlineRegressor("a0 + a1*x + a2*exp(a3*x)", "x")
reg.fit(x, y)
y_hat = reg.predict(x)

Real-time / streaming

dt.ImageFilter tracks parameters online on the sliding window's image; ImageFilter(basis="legendre") and ImageFilter(basis="block") fix its basis to Legendre and block:

flt = dt.ImageFilter("A*sin(w*t)", "t", p0=[1.0, 1.0], window_size=50, basis="block")
for t, y in stream:           # bounded-cost per-sample update
    flt.partial_fit(t, y)
print(flt.params_)            # tracks time-varying parameters
print(flt.result().stderr())  # the window as a calibrated batch fit

Picking a model (the model framework)

Most of fitting is choosing the right structure. dtfit.models is a catalog of named families that seed their own p0/bounds from the data, compose with +, and can be ranked for you:

from dtfit import models, suggest_models

fit = models.logistic().fit(x, y)            # self-seeded; no p0/bounds to guess
fit = (models.linear() + models.sine()).fit(x, y)   # trend + cycle

for s in suggest_models(x, y)[:3]:           # infer the model from a scored shortlist
    print(s.name, s.r2, s.aic)

Uncertainty & serialization

A FittingResult is self-describing - named parameters, uncertainty, and a JSON-friendly round-trip:

r = dt.fit("a*exp(b*x)", dt.Original(x, y), "x", basis="legendre", p0=[1, 1])
r.params                       # {'a': ..., 'b': ...}
r.stderr(); r.confidence_intervals(0.95)
r.rsquared, r.aic, r.bic       # fit-quality diagnostics
y_hat, y_std = r.predict(x, return_std=True)   # prediction band
dt.FittingResult.from_dict(r.to_dict())        # save / ship a fitted model

# a Python callable model + per-point sigma both work (numerically opt-in):
r2 = dt.fit(lambda x, a, b: a * np.exp(b * x), dt.Original(x, y, sigma=noise_std), basis="block")               # signature-order params, weighted fit

Diagnostics & visualization

dtfit.diagnostics is fit-aware (it takes a FittingResult, not bare arrays) and does not reimplement sklearn.metrics - use those / scipy.stats for plain scalar metrics. It adds what's specific to evaluating a DT fit: information criteria for model comparison and residual-structure tests, plus the *Display plot helpers (which never call plt.show()).

from dtfit.diagnostics import fit_report, residual_diagnostics, FitDisplay

print(fit_report(r, x, y))            # n, rmse, r2, aic, bic, durbin_watson, params+-se
print(residual_diagnostics(r, x, y)) # autocorrelation / normality of residuals

FitDisplay.from_estimator(reg, x, y)  # data + fitted curve (needs the viz extra)

Methods

• LSI (fit(basis="legendre"), basis="legendre") - least-squares integral; numeric integral-OLS in the differential-transformation scheme (successor to DSBI).
• EAC (fit(basis="block"), basis="block") - equal-areas criterion; numeric, integration-based and noise-robust (successor to DSBE).
• ImageFilter - recursive/online tracker on the window image, with NIS drift detection; ImageFilter(basis="legendre") and ImageFilter(basis="block") fix its basis to Legendre and block for real-time tracking.
• Image analytics - an Image reports its own noise level (noise_sigma), how many orders carry signal (effective_order), how its coefficients decay (decay), whether it matches another image (test_equal) or a model (test_structure), and draws synthetic records from itself (simulate).
• DSB (dtfit_legacy.dsb.fit_dsb, in the dtfit-legacy package) - symbolic differential spectra balance; kept as the analytical reference, out of fit (it takes the coefficients of a polynomial pre-fit, not samples).
• Stochastic (fit_stochastic) - second-order characterization of a random series: every gate, estimator and forecaster reads one additive image of the record (SecondOrderImage), with a block-stream form (SecondOrderStream) and an online tracker (StochasticFilter).

Scaling out

• ImageStream accumulates a signal in fixed O(order) memory as it arrives (the default uniform accumulator; grid="explicit" retains every position and weight, and block mode grows coarse_ by one block per fold); its blocks are retained and assemble merges them into one Image, and a channel axis batches many signals through the same accumulator:

  s = dtfit.ImageStream("legendre", 6, domain=(0, 10), block=200)
  for x, y in chunks_of_the_stream():
      s.update(x, y)
  img = s.assemble(0, 10)
  

• dtfit.fit_many(problems, n_jobs=-1) fans many independent fits across cores (process or threading backend); the threading backend shares memory and avoids pickling.

Adaptations still in trial (further image bases, fits of aggregated data and of windows aligned to jumps, weak-form ODE identification, the filter bank) live in the separate dtfit-experimental package (dtfit_experimental); the notebooks that measure them are indexed in dtfit-experimental/experiments/README.md.

Each method's mathematical grounding (in differential / non-Taylor transformations), full algorithm, optimizations, guards, applicability, usage figures and comparison tables are documented in the project wiki.

Core dependencies: numpy, scipy, sympy, scikit-learn. Plotting helpers require the optional viz extra (matplotlib).