Skip to main content

scalable pythonic model fitting for high energy physics

Project description

zfit: scalable pythonic fitting

https://zenodo.org/badge/126311570.svg https://img.shields.io/pypi/v/zfit.svg https://travis-ci.org/zfit/zfit.svg?branch=develop https://coveralls.io/repos/github/zfit/zfit/badge.svg?branch=meta_changes CodeFactor

UPGRADE zfit: Upgrade guide to 0.4.x

zfit is a highly scalable and customizable model manipulation and fitting library. It uses TensorFlow as its computational backend and is optimised for simple and direct manipulation of probability density functions.

If you use zfit in research, please consider citing.

N.B.: zfit is currently in beta stage, so while most core parts are established, some may still be missing and bugs may be encountered. It is, however, mostly ready for production, and is being used in analyses projects. If you want to use it for your project and you are not sure if all the needed functionality is there, feel free to contact.

Why?

The basic idea behind zfit is to offer a Python oriented alternative to the very successful RooFit library from the ROOT data analysis package that can integrate with the other packages that are part if the scientific Python ecosystem. Contrary to the monolithic approach of ROOT/RooFit, the aim of zfit is to be light and flexible enough to integrate with any state-of-art tools and to allow scalability going to larger datasets.

These core ideas are supported by two basic pillars:

  • The skeleton and extension of the code is minimalist, simple and finite: the zfit library is exclusively designed for the purpose of model fitting and sampling with no attempt to extend its functionalities to features such as statistical methods or plotting.
  • zfit is designed for optimal parallelisation and scalability by making use of TensorFlow as its backend. The use of TensorFlow provides crucial features in the context of model fitting like taking care of the parallelisation and analytic derivatives.

How to use

While the zfit library provides a model fitting and sampling framework for a broad list of applications, we will illustrate its main features with a simple example by fitting a Gaussian distribution with an unbinned likelihood fit and a parameter uncertainty estimation.

Example in short

obs = zfit.Space('x', limits=(-10, 10))

# create the model
mu    = zfit.Parameter("mu"   , 2.4, -1, 5)
sigma = zfit.Parameter("sigma", 1.3,  0, 5)
gauss = zfit.pdf.Gauss(obs=obs, mu=mu, sigma=sigma)

# load the data
data_np = np.random.normal(size=10000)
data = zfit.Data.from_numpy(obs=obs, array=data_np)

# build the loss
nll = zfit.loss.UnbinnedNLL(model=gauss, data=data)

# minimize
minimizer = zfit.minimize.Minuit()
result = minimizer.minimize(nll)

# calculate errors
param_errors = result.error()

This follows the zfit workflow

zfit workflow

Full explanation

The default space (e.g. normalization range) of a PDF is defined by an observable space, which is created using the zfit.Space class:

obs = zfit.Space('x', limits=(-10, 10))

To create a simple Gaussian PDF, we define its parameters and their limits using the zfit.Parameter class.

# syntax: zfit.Parameter("any_name", value, lower, upper)
  mu    = zfit.Parameter("mu"   , 2.4, -1, 5)
  sigma = zfit.Parameter("sigma", 1.3,  0, 5)
  gauss = zfit.pdf.Gauss(obs=obs, mu=mu, sigma=sigma)

For simplicity, we create the dataset to be fitted starting from a numpy array, but zfit allows for the use of other sources such as ROOT files:

mu_true = 0
sigma_true = 1
data_np = np.random.normal(mu_true, sigma_true, size=10000)
data = zfit.Data.from_numpy(obs=obs, array=data_np)

Fits are performed in three steps:

  1. Creation of a loss function, in our case a negative log-likelihood.
  2. Instantiation of our minimiser of choice, in the example the Minuit.
  3. Minimisation of the loss function.
# Stage 1: create an unbinned likelihood with the given PDF and dataset
nll = zfit.loss.UnbinnedNLL(model=gauss, data=data)

# Stage 2: instantiate a minimiser (in this case a basic minuit)
minimizer = zfit.minimize.Minuit()

# Stage 3: minimise the given negative log-likelihood
result = minimizer.minimize(nll)

Errors are calculated with a further function call to avoid running potentially expensive operations if not needed:

param_errors = result.error()

Once we’ve performed the fit and obtained the corresponding uncertainties, we can examine the fit results:

print("Function minimum:", result.fmin)
print("Converged:", result.converged)
print("Full minimizer information:", result.info)

# Information on all the parameters in the fit
params = result.params
print(params)

# Printing information on specific parameters, e.g. mu
print("mu={}".format(params[mu]['value']))

And that’s it! For more details and information of what you can do with zfit, checkout the documentation.

Prerequisites

zfit works with Python versions 3.6 and 3.7. The following packages (amongst others) are required:

… and some minor packages. For a full list, check the requirements.

Installing

zfit is available on conda-forge and pip. If possible, use a conda or virtual environment and do:

For conda:

$ conda install zfit -c conda-forge

For pip (if you don’t use conda):

$ pip install zfit

For the newest development version, you can install the version from git with

$ pip install git+https://github.com/zfit/zfit

Contributing

Any idea of how to improve the library? Or interested to write some code? Contributions are always welcome, please have a look at the Contributing guide.

Contact

You can contact us directly:

Original Authors

Jonas Eschle <jonas.eschle@cern.ch>
Albert Puig <albert.puig@cern.ch>
Rafael Silva Coutinho <rsilvaco@cern.ch>

See here for all authors and contributors

Acknowledgements

zfit has been developed with support from the University of Zürich and the Swiss National Science Foundation (SNSF) under contracts 168169 and 174182.

The idea of zfit is inspired by the TensorFlowAnalysis framework developed by Anton Poluektov using the TensorFlow open source library.

Changelog

Develop

Major Features and Improvements

Behavioral changes

Bug fixes and small changes

Requirement changes

Thanks

0.4.1 (12.1.20)

Release to keep up with TensorFlow 2.1

Major Features and Improvements

  • Fixed the comparison in caching the graph (implementation detail) that leads to an error.

0.4.0 (7.1.2020)

This release switched to TensorFlow 2.0 eager mode. In case this breaks things for you and you need urgently a running version, install a version < 0.4.1. It is highly recommended to upgrade and make the small changes required.

Please read the upgrade guide <docs/project/upgrade_guide.rst> on a more detailed explanation how to upgrade.

TensorFlow 2.0 is eager executing and uses functions to abstract the performance critical parts away.

Major Features and Improvements

  • Dependents (currently, and probably also in the future) need more manual tracking. This has mostly an effect on CompositeParameters and SimpleLoss, which now require to specify the dependents by giving the objects it depends (indirectly) on. For example, it is sufficient to give a ComplexParameter (which itself is not independent but has dependents) to a SimpleLoss as dependents (assuming the loss function depends on it).

  • ComposedParameter does no longer allow to give a Tensor but requires a function that, when evaluated, returns the value. It depends on the dependents that are now required.

  • Added numerical differentiation, which allows now to wrap any function with z.py_function (zfit.z). This can be switched on with zfit.settings.options[‘numerical_grad’] = True

  • Added gradient and hessian calculation options to the loss. Support numerical calculation as well.

  • Add caching system for graph to prevent recursive graph building

  • changed backend name to z and can be used as zfit.z or imported from it. Added:

    • function decorator that can be used to trace a function. Respects dependencies of inputs and automatically caches/invalidates the graph and recreates.
    • py_function, same as tf.py_function, but checks and may extends in the future
    • math module that contains autodiff and numerical differentiation methods, both working with tensors.

Behavioral changes

  • EDM goal of the minuit minimizer has been reduced by a factor of 10 to 10E-3 in agreement with the goal in RooFits Minuit minimizer. This can be varied by specifying the tolerance.
  • known issue: the projection_pdf has troubles with the newest TF version and may not work properly (runs out of memory)

Bug fixes and small changes

Requirement changes

  • added numdifftools (for numerical differentiation)

Thanks

0.3.7 (6.12.19)

This is a legacy release to add some fixes, next release is TF 2 eager mode only release.

Major Features and Improvements

  • mostly TF 2.0 compatibility in graph mode, tests against 1.x and 2.x

Behavioral changes

Bug fixes and small changes

  • get_depentents returns now an OrderedSet
  • errordef is now a (hidden) attribute and can be changed
  • fix bug in polynomials

Requirement changes

  • added ordered-set

0.3.6 (12.10.19)

Special release for conda deployment and version fix (TF 2.0 is out)

This is the last release before breaking changes occur

Major Features and Improvements

  • added ConstantParameter and zfit.param namespace
  • Available on conda-forge

Behavioral changes

  • an implicitly created parameter with a Python numerical (e.g. when instantiating a model) will be converted to a ConstantParameter instead of a fixed Parameter and therefore cannot be set to floating later on.

Bug fixes and small changes

  • added native support TFP distributions for analytic sampling
  • fix Gaussian (TFP Distribution) Constraint with mixed up order of parameters
  • from_numpy automatically converts to default float regardless the original numpy dtype, dtype has to be used as an explicit argument

Requirement changes

  • TensorFlow >= 1.14 is required

Thanks

  • Chris Burr for the conda-forge deployment

0.3.4 (30-07-19)

This is the last release before breaking changes occur

Major Features and Improvements

  • create Constraint class which allows for more fine grained control and information on the applied constraints.
  • Added Polynomial models
  • Improved and fixed sampling (can still be slightly biased)

Behavioral changes

None

Bug fixes and small changes

  • fixed various small bugs

Thanks

for the contribution of the Constraints to Matthieu Marinangeli <matthieu.marinangeli@cern.ch>

0.3.3 (15-05-19)

Fixed Partial numeric integration

Bugfixes mostly, a few major fixes. Partial numeric integration works now.

Bugfixes
  • data_range cuts are now applied correctly, also in several dimensions when a subset is selected (which happens internally of some Functors, e.g. ProductPDF). Before, only the selected obs was respected for cuts.
  • parital integration had a wrong take on checking limits (now uses supports).

0.3.2 (01-05-19)

With 0.3.2, bugfixes and three changes in the API/behavior

Breaking changes

  • tfp distributions wrapping is now different with dist_kwargs allowing for non-Parameter arguments (like other dists)
  • sampling allows now for importance sampling (sampler in Model specified differently)
  • model.sample now also returns a tensor, being consistent with pdf and integrate

Bugfixes

  • shape handling of tfp dists was “wrong” (though not producing wrong results!), fixed. TFP distributions now get a tensor with shape (nevents, nobs) instead of a list of tensors with (nevents,)

Improvements

  • refactor the sampling for more flexibility and performance (less graph constructed)
  • allow to use more sophisticated importance sampling (e.g. phasespace)
  • on-the-fly normalization (experimentally) implemented with correct gradient

0.3.1 (30-04-19)

Minor improvements and bugfixes including:

  • improved importance sampling allowing to preinstantiate objects before it’s called inside the while loop
  • fixing a problem with ztf.sqrt

0.3.0 (2019-03-20)

Beta stage and first pip release

0.0.1 (2018-03-22)

  • First creation of the package.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for zfit, version 0.4.1
Filename, size File type Python version Upload date Hashes
Filename, size zfit-0.4.1-py2.py3-none-any.whl (147.4 kB) File type Wheel Python version py2.py3 Upload date Hashes View hashes
Filename, size zfit-0.4.1.tar.gz (252.3 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page