Skip to main content

The stand-alone ultra-fast simulation option for LHCb

Project description

GitHub address Doxygen on Pages

Lamarr logo

SQLamarr

The stand-alone ultra-fast simulation option for the LHCb experiment.

The detailed simulation of the hadron collisions at the LHC, and of the interaction of the generated particles with the detector material dominates the cost for the computing infrastructure pledged to the LHCb Collaboration.

Among the various options explored towards a faster simulation, there is Lamarr, a framework defining a pipeline of parametrizations transforming generator-level quantities to reconstructed, analysis-level features. Most of the parametrizations are defined using machine-learning, and in particular Deep Neural Networks and Gradient Boosted Decision Trees, with a training procedure defined in independent packages (e.g. landerlini/lb-trksim-train and mbarbetti/lb-pidsim-train).

To be integrated in the LHCb software stack, models must be queried from a C++ application, running in the Gaudi framework, which includes a dedicated multithreading scheduler which was found to conflict with the schedulers of TensorFlow and ONNX runtimes. In addition, since the models are relatively simple and fast to evaluate, the overhead of context switching from Gaudi to a dedicated runtime was observed to be unaccptably large. Hence, models are converted into compatible C code using the landerlini/scikinC package and distributed through the CernVM FileSystem releasing the LamarrData package.

While crucial to the applications within LHCb, the integration with Gaudi and Gauss makes the adoption of Lamarr unappealing for researchers outside of the LHCb community approaching the LHCb simulation to evaluate the experiment sensitivity to new physics phenomena or studying the recently-released LHCb Open Data. The landerlini/SQLamarr package aims at decoupling Lamarr from Gaudi providing a stand-alone application with minimal dependencies that can be easily set up and run in any Linux machine. The parametrizations are shared between the Gauss-embedded implementation LbLamarr and SQLamarr. In the future, the exact same package might be integrated within Gaudi to reduce the maintainance effort.

To replace the ROOT-based TransientEventStore concept defind in Gaudi, SQLamarr adopts the SQLite3 package, enabling vectorized processing of batches of events, for a better performance.

To avoid dependencies on ROOT, also the persistency is handled using SQLite3, writing the reconstructed (or intermediate) quantities in the form of SQLite3 databases. Note that converting an SQLite3 table to a ROOT nTuple requires no more than 3 lines of Python:

import sqlite3, uproot, pandas
with sqlite3.connect("SomeInput.db") as conn:
  uproot.open("SomeFile.root", "RECREATE")["myTree"] = pandas.read_sql_table("myTable", conn)

Dependencies

  • SQLite3 with C/C++ headers
  • HepMC3 as a standard interface to event generators.

Build from source

Make sure you have conda (or similar) installed, if not get miniconda3. Create and activate a dedicated conda environment, say sqlamarr:

conda create -y -n sqlamarr -c conda-forge python=3.10 gxx gxx_linux-64 hepmc3 doxygen
conda activate sqlamarr

Create a out-of-source directory:

mkdir build
cd build

Configure and build

cmake .. 
cmake --build .

How to use SQLamarr

The project is not mature enough to provide a good user experience. For the time being, clone the repository and compile the package with CMake, then edit the file src/main.cpp to define the desired pipeline, by using the building blocks provided by the package.

BlockLib

The namespace SQLamarr::BlockLib groups functions defining specialized blocks making assumptions on the workflow in which these blocks will be deployed. While useful for testing and for organizing the code defining pipelines, it is not supposed to be stable (as it is being modified while the workflow under test is modified) and should not be used as part of other packages.

Other packages, however, may take inspiration from the structure of SQLamarr::BlockLib to design specialized blocks, resident in their codebase, in a more organized way than having everything pipelined in a single file.

To test the completeness of the feature set in the main part of the library, SQLamarr::BlockLib is designed to only include functions, accessing public methods of the objects defined in the main part of the library.

Copyright and Licence

(c) Copyright 2022 CERN for the benefit of the LHCb Collaboration.

This software is distributed under the terms of the GNU General Public Licence version 3 (GPL Version 3), copied verbatim in the file "LICENCE".

In applying this licence, CERN does not waive the privileges and immunities granted to it by virtue of its status as an Intergovernmental Organization or submit itself to any jurisdiction.

We acknowledge the support of the ICSC Foundation to the development of SQLamarr.

image

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

SQLamarr-0.0rc7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

SQLamarr-0.0rc7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

SQLamarr-0.0rc7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

SQLamarr-0.0rc7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

SQLamarr-0.0rc7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

SQLamarr-0.0rc7-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

File details

Details for the file SQLamarr-0.0rc7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for SQLamarr-0.0rc7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 36599a069b638734d9829cb71452496e34589fc9abacdd56f3bcf8366492f3c9
MD5 dc43adb29195557db3ee7aa9d126b70e
BLAKE2b-256 8d63bec5f510766534f329ee1634d7cf17a1e8deb4e6fa0f1cedc71c415045f4

See more details on using hashes here.

File details

Details for the file SQLamarr-0.0rc7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for SQLamarr-0.0rc7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 3188234dbf4df4751af0a5a1a8c56b1de957d3e35800c4d63bc8fa39aabe40b6
MD5 342c481cbd461ecf6b9773e48a780c0d
BLAKE2b-256 a3b83be8beaa50d90e040384262cbd83272047e34aafdc1d64bd729399c98a10

See more details on using hashes here.

File details

Details for the file SQLamarr-0.0rc7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for SQLamarr-0.0rc7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 821d66d94fab464898bf37b739439911d24bfd67870c052524309edbf4e8f81c
MD5 7b5c069c7968c4b31c18d491f466cb58
BLAKE2b-256 5a6c3cfa6cc65a4a691d2acf288725ba5d8e7cf32ae080d097701e4b6f92231e

See more details on using hashes here.

File details

Details for the file SQLamarr-0.0rc7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for SQLamarr-0.0rc7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 8e899a67943ca58ed2f574157ec3d6312017be34ca440c7b36ab5e9fc4afe463
MD5 d8b025781193131fe02399df3ef7bba4
BLAKE2b-256 557b3e2f0ca08f2836296adff60ed9d99382e12659543cd00a7fe03262818e2d

See more details on using hashes here.

File details

Details for the file SQLamarr-0.0rc7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for SQLamarr-0.0rc7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c5eee977b6825e009f673e759b36c88900ecd7a3184bd2a750f7b81f24e150b2
MD5 c18f48aa23934a5ad83cc549566d1a08
BLAKE2b-256 7af843acfe1a536468b20145e39fdce517babc98eb49e78d5f3805465bffbdfb

See more details on using hashes here.

File details

Details for the file SQLamarr-0.0rc7-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for SQLamarr-0.0rc7-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ebff593a9a2657119bae3ac5867e1a585bf343be1189277141ac6b903a42d6ea
MD5 ad097d93f543245554f3c292e817ceb5
BLAKE2b-256 740ce4bf610eeb92c3f80e8e3e6be036b0f7552fb8b074e866b06e57b92fb75d

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page