Skip to main content

Library to help analyze crowdsourcing results

Project description

IIIA-CSIC PyPI codecov


Crowdsourcing Citizen Science projects usually require citizens to classify items (images, pdfs, songs,…) into one of a finite set of categories. Once an image is classified by different citizens, the different votes need to be aggregated to obtain a consensus classification. Usually this is done by selecting the most voted category. crowdnalysis allows Crowdsourcing Citizen Science projects to compute consensus that go beyond the selection of the most voted category, by computing a model of quality for each of the citizen scientist involved in the project. This more advanced consensus results in higher quality information for the Crowdsourcing Citizen Science project.

Implemented consensus algorithms

  • Majority Voting
  • Probabilistic
  • Multinomial
  • Dawid-Skene

In addition to the pure Python implementations above, the following models are implemented in the probabilistic programming language Stan by using the CmdStanPy interface:

  • Multinomial
  • Multinomial Eta
  • Dawid-Skene
  • Dawid-Skene Eta Hierarchical

~ Eta models impose that the probability of the labels are higher for the real classes in the error-rate (a.k.a. confusion) matrix.


  • Import annotation data from a CSV file with a preprocessing option
  • Calculate inter-rater reliability with different measures
  • Fit selected model to annotation data and compute the consensus
  • Compute the consensus with a fixed pre-determined set of parameters
  • Fit the model parameters provided that the consensus is already known
  • Given the parameters of a generative model (Multinomial, Dawid-Skene), sample annotations, tasks, and workers (i.e., annotators)
  • Visualize the error-rate matrix for annotators
  • Conduct predictive analysis of the accuracy vs. number of annotations for a given set of models
  • Visualize the consensus on annotated images in HTML format

Quick start

crowdnalysis is distributed via PyPI:

Install as a standard Python package:

$ pip install crowdnalysis

CmdStanPy will be installed as a dependency, however, this package requires the installation of the CmdStan command-line interface too. This can be done via executing the install_cmdstan utility that comes with CmdStanPy. See the package docs for more information.

$ install_cmdstan

Use the package in the code:

import crowdnalysis

Check available consensus models:


How to run unit tests

We use pytest as the testing framework. Tests can be run by:

$ pytest

If you want to get the logs of the execution, do

$ pytest --log-cli-level 0


We use the standard logging library according to the rules here.


This project is licensed under the GNU Affero General Public License v3.0 - see the LICENSE file for details.


crowdnalysis is being developed within the Crowd4SDG project funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 872944.


For the details of the conceptual and mathematical model of crowdnalysis, see:

[1] Cerquides, J.; Mülâyim, M.O.; Hernández-González, J.; Ravi Shankar, A.; Fernandez-Marquez, J.L. A Conceptual Probabilistic Framework for Annotation Aggregation of Citizen Science Data. Mathematics 2021, 9, 875,

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

crowdnalysis-0.1.1.tar.gz (47.7 kB view hashes)

Uploaded Source

Built Distribution

crowdnalysis-0.1.1-py3-none-any.whl (63.1 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page