synthetic-disagreement

Stress-test an inter-annotator agreement pipeline with deterministic synthetic reviewer labels.

synthetic-disagreement is for annotation leads, evaluation methodologists, and test engineers who want to exercise calibration code before collecting real labels. It provides uniform, biased, ordinal-slip, adversarial, and drift noise models, then feeds generated rating matrices into iaa-kit to show how agreement changes as noise increases.

Inspectable Output

  • Python API: generate returns a deterministic mapping keyed by (item_id, rater_id).
  • Installed CLI generate: a JSON report whose labels rows contain item id, rater id, generated label, noise model, seed, and synthetic: true.
  • Installed CLI stress-curve: a JSON report whose curve points contain noise level and Fleiss kappa agreement, with an optional deterministic SVG.

The same inputs, seed, model, and levels produce the same output.

Runtime Boundary

Generation and stress curves run entirely in the local Python process with NumPy and iaa-kit. There are no network requests, model calls, reviewer services, telemetry hooks, or uploaded task data.

Install

Python library and CLI:

python -m pip install synthetic-disagreement==0.1.2

Python API Quickstart

python - <<'PY'
from synthetic_disagreement import UniformNoise, generate

labels = generate(
    [{"item_id": "1", "label": 1}],
    ["r1", "r2"],
    UniformNoise(rate=0.2, seed=7),
    [0, 1],
)
print(labels)
PY

CLI Quickstart

synthetic-disagreement generate \
  --items examples/synthetic_items.json \
  --raters r1,r2,r3 \
  --noise-model uniform \
  --rate 0.2 \
  --seed 7 \
  --out synthetic-labels.json

synthetic-disagreement stress-curve \
  --items examples/synthetic_items.json \
  --raters r1,r2,r3 \
  --noise-model uniform \
  --levels 0,0.25,0.5 \
  --seed 7 \
  --out agreement-curve.json \
  --svg-out agreement-curve.svg

See docs/methodology.md for noise-model and data-policy details.

Release Status

Registry and artifact status verified July 13, 2026: version 0.1.2 is published on PyPI and tagged v0.1.2 in the public repository. The wheel includes the Python API plus the JSON and SVG CLI. The project is alpha software. No reviewer-volume, quality, or adoption claim is made.

Limits

Synthetic noise is a test fixture, not a model of every real reviewer population. The generated curves do not replace pilot annotation, reviewer training, adjudication, or empirical agreement studies.

Next Action

Use the Python API or installed CLI to generate seeded labels or a stress curve, confirm the pipeline responds as expected, then compare the result with a real pilot annotation study.

Project links

Data verified by PyPI on Jul 13, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for auraoneai from gravatar.com auraoneai

Credits

Author: AuraOne

GitHub Statistics

Data verified by PyPI on Jul 13, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT
View SPDX License List

Requires

Python >=3.10

Tags

inter-annotator-agreement synthetic-data annotation-quality reviewer-disagreement fleiss-kappa evals

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

synthetic_disagreement-0.1.2.tar.gz (9.8 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

synthetic_disagreement-0.1.2-py3-none-any.whl (10.8 kB view details)

Uploaded Python 3

File details

Details for the file synthetic_disagreement-0.1.2.tar.gz.

File metadata

  • Download URL: synthetic_disagreement-0.1.2.tar.gz
  • Upload date:
  • Size: 9.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for synthetic_disagreement-0.1.2.tar.gz
Algorithm Hash digest
SHA256 4a4aa52dd1235cb4c178ecd7a0ab8a822b4ce4f2ebd037d328a2e29b480c8811
MD5 846a7ad94b40466f1fca65d73b1f65da
BLAKE2b-256 be2b71080e37f69fd48f6b023c0182f51f0614f6369d63e0bcd3e0791c36c282

See more details on using hashes here.

Provenance

The following attestation bundles were made for synthetic_disagreement-0.1.2.tar.gz:

Publisher: release-python.yml on auraoneai/synthetic-disagreement

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file synthetic_disagreement-0.1.2-py3-none-any.whl.

File metadata

File hashes

Hashes for synthetic_disagreement-0.1.2-py3-none-any.whl
Algorithm Hash digest
SHA256 7929908b32db361e686e0868a8e37c308a11a3bb17b2430afe77675b3a71800e
MD5 5eb8f80626afe252f15d51cb1397968b
BLAKE2b-256 32576cfea2ca7e3dd38595fb2b4cbd66c7a6ac2fd037fdc715a4eb302c6b5f65

See more details on using hashes here.

Provenance

The following attestation bundles were made for synthetic_disagreement-0.1.2-py3-none-any.whl:

Publisher: release-python.yml on auraoneai/synthetic-disagreement

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.1.2 This release

2 files

0.1.1

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page