conformly

Declarative test data generator for Python. Turns data models (dataclasses, Pydantic, TypedDict) and type constraints into valid fixtures and negative test cases. Define constraints once in type annotations — generate both valid and minimal invalid test data automatically. No factories, no hardcoded fixtures, no drift when schema changes.

Table of contents

• Key Features
• Install
• Quickstart
  • With dataclasses
  • With Pydantic
  • With TypedDict
  • With Attrs
  • With Mixed Models
• API Reference
• Error Handling
• Invalid Generation Contract
• Optional Fields and Defaults
• Constraints
  • Supported Constraints
  • Defining Constraints
  • Special String types
• User Cases
• Nested Models
• Collections
• Experimental
• Development
• Changelog
• License
• Contributing

Key Features

• Constraint-driven generation - type constraints act as executable generation rules
• Minimal invalid cases - only the targeted field violates constraints; everything else stays valid
• Schema as single source of truth - change a constraint → all test data adapts automatically
• Unified constraint model - multiple declaration styles normalized internaly
• Framework adapters - dataclasses (built-in), Pydantic (optional via conformly[pydantic])
• Cross-adapter nesting - freely combine schemas across frameworks (e.g. TypedDict inside dataclass, dataclass inside Pydantic)

Install

# Core functionality (dataclasses support)
pip install conformly

# With Pydantic support
pip install conformly[pydantic]

# With Attrs support
pip install conformly[attrs]

Quickstart

With dataclasses

from dataclasses import dataclass
from typing import Annotated
from conformly import case, MinLength, Pattern, GreaterOrEqual, LessOrEqual


@dataclass
class User:
    username: Annotated[str, MinLength(3)]
    email: Annotated[str, Pattern(r"^[^\s@]+@[^\s@]+\.[^\s@]+$")]
    age: Annotated[int, GreaterOrEqual(18), LessOrEqual(120)]

valid = case(User, valid=True)
# -> {"username": "Abc", "email": "x@y.z", "age": 42}

With Pydantic

from pydantic import BaseModel, Field
from conformly import case

class User(BaseModel):
    username: str = Field(..., min_length=3, max_length=32)
    email: str = Field(..., pattern=r"^[^\s@]+@[^\s@]+\.[^\s@]+$")
    age: int = Field(..., ge=18, le=120)

valid = case(User, valid=True)
# -> {"username": "Abc", "email": "x@y.z", "age": 42}

With TypedDict

from typing import Annotated, TypedDict
from conformly import case, MinLength, Pattern, GreaterOrEqual, LessOrEqual


class User(TypedDict):
    username: Annotated[str, MinLength(3)]
    email: Annotated[str, Pattern(r"^[^\s@]+@[^\s@]+\.[^\s@]+$")]
    age: Annotated[int, GreaterOrEqual(18), LessOrEqual(120)]

valid = case(User, valid=True)
# -> {"username": "Abc", "email": "x@y.z", "age": 42}

With Attrs

import attrs
from typing import Annotated
from conformly import case, MinLength, Pattern, GreaterOrEqual, LessOrEqual


@attrs.define
class User:
    username: Annotated[str, MinLength(3)]
    email: Annotated[str, Pattern(r"^[^\s@]+@[^\s@]+\.[^\s@]+$")]
    age: Annotated[int, GreaterOrEqual(18), LessOrEqual(120)]

valid = case(User, valid=True)
# -> {"username": "Abc", "email": "x@y.z", "age": 42}

Note: Native attrs validators are not yet supported. To define constraints for attrs models, please use typing.Annotated with conformly constraints as shown in the example above. Support for native attrs validators will be added later

With Mixed Models

from dataclasses import dataclass
from typing import TypedDict
from pydantic import BaseModel


class Meta(TypedDict):
    version: int


@dataclass
class Profile:
    meta: Meta


class User(BaseModel):
    profile: Profile


valid = case(User, valid=True)
# -> {"profile": {"meta": {"version": 1}}}

API Reference

case(
    model,
    *,
    valid: bool,
    seed: int | None = None,
    strategy: str | None = None,
    overrides: list[PathSelector] | None = None,
    allow_type_mismatch: bool = False,
) -> dict

cases(
    model,
    *,
    valid: bool,
    seed: int | None = None,
    strategy: str = "all",
    overrides: list[PathSelector] | None = None,
    count: int | None = None,
    allow_type_mismatch: bool = False,
    allow_structural_violations: bool = False
) -> list[dict]

strategy values:

• <field_name>(legacy) - target specific field for invalidation (for nested fields using dot syntax "profile.name")
• <field_name>::<violations>(legacy) - target specific violation for field (syntax "profile.name::too_long")
• DSL-based targeting (recommended for violation targeting):

from conformly import path, V

# type safe
path(User, lambda u: u.email).violate(V.TOO_SHORT)
path(User, lambda u: u.profile.name).violate(V.PATTERN_MISMATCH)

# string-based
path("user.email").violate(V.TOO_SHORT)
path("profile.name").violate(V.PATTERN_MISMATCH)

• "random" - choose a random field/constraint to violate
• "all" - (for cases) produce all minimal invalid variations for the model
• "first" - violate the first constrained field (for case) or take the first N constrained fields (for cases)
• "all_violations" - generate one invalid case per every available violations including constraints, structural and type violations (ignores count)
• "overrides" - default values for fields in current generation

Note: If a field is selected for invalidation → override is ignored. Otherwise → override is applied.

overrides=[
    path(User, lambda u: u.full_name).set("Amogus"),
    path(User, lambda u: u.email).set("amogus@example.com"),
]

Integer seeds, including 0 and negative values, make case() and cases() reproducible for internal generation, including nested models and UUID values. The same model, options, and seed produce the same payloads. User-provided default factories run once per generated field and must themselves be deterministic; the seed does not control their randomness or external state.

Error Handling

All conformly errors inherit from ConformlyError, providing consistent interface for debugging and programmatic handling.

Structured context

Every exception includes:

• message: str — human-readable description
• context: dict — diagnostic data with stable code for programmatic checks

from conformly import case
from conformly.exceptions import GenerationError

try:
    case(User, valid=False, strategy="email::invalid_type")
except GenerationError as e:
    print(e.message)           # "Unknown violation type 'invalid_type'"
    print(e.context["code"])   # "invalid_violation_type"
    print(e.context["available"])  # ["too_short", "too_long", "pattern_mismatch", ...]

Invalid Generation Contract

For case(Model, valid=False, strategy="<field>"):

• Violation priority: generator choose first violation type from allowed based on priority (Structural (Missing/Extra) > Type Mismatch > Semantic (Range/Pattern/Value))
• If allow_type_mismatch=True, the generator may substitute a type mismatch (e.g., string instead of int) in place of a semantic constraint violation for the targeted field

• If allow_structural_violations=True, generator may substitute field missing in place of any other violations (avaliable only with strategy="all")
• Exactly one field is targeted (the one specified by strategy).
• The generator will violate constraints for that field, making it invalid.
• Inside collections, planning selects a compatible dictionary key, value, or element before generation. Fixed tuples retain their length and order; unrelated elements, keys, values, and model fields remain valid.
• Only the selected violation is guaranteed on its target. Other constraints on that same target are not guaranteed to remain satisfied. A DUPLICATE violation preserves an allowed collection length; collections with a maximum size below two cannot be targeted for duplication.
• Explicit impossible or unsupported violations raise PlanningError; empty-only collections do not offer element violations.
• For numeric bounds, invalid values may violate the lower or upper bound (e.g., age > 120 or age < 18).
• For float bounds, invalid generation may produce inf when violating the upper boundary.

Request an exact violation with strategy="items::below_min" or path(Model, lambda m: m.items).violate(V.BELOW_MIN). Otherwise, the planner uses violation priority and chooses the first compatible key, value, or tuple element.

Optional Fields and Defaults

• For untargeted fields, priority is override → nullable None → model default or default factory → generation. Nullable fields currently always return None without an override, even with a non-None default; their factories are not called.
• Model defaults, factory results and overrides are trusted and returned without validation. valid=True guarantees the validity of generated values, not of caller-supplied data. An invalid supplied value can also make an unrelated field invalid when generating a negative case.
• Invalid targets bypass their defaults, nullable priority and overrides. Other fields retain the priority above. Default factories are called once per generated payload when selected by this priority; attrs factories requiring self are unsupported.
• TypedDict NotRequired[T] controls key presence independently of whether T accepts None. Valid generation currently includes these keys with values of T; NotRequired[int] does not become a present None.
• MISSING_FIELD is offered only for required fields: nullable required fields can be removed, but fields with defaults and TypedDict NotRequired keys cannot. EXTRA_FIELD is offered for Pydantic only with extra="forbid"; ignore and allow accept extra data. Other adapters treat undeclared keys as structural violations.
• A target needs an applicable semantic, type or structural violation; impossible explicit requests raise PlanningError.

Constraints

Supported constraints

Pattern uses Python regex search semantics, consistent with Pydantic: a match anywhere in the string is sufficient. Use anchors (^...$) to require a whole-string match. Pattern mismatches are verified; unsuccessful inversion raises a structured GenerationError instead of returning a matching value. Obvious universal patterns (such as (?s).*) are excluded from automatic planning, and explicit mismatch requests raise PlanningError.

MultipleOf requires a finite positive divisor. For floats, a quotient within 1e-9 of an integer is treated as a multiple (no relative tolerance). Integer NOT_MULTIPLE requests for divisors such as 1 or 0.5, which divide every integer, raise PlanningError and are excluded from automatic planning. A bounded range without a representable multiple raises GenerationError with code no_multiples_of.

Important: Pydantic's constr(), conint(), and functional validators are not interpreted as constraints. Use Field() parameters for constraint extraction.

Defining constraints

1) Annotated[..., Constraint(...)] (recommended)

You can use for all model types

from typing import Annotated
from conformly import MinLength, GreaterOrEqual

username: Annotated[str, MinLength(3)]

2) Annotated[..., "k=v"] (shorthand string syntax)

title: Annotated[str, "min_length=5", "max_length=200"]

3) Field(...) (Pydantic only)

from pydantic import Field

username: str = Field(..., min_length=3)

4) field(metadata={...}) (dataclasses only)

from dataclasses import field

sku: str = field(metadata={"pattern": r"^[A-Z0-9]{8}$"})

All syntaxes are fully compatible within their respective frameworks.

Special String types

conformly provides semantic marker types for common string formats. These types enable realistic test data generation while maintaining type safety.

Available types

Usage

from typing import Annotated
from conformly import Email, MinLength

# Basic usage
contact: Email

# With additional constraints
website: Annotated[URL, MinLength(20)]

# Pydantic models (auto-mapped)
from pydantic import BaseModel, EmailStr

class Config(BaseModel):
    admin_email: EmailStr  # Automatically uses Email generator

UUID support

Standard uuid.UUID fields are supported out of the box. Values are generated in canonical RFC 4122 v4 format (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, lowercase).

import uuid
from dataclasses import dataclass
from conformly import case, path, V

@dataclass
class Session:
    id: uuid.UUID
    user_id: uuid.UUID

valid = case(Session, valid=True)
# -> {"id": UUID("550e8400-e29b-41d4-a716-446655440000"), ...}

invalid = case(Session, valid=False, strategy=path("id").violate(V.TOO_SHORT))
# -> {"id": "550e8400-e29b-41d4-a716-44665544000"}  # TOO_SHORT
invalid = case(Session, valid=False, strategy=path(id).violate(V.WRONG_UUID_CHARACTER))
# -> {"id": "550e8400-e29b-41d4-a71@-446655440000"} # WRONG_UUID_CHARACTER

Use Cases

API Testing

# Valid payloads for happy-path tests
for _ in range(100):
    payload = case(CreateUserRequest, valid=True)
    response = client.post("/users", json=payload)
    assert response.status_code == 201

# Invalid payloads for error handling tests
invalid = case(CreateUserRequest, valid=False, strategy="age")
response = client.post("/users", json=invalid)
assert response.status_code == 400

# As option create all possible invalid cases for payload in one only line
invalid_payloads = cases(CreateUserRequest, valid=False, strategy="all")
for payload in invalid_payloads:
    response = client.post("/users", json=payload)
    assert response.status_code == 400

Database Seeding

# Generate realistic test data respecting schema constraints
products = cases(Product, valid=True, count=1000)
db.insert_many("products", products)

Fuzzing & Property-Based Testing

Conformly is not a replacement of Hypothesis, but a complementary tool for schema-driven testing and negative case generation.

# Generate random invalid data to stress-test validation
for _ in range(500):
    invalid = case(Model, valid=False, strategy="random")
    assert validate(invalid) is False  # Should always reject

Nested Models

Conformly supports nested models represented as tree structures (e.g. dataclasses containing other dataclasses).

Cyclic references between models are not supported

Constraints defined on nested fields are discovered recursively and can be used for both valid and invalid data generation.

Model Declaration

from dataclasses import dataclass
from typing import Annotated

from conformly import MinLength, GreaterOrEqual, Pattern

@dataclass
class Profile:
    email: Annotated[str, Pattern(r"^[^\s@]+@[^\s@]+\.[^\s@]+$")]
    phone: Annotated[str, Pattern(r"^\+[1-9]\d{1,14}$")]


@dataclass
class User:
    name: Annotated[str, MinLength(3)]
    age: Annotated[int, GreaterOrEqual(18)]
    profile: Profile

Generation Example

from conformly import case

valid_data = case(User, valid=True)
print(valid_data)
# {
#     "name": "validname",
#     "age": 25,
#     "profile": {
#            "email": "some@email.com",
#            "phone": "+12025550123"
#         }
# }

invalid_data_by_field = case(User, valid=False, strategy="profile.email")
print(invalid_data_by_field)
# {
#     "name": "validname",
#     "age": 25,
#     "profile": {
#            "email": "nonemailstring",
#            "phone": "+12025550123"
#         }
# }

Collections

List support

conformly supports basic generation of list[T] where T is a constrained primitive or a nested model.

Example

from dataclasses import dataclass
from typing import Annotated
from conformly import case, MinLength, MaxItems, UniqueItems

@dataclass
class Product:
    sku: str
    price: float

@dataclass
class Order:
    tags: list[str]                                                   # list of unconstrained strings
    codes: Annotated[list[Annotated[str, MinLength(5)]], MaxItems(6)] # each element ≥5 chars, max 6 items
    items: Annotated[list[Product], UniqueItems(True)]                # unique nested models

valid = case(Order, valid=True)
# -> {
#   "tags": ["abc", "def"],
#   "codes": ["ABCDE", "FGHIJ"],
#   "items": [{"sku": "...", "price": 10.0}]
# }

invalid = case(Order, valid=False, strategy="codes")
# -> {
#   "tags": [...],
#   "codes": ["ab", "VALID"],  # one element violates MinLength
#   "items": [...]
# }

Set and frozenset support

set[T] and frozenset[T] are normalized internally to list generation with UniqueItems(True) semantics:

@dataclass
class Model:
    tags: set[str]

case(Model)
# -> {"tags": ["a", "b", "c"]}  # always unique

This ensures consistent output format (list) while preserving uniqueness guarantees.

Tuple support

Fixed tuples and variadic tuples (tuple[T, ...]) are generated in their declared order. Element annotations and nested models are handled independently:

@dataclass
class Event:
    point: tuple[float, float]
    labels: tuple[str, ...]

case(Event, valid=True)
# -> {"point": (1.2, 3.4), "labels": ("abc", "def")}

For invalid generation, a constrained tuple element is selected and invalidated without changing the other elements.

Dict support

conformly supports generation of dict[K, V], where K must be a hashable type (str or Enum) and V can be any supported type (primitives, nested models, etc.). Collection-level constraints (MinItems, MaxItems) control the number of key-value pairs. The UniqueItems constraint is automatically ignored for dictionaries, as Python enforces key uniqueness natively.

Example

from dataclasses import dataclass
from typing import Annotated
from conformly import case, MinItems, MaxItems, path, V

@dataclass
class Product:
    sku: str
    price: float

@dataclass
class Inventory:
    metadata: dict[str, str]                                                    # simple key-value mapping
    products: Annotated[dict[str, Product], MinItems(2), MaxItems(4)]           # constrained pair count
    settings: Annotated[dict[str, int], MaxItems(3)]

valid = case(Inventory, valid=True)
# -> {
#   "metadata": {"brand": "acme", "status": "active"},
#   "products": {"P1": {"sku": "...", "price": 10.0}, "P2": {...}},
#   "settings": {"timeout": 30}
# }

invalid = case(Inventory, valid=False, strategy=path("products").violate(V.TOO_SHORT))
# -> {
#   "metadata": {...},
#   "products": {"P1": {"sku": "...", "price": 10.0}},  # violates MinItems(2)
#   "settings": {...}
# }

Known limitations

• No nested collections (list[list[T]] not supported yet)
• No fine-grained control over which index is violated (random selection only)
• Uniqueness for non-hashable elements (e.g., dicts) is best-effort (based on structural comparison fallback)
• Dictionary keys are restricted to str and Enum types; complex objects as keys are not supported

Experimental

APIs in this section are experimental and may change in future minor releases.

Tracing

Tracer captures metadata about case generation, making it easier to debug, reproduce, and report generated test cases.

from conformly import case, Email
from conformly.tracer import Tracer

tracer = Tracer()

payload = case(User, valid=False, tracer=tracer)

trace = tracer.build()

print(trace.target_path)
print(trace.generated_value)
print(trace.seed)

Tracing is available for case(..., valid=False). The trace describes the selected target: its dotted path, violation, generated value, seed, and value source. This applies to DSL and string paths as well as "first" and "random" selection. Unrelated fields, defaults, and overrides do not overwrite target metadata. For MISSING_FIELD, the value is the internal UNSET sentinel because the field is absent from the payload.

Enabling tracing leaves payloads, internal RNG consumption, and default factory call counts unchanged. A reused tracer describes the latest completed generation.

Development

Setup

Create a virtual environment and install dependencies:

make install-dev

Install with all optional features:

make install-all

Running tests

Run unit tests:

make test

Run all tests (including benchmarks):

make test-all

Run with coverage:

make test-cov

Code quality

Run linter:

make lint

Auto-fix lint issues:

make lint-fix

Run type checker:

make typecheck

Run all checks:

make check

Dependencies managment

Sync dependencies from lockfile: make sync Strict sync (CI mode): make sync-strict Update dependencies: make update

Changelog

See CHANGELOG.md for release notes and migration guidance.

License

MIT — see LICENSE file for details

Contributing

Contributions welcome!

• Fork the repo
• Create a feature branch
• Add tests for new functionality
• Run uv run -m pytest and uv run -m ruff check .
• Submit a pull request