conformly 0.3.10

Generate valid & invalid test data from your typed schemas

conformly

Declarative test data generator for Python. Turns data models (dataclasses, Pydantic) and type constraints into valid fixtures and negative test cases. Define constraints once in type annotations — generate both valid and minimal invalid test data automatically. No factories, no hardcoded fixtures, no drift when schema changes.

---

Table of contents

• Key Features
• Install
• Quickstart
  • With dataclasses
  • With Pydantic
• API Reference
• Invalid Generation Contract
• Optional Fields and Defaults
• Constraints
  • Supported Constraints
  • Defining Constraints
  • Special String types
• User Cases
• Nested Models
• Collections
• Development
• Roadmap
• Changelog
• License
• Contributing

Key Features

• Constraint-driven generation - type constraints act as executable generation rules
• Minimal invalid cases - only the targeted field violates constraints; everything else stays valid
• Schema as single source of truth - change a constraint → all test data adapts automatically
• Unified constraint model - multiple declaration styles normalized internaly
• Framework adapters - dataclasses (built-in), Pydantic (optional via conformly[pydantic])

Install

# Core functionality (dataclasses support)
pip install conformly

# With Pydantic support
pip install conformly[pydantic]

Quickstart

With dataclasses

from dataclasses import dataclass
from typing import Annotated
from conformly import case, MinLength, Pattern, GreaterOrEqual, LessOrEqual


@dataclass
class User:
    username: Annotated[str, MinLength(3)]
    email: Annotated[str, Pattern(r"^[^\s@]+@[^\s@]+\.[^\s@]+$")]
    age: Annotated[int, GreaterOrEqual(18), LessOrEqual(120)]

valid = case(User, valid=True)
# -> {"username": "Abc", "email": "x@y.z", "age": 42}

With Pydantic

from pydantic import BaseModel, Field
from conformly import case

class User(BaseModel):
    username: str = Field(..., min_length=3, max_length=32)
    email: str = Field(..., pattern=r"^[^\s@]+@[^\s@]+\.[^\s@]+$")
    age: int = Field(..., ge=18, le=120)

valid = case(User, valid=True)
# -> {"username": "Abc", "email": "x@y.z", "age": 42}

API Reference

case(
    model,
    *,
    valid: bool,
    seed: int | None = None,
    strategy: str | None = None,
    allow_type_mismatch: bool = False
) -> dict

cases(
    model,
    *,
    valid: bool,
    seed: int | None = None,
    strategy: str = "all",
    count: int | None = None,
    allow_type_mismatch: bool = False,
    allow_structural_violations: bool = False
) -> list[dict]

strategy values:

• <field_name> - target specific field for invalidation (for nested fields using dot syntax "profile.name")
• <field_name>::<violations> - target specific violation for field (syntax "profile.name::too_long")
• "random" - choose a random field/constraint to violate
• "all" - (for cases) produce all minimal invalid variations for the model
• "first" - violate the first constrained field (for case) or take the first N constrained fields (for cases)
• "all_violations" - generate one invalid case per every available violations including constraints, structural and type violations (ignores count)

Invalid Generation Contract

For case(Model, valid=False, strategy="<field>"):

• Violation priority: generator choose first violation type from allowed based on priority (Structural (Missing/Extra) > Type Mismatch > Semantic (Range/Pattern/Value))
• If allow_type_mismatch=True, the generator may substitute a type mismatch (e.g., string instead of int) in place of a semantic constraint violation for the targeted field

Type	Mismatch
String	Integer
Integer	String
Float	String
Boolean	String
Enum/Literal	Float

• If allow_structural_violations=True, generator may substitute field missing in place of any other violations (avaliable only with strategy="all")
• Exactly one field is targeted (the one specified by strategy).
• The generator will violate constraints for that field, making it invalid.
• If a field has multiple constraints, the violated constraint may be chosen by generator logic (not necessarily the one you expect).
• For numeric bounds, invalid values may violate the lower or upper bound (e.g., age > 120 or age < 18).
• For float bounds, invalid generation may produce inf when violating the upper boundary.

If you need deterministic control over which exact constraint to violate, that is not implemented in yet (see Roadmap).

Optional Fields and Defaults

• If a field is optional (Optional[T]), valid generation may produce None.
• If a field has a default value, valid generation returns the default.
• Invalid generation requires at least one constraint on the targeted field (raises ValueError otherwise).

Constraints

Supported constraints

Type	Constraint	Pydantic equivalent
String	MinLength(n)	min_length=n
String	MaxLength(n)	max_length=n
String	Pattern(regex)	pattern=regex
Numeric	GreaterThan(v)	gt=v
Numeric	GreaterOrEqual(v)	ge=v
Numeric	LessThan(v)	lt=v
Numeric	LessOrEqual(v)	le=v
Numeric	MultipleOf(v)	multitiple_of=v
Closed-set	OneOf(values)	Literal[...], Enum

Important: Pydantic's constr(), conint(), and functional validators are not interpreted as constraints. Use Field() parameters for constraint extraction.

Defining constraints

1) Annotated[..., Constraint(...)] (recommended)

You can use for both model types (dataclasses, Pydantic)

from typing import Annotated
from conformly.constraints import MinLength, GreaterOrEqual

username: Annotated[str, MinLength(3)]

2) Annotated[..., "k=v"] (shorthand string syntax)

title: Annotated[str, "min_length=5", "max_length=200"]

3) Field(...) (Pydantic only)

from pydantic import Field

username: str = Field(..., min_length=3)

4) field(metadata={...}) (dataclasses only)

from dataclasses import field

sku: str = field(metadata={"pattern": r"^[A-Z0-9]{8}$"})

All syntaxes are fully compatible within their respective frameworks.

Special String types

conformly provides semantic marker types for common string formats. These types enable realistic test data generation while maintaining type safety.

Available types

Type	Description	Pydantic type	Example output	Availiable constraints
Email	RFC 5322-compliant email address	EmailStr	[e-mail]	MinLength, MaxLength
IPv4	RFC 791-compliant IPv4 address (dotted-quad notation)	IPv4Address	192.0.2.1	-
IPv6	RFC 4291-compliant IPv6 address (hex groups with :: compression)	IPv6Address	2001:db8::1	-
IPvAny	Either IPv4 or IPv6 address (randomly selected at generation time)	IPvAnyAddress	fe80::1 or 10.0.0.1	-
Url	Generic URL with any RFC 3986-compliant scheme	https://example.com/path	MinLength, MaxLength
HttpUrl	URL with scheme restricted to http or https only	https://example.com/path	MinLength, MaxLength

Usage

from typing import Annotated
from conformly import Email, MinLength

# Basic usage
contact: Email

# With additional constraints
website: Annotated[URL, MinLength(20)]

# Pydantic models (auto-mapped)
from pydantic import BaseModel, EmailStr

class Config(BaseModel):
    admin_email: EmailStr  # Automatically uses Email generator

UUID support

Standard uuid.UUID fields are supported out of the box. Values are generated in canonical RFC 4122 v4 format (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, lowercase).

import uuid
from dataclasses import dataclass
from conformly import case

@dataclass
class Session:
    id: uuid.UUID
    user_id: uuid.UUID

valid = case(Session, valid=True)
# -> {"id": UUID("550e8400-e29b-41d4-a716-446655440000"), ...}

invalid = case(Session, valid=False, strategy="id::too_short")
# -> {"id": "550e8400-e29b-41d4-a716-44665544000"}  # TOO_SHORT
invalid = case(Session, valid=False, strategy="id::wrong_uuid_character")
# -> {"id": "550e8400-e29b-41d4-a71@-446655440000"} # WRONG_UUID_CHARACTER

Use Cases

API Testing

# Valid payloads for happy-path tests
for _ in range(100):
    payload = case(CreateUserRequest, valid=True)
    response = client.post("/users", json=payload)
    assert response.status_code == 201

# Invalid payloads for error handling tests
invalid = case(CreateUserRequest, valid=False, strategy="age")
response = client.post("/users", json=invalid)
assert response.status_code == 400

# As option create all possible invalid cases for payload in one only line
invalid_payloads = cases(CreateUserRequest, valid=False, strategy="all")
for payload in invalid_payloads:
    response = client.post("/users", json=payload)
    assert response.status_code == 400

Database Seeding

# Generate realistic test data respecting schema constraints
products = cases(Product, valid=True, count=1000)
db.insert_many("products", products)

Fuzzing & Property-Based Testing

Conformly is not a replacement of Hypothesis, but a complementary tool for schema-driven testing and negative case generation.

# Generate random invalid data to stress-test validation
for _ in range(500):
    invalid = case(Model, valid=False, strategy="random")
    assert validate(invalid) is False  # Should always reject

Nested Models

Conformly supports nested models represented as tree structures (e.g. dataclasses containing other dataclasses).

Cyclic references between models are not supported

Constraints defined on nested fields are discovered recursively and can be used for both valid and invalid data generation.

Model Declaration

from dataclasses import dataclass
from typing import Annotated

from conformly import MinLength, GreaterOrEqual, Pattern

@dataclass
class Profile:
    email: Annotated[str, Pattern(r"^[^\s@]+@[^\s@]+\.[^\s@]+$")]
    phone: Annotated[str, Pattern(r"^\+[1-9]\d{1,14}$")]


@dataclass
class User:
    name: Annotated[str, MinLength(3)]
    age: Annotated[int, GreaterOrEqual(18)]
    profile: Profile

Generation Example

from conformly import case

valid_data = case(User, valid=True)
print(valid_data)
# {
#     "name": "validname",
#     "age": 25,
#     "profile": {
#            "email": "[e-mail]",
#            "phone": "[phone]"
#         }
# }

invalid_data_by_field = case(User, valid=False, strategy="profile.email")
print(invalid_data_by_field)
# {
#     "name": "validname",
#     "age": 25,
#     "profile": {
#            "email": "nonemailstring",
#            "phone": "[phone]"
#         }
# }

Collections

List support (experimental)

conformly supports basic generation of list[T] where T is a constrained primitive or a nested model.

Example

from dataclasses import dataclass
from typing import Annotated
from conformly import case, MinLength

@dataclass
class Product:
    sku: str
    price: float

@dataclass
class Order:
    tags: list[str]                           # list of unconstrained strings
    codes: Annotated[list[str], MinLength(5)] # each element must be ≥5 chars
    items: list[Product]                      # list of nested models

valid = case(Order, valid=True)
# -> {"tags": ["abc"], "codes": ["ABCDE"], "items": [{"sku": "...", "price": 10.0}]}

invalid = case(Order, valid=False, strategy="codes")
# -> {"tags": [...], "codes": ["ab", "VALID"], "items": [...]}  # exactly one code violates min_length

Current limitations (v0.3.10)

• No min_items/max_items control (list length is always 1–3 elements)
• No nested collections (list[list[T]] not supported)
• Invalid generation targets one random element; specific index targeting not available yet

Development

Install dependencies:

uv sync

Run tests:

uv run -m pytest -q

Run with coverage:

uv run -m pytest --cov=conformly --cov-report=term-missing

Build & check package:

uv build
uv run -m twine check dist/*

Changelog

See CHANGELOG.md for release notes and migration guidance.

License

MIT — see LICENSE file for details

Contributing

Contributions welcome!

• Fork the repo
• Create a feature branch
• Add tests for new functionality
• Run uv run -m pytest and uv run -m ruff check .
• Submit a pull request