📝 avro-datagen

Schema-driven fake data generator for Avro schemas. Reads .avsc files with arg.properties hints and produces realistic records. Includes Faker for names, emails, addresses, and more.

avro-datagen generates data -- nothing more. Pipe JSON output to Kafka, databases, files, or any other sink using the tools you already have.

Install

pip install avro-datagen                 # CLI + library + Faker
pip install "avro-datagen[ui]"           # + Streamlit web UI
pip install "avro-datagen[app]"          # + UI + Kafka producer (for interactive testing)

Web UI

pip install "avro-datagen[ui]"
avro-datagen ui

Opens a Streamlit dashboard at localhost:8501 with a bundled example schema. Browse schemas, edit them live, preview generated data, and download samples.

The UI creates a schemas/ folder in your working directory when you save a schema. Point --schema-dir at an existing folder to use your own schemas.

avro-datagen ui --schema-dir ./my-schemas      # use your own schemas
avro-datagen ui --port 3000                    # custom port
avro-datagen ui --kafka                        # enable Kafka producer section

CLI

# Create a schema file
cat > order.avsc << 'EOF'
{
  "type": "record",
  "name": "Order",
  "fields": [
    { "name": "id", "type": { "type": "string", "logicalType": "uuid" } },
    { "name": "amount", "type": "double", "arg.properties": { "range": { "min": 5, "max": 500 } } },
    { "name": "customer", "type": "string", "arg.properties": { "faker": "name" } }
  ]
}
EOF

# Generate 10 records
avro-datagen -s order.avsc

# Pretty-print, seeded for reproducibility
avro-datagen -s order.avsc -c 5 --seed 42 --pretty

# Rate-limited, infinite
avro-datagen -s order.avsc -c 0 --rate 10

# ISO-8601 timestamps instead of the Avro wire encoding, for pasting into
# a Kafka UI produce form
avro-datagen -s order.avsc -c 5 --json-format human

Library

from avro_datagen import generate

for record in generate("order.avsc", count=100):
    print(record)

# Deterministic output
records = list(generate("order.avsc", count=10, seed=42))

Sinks and integrations

avro-datagen does not bundle integrations for databases, cloud storage, or message queues. The CLI emits JSON lines to stdout -- pipe it anywhere:

avro-datagen -s schema.avsc -c 1000 | kcat -b localhost:9092 -t topic   # Kafka
avro-datagen -s schema.avsc -c 1000 > data.jsonl                        # File
avro-datagen -s schema.avsc -c 1000 | psql -c "COPY t FROM STDIN"       # Postgres

The Kafka producer in the UI (--kafka) is a convenience for interactive testing, not a production integration.

Development

git clone https://github.com/ConsciousExplorer/avro-datagen.git
cd avro-datagen

With uv (recommended):

uv sync --all-extras

make test       # run tests
make check      # lint + typecheck + tests
make app        # streamlit UI
make docs       # mkdocs dev server

With pip:

python3 -m venv .venv && source .venv/bin/activate
pip3 install -e ".[dev,ui]"
make test

Documentation

Full documentation: consciousexplorer.github.io/avro-datagen

License

MIT