SSB POC Statlog Model

This repo is a proof of concept (POC) of models for logging data from a statistics production run.

Features

• Contains json schema models for the data to be logged.
• Contains pydantic models, that is python data validation classes for the json schemas.

Requirements

• Python >= 3.13

Installation

You can install SSB POC Statlog Model via pip from PyPI:

pip install ssb-poc-statlog-model

Usage

Please see the Reference Guide for API details. A quick example using the generated ChangeDataLog model:

from datetime import datetime, timezone
from ssb_poc_statlog_model.change_data_log import ChangeDataLog, DataChangeType

change = ChangeDataLog(
    statistics_name="arblonn",
    data_source=["gs://ssb-prod-superteam-data-produkt/arblonn/inndata/arbeidloenn_p2023-12_v1.parquet"],
    data_target="gs://ssb-prod-superteam-data-produkt/arblonn/klargjorte-data/arbeidloenn_p2023-12_v1.parquet",
    data_period="2023-12",
    change_event="A",
    change_event_reason="OTHER_SOURCE",
    change_datetime = datetime(2024, 1, 10, 15, 0, tzinfo=timezone.utc),
    changed_by="user@example.com",
    data_change_type=DataChangeType.NEW,
    change_comment="Opprettet ny enhet (person) fra ny datakilde ...",
    change_details={
        "detail_type": "unit",
        "unit_id": [
            {"unit_id_variable": "fnr", "unit_id_value": "170598nnnnn"},
            {"unit_id_variable": "orgnr", "unit_id_value": "123456789"}
        ],
        "new_value": [
            {"variable_name": "bostedskommune", "value": "0101"},
            {"variable_name": "type_loenn", "value": "time"},
            {"variable_name": "loenn", "value": "38000"},
            {"variable_name": "overtid_loenn", "value": "3000"}
        ]
    }
)

print(change.model_dump_json())

Tip about timestamps in JSON: use ISO 8601 with timezone information (e.g. …Z for UTC) to satisfy Pydantic’s AwareDatetime requirement used in several models.

Project structure

• src/model → JSON Schemas for the domain models (source of truth)
  • example_logs/*.json → Example payloads used in tests
• src/ssb_poc_statlog_model → Generated Pydantic models (Python)
• src/scripts → Scripts for generating examples from the pydantic models
• tests → Pytest suite validating models and examples

Schema versioning

The schema version is defined in the schema_version and $id field in the root of each schema. It uses semantic versioning (major.minor.patch) with the following rules:

• Major (breaking change):
  • Field removed or renamed
  • Field type changed
  • Required field added
• Minor (backward compatible):
  • New optional field added
  • Enum extended
• Patch (non-structural change):
  • Description changes

Development

Set up the environment (installs runtime + dev tools):

poetry install

Run tests:

poetry run pytest -v

Code style and quality:

poetry run pre-commit run --all-files

Regenerate the Pydantic models from JSON Schemas

This repository keeps the source of truth for the models as JSON Schema files in src\model. Python classes are generated into src\ssb_poc_statlog_model using datamodel-code-generator via a small helper CLI.

You can run the generator using the console script (defined in pyproject.toml). All examples assume you are in the project root.

# Ensure dev dependencies are available (only needed once)
poetry install

# Generate models for all *-json-schema.json files under src/model
poetry run generate-ssb-models

Useful options:

• Generate a single schema only:

poetry run generate-ssb-models --schemas src/model/change-data-log-json-schema.json

• Use explicit directories (defaults shown):

poetry run generate-ssb-models \
  --schemas-dir src/model \
  --out-dir src/ssb_poc_statlog_model

• Forward extra flags directly to datamodel-code-generator (repeatable):

poetry run generate-ssb-models \
  --extra-arg --collapse-root \
  --extra-arg --use-schema-description

What the helper does under the hood:

• Discovers *-json-schema.json files in src/model (or uses --schemas if given)
• Runs datamodel-code-generator targeting Pydantic v2 with options compatible with Python 3.10+ (see src/ssb_poc_statlog_model/generate_python.py for the exact flags)
• Writes the generated models into src/ssb_poc_statlog_model

After regenerating, commit the updated Python files to version control.

Contributing

Contributions are very welcome. To learn more, see the Contributor Guide.

License

Distributed under the terms of the MIT license, SSB POC Statlog Model is free and open source software.

Issues

If you encounter any problems, please file an issue along with a detailed description.

Credits

This project was generated from Statistics Norway's SSB PyPI Template.