Arca Models

ETH Library Zurich · Data Archive · Domain Models

A collection of JSON schemas defining data models for digital archiving systems based on the OAIS (Open Archival Information System) reference model. These schemas provide standardized definitions for various components of a data archive.

Core Functionality

The project defines JSON schemas for key entities in a digital archiving workflow:

• Producer: Entity that provides data to be archived
• Deposit: Information about a submission to the archive
• SIP (Submission Information Package): Package of information submitted to the archive
• IntellectualEntity: Conceptual object being preserved
• Representation: Digital manifestation of an intellectual entity
• File: Individual digital file within a representation
• Fixity: Integrity information for digital files

All schemas are currently at version 0.1.0, indicating this is an early-stage project.

Project Status Notice

This project is in alpha and actively evolving.

• Stability: Bugs, breaking changes, or incomplete features may occur.
• Evolution: APIs and behaviors can change as we refine functionality.

Provided as‑is, with no guarantees on stability.

We appreciate your patience and feedback!

Setup/Installation

Prerequisites

• Java 21
• Maven
• Python 3.12
• Nix with flakes enabled (recommended)
• direnv for environment management (recommended)

Recommended Nix + Direnv Setup

We recommend using the fully automatic setup method using Nix Flakes and Direnv:

Prerequisites

• Nix package manager with flakes enabled
• direnv for environment management

Steps

1. Clone the repository
2. Allow direnv in the project directory:

   direnv allow
   

This will automatically:

• Create a Python 3.12 virtual environment in .venv
• Install all dependencies using UV package manager
• Set up the development environment

If you'd like to activate the environment manually without direnv:

nix develop

Development Workflow

Python Model Generation

Python models are automatically generated from JSON schemas:

# Generate Python models from JSON schemas
datamodel-codegen \
  --input-file-type jsonschema \
  --input schemas/arca/ \
  --output src/arca_models/ \
  --output-model-type pydantic_v2.BaseModel \
  --field-constraints \
  --use-schema-description

The generated models use Pydantic v2 and are stored in the src/arca_models/ directory.

JSON Schema Best Practices Review

For an in‑depth analysis of the current JSON Schemas, recommended improvements, and CI/CD pipeline suggestions related to schema validation and code generation, see:

• docs/json-schema-review.md

Java Class Generation

Java classes are automatically generated from JSON schemas using the jsonschema2pojo Maven plugin:

<!-- Plugin configuration in pom.xml -->
<plugin>
    <groupId>org.jsonschema2pojo</groupId>
    <artifactId>jsonschema2pojo-maven-plugin</artifactId>
    <version>1.2.1</version>
    <configuration>
        <sourceDirectory>${project.basedir}/schemas/arca</sourceDirectory>
        <outputDirectory>${project.build.directory}/generated-sources/</outputDirectory>
        <targetPackage>ch.ethz.library.arca.model</targetPackage>
        <excludes>
            <exclude>_shared/**</exclude>
            <exclude>catalog.json</exclude>
        </excludes>
    </configuration>
</plugin>

The plugin is executed during the Maven prepare-package phase and generates Java classes from the JSON schemas in the schemas/arca/ directory. The generated classes are stored in the target/generated-sources/ directory under the package ch.ethz.library.arca.model.

To generate the Java classes, you can use one of the Maven commands listed in the Java Build Options section.

Dependency Management

The project uses uv for Python dependency management:

# Generate lock file
uv lock

# Install dependencies
uv sync

# Build Python package
uv build

Java Build Options

The project provides several Maven build commands:

1. Validate JSON schemas only:

   mvn -Dtest=JsonSchemaValidationTest test
   

2. Generate Java classes without validation (skip tests):

   mvn prepare-package -DskipTests
   

3. Standard build (validate schemas then generate classes):

   mvn package
   

Validation

Validate JSON Schemas

Validate that all JSON schemas are syntactically correct and comply with JSON Schema Draft 2020-12:

check-jsonschema --check-metaschema schemas/arca/*.json

This command validates:

• Schema syntax and structure
• Compliance with JSON Schema specification
• Cross-references between schemas ($ref resolution)

Validate Generated Pydantic Models

After generating Python models, validate them for type correctness:

# Type-check with mypy
mypy src/arca_models/ --ignore-missing-imports

# Lint with ruff (faster alternative)
ruff check src/arca_models/

This validates:

• Python syntax correctness
• Type annotations
• Import resolution (model cross-references)
• Pydantic model structure

Data Model

Class Diagram

The following class diagram visualizes the relationships between the different entities in the data archive model. The diagram is automatically generated from the Pydantic models in the codebase.

classDiagram
    class Schema {
        +str name
        +str version
        +AnyUrl url
    }
    class DataArchiveModelCatalog {
        %% Catalog of all schemas in the Data Archive Model
        +Sequence[Schema] schemas
    }
    class Checksum {
        %% A checksum value with its algorithm
        +Algorithm algorithm
        +str value
    }
    class Deposit {
        %% An OAIS Deposit entity representing a submission from a Producer
        +UUID identifier
        +datetime dateCreated
        +Producer producer
        +str name
        +Optional[Status] status
        +List[SIP.SubmissionInformationPackage] sips
        +DepositMetadataFile metadataFile
    }
    class SubmissionInformationPackage {
        %% An OAIS Submission Information Package (SIP) entity
        +UUID identifier
        +str name
        +Producer producer
        +List[IntellectualEntity] intellectualEntities
        +Optional[State] state
    }
    class Producer {
        %% An OAIS Producer entity that creates and submits content
        +UUID identifier
        +datetime dateCreated
        +str name
        +Optional[str] contact
    }
    class IEMetadataFile {
        %% An external metadata file accompanying an intellectual entity
        +str path
        +Format format
    }
    class File {
        %% An OAIS File entity representing a digital file
        +UUID identifier
        +Optional[datetime] dateCreated
        +Optional[str] name
        +str path
        +List[Fixity] fixities
    }
    class DepositMetadataFile {
        %% An external metadata file accompanying a deposit
        +str path
        +Format format
    }
    class Fixity {
        %% An OAIS Fixity entity representing integrity information for a digital file
        +UUID identifier
        +List[Checksum] checksums
    }
    class Representation {
        %% An OAIS Representation entity representing a specific form of an Intellectual Entity
        +UUID identifier
        +str name
        +List[File] files
    }
    class IntellectualEntity {
        %% An OAIS Intellectual Entity representing a conceptual object
        +UUID identifier
        +List[Representation] representations
        +IEMetadataFile metadataFile
    }
    Deposit *-- File : contains
    Representation *-- File : contains many
    IntellectualEntity *-- File : contains
    Deposit *-- SubmissionInformationPackage : contains many
    SubmissionInformationPackage *-- Producer : contains
    IntellectualEntity *-- Representation : contains many
    Deposit *-- DepositMetadataFile : contains
    SubmissionInformationPackage *-- IntellectualEntity : contains many
    Fixity *-- Checksum : contains many
    File *-- Fixity : contains many
    Deposit *-- Producer : contains
    IntellectualEntity *-- IEMetadataFile : contains
    Deposit *-- Deposit : contains
    DataArchiveModelCatalog *-- Schema : contains

Generating the Diagram

The class diagram is generated using Python scripts located in the scripts directory:

• scripts/generate_mermaid_diagram.py: Generates a Mermaid class diagram from Pydantic models in the arca_models package
• scripts/update_readme.py: Updates this README.md file with the generated diagram

To update the diagram:

python scripts/update_readme.py

This will:

1. Scan the Pydantic models in the src/arca_models directory
2. Generate a Mermaid class diagram
3. Update the diagram in this README.md file between the marker comments

Continuous Integration

The project uses GitHub Actions for CI. The workflow automatically:

• Sets up the Nix environment shell
• Implements Maven dependency caching for all jobs
• Runs schema validation tests
• Generates Java classes from JSON schemas
• Generates Python models from JSON schemas
• Publishes Java artifacts to GitHub Packages
• Publishes Python packages to TestPyPI