sysnet-sekm-migrate 0.2.0

SEKM migration utility: Elasticsearch → MongoDB/Beanie

sysnet-sekm-migrate

Migration utility for SEKM — transfers data from Elasticsearch indexes into MongoDB documents managed by Beanie and sysnet-docversion.

Depends on sysnet-sekm-model for the domain and persistence models.

Installation

pip install sysnet-sekm-migrate

Requirements

• Python ≥ 3.12
• Access to the source Elasticsearch cluster
• Access to a target MongoDB instance
• .env file with both ES and Mongo credentials (see below)

Configuration

Create a .env file (or export environment variables):

ELK_URI=https://your-es-host:9200
ELK_USER=elastic
ELK_PASSWORD=...

MONGO_HOST=localhost
MONGO_PORT=27017
MONGO_USERNAME=...
MONGO_PASSWORD=...
MONGO_DB=sekm

CLI usage

After installation the sekm-migrate command is available:

# Migrate everything (all 21 entities)
sekm-migrate

# Single entity
sekm-migrate --entity area-source

# Multiple entities
sekm-migrate --entity area-source --entity user

# Dry run — reads ES, validates, but writes nothing to MongoDB
sekm-migrate --entity area-source --dry-run

# Skip archive version history (faster)
sekm-migrate --entity as-object --no-history

# Limit documents (trial run)
sekm-migrate --entity sample --limit 10000

# List all 21 entity names
sekm-migrate --list-entities

# Verbose logging
sekm-migrate --log-level DEBUG

Programmatic usage

from sysnet_sekm_model.config import MongoSettings
from sysnet_sekm_model.db import init_mongo, close_mongo
from sysnet_sekm_migrate.services.elasticsearch import es_client
from sysnet_sekm_migrate.services.migration import migrate_all

settings = MongoSettings()
client = await init_mongo(settings.mongo_uri, settings.mongo_db)
try:
    async with es_client() as es:
        results = await migrate_all(
            es,
            entity_names=["area-source", "user"],  # None = all entities
            import_history=True,
            dry_run=False,
        )
finally:
    await close_mongo(client)

for entity, result in results.items():
    result.log_summary()

Supported entities

Entity	Collection	Versioned	~Documents
area-source	area_sources	Yes	29 000
as-object	as_objects	Yes	177 000
as-region	as_regions	Yes	2 800
as-document	as_documents	Yes	17 000
observation	observations	Yes	19 000
user	users	Yes	9 200
organisation	organisations	Yes	308
template	templates	Yes	6 500
dictionary	dictionaries	Yes	39
substance	substances	Yes	365
sample	samples	No	~15 300 000
sample-advanced	samples_advanced	No	~13 600 000
history	history_events	No	287 000
lock	lock_events	No	106 000
approval	approvals	No	302
region	regions	No	14
district	districts	No	77
municipality	municipalities	No	6 300
orp-district	orp_districts	No	206
pou-district	pou_districts	No	394
registry	registry_entries	No	13 000

Migration is idempotent — it can be run multiple times against the same MongoDB database. Already-migrated documents are detected and skipped.

Documentation

Document	Contents
Migration Guide	Full pipeline description, CLI reference, transformation details
Data Model	Domain and persistence model reference
Trial Migration Report	Trial migration results and entity volumes
Data Quality Report	Post-migration data quality findings and remediation

License

AGPL-3.0