sysnet-sekm-migrate

Migration utility for SEKM — transfers data from Elasticsearch indexes into MongoDB documents managed by Beanie and sysnet-docversion.

Depends on sysnet-sekm-model for the domain and persistence models.

Installation

pip install sysnet-sekm-migrate

Requirements

• Python ≥ 3.12
• Access to the source Elasticsearch cluster
• Access to a target MongoDB instance
• .env file with both ES and Mongo credentials (see below)

Configuration

Create a .env file (or export environment variables):

ELK_URI=https://your-es-host:9200
ELK_USER=elastic
ELK_PASSWORD=...

MONGO_HOST=localhost
MONGO_PORT=27017
MONGO_USERNAME=...
MONGO_PASSWORD=...
MONGO_DB=sekm

CLI usage

After installation the sekm-migrate command is available:

# Migrate everything (all 21 entities)
sekm-migrate

# Single entity
sekm-migrate --entity area-source

# Multiple entities
sekm-migrate --entity area-source --entity user

# Dry run — reads ES, validates, but writes nothing to MongoDB
sekm-migrate --entity area-source --dry-run

# Skip archive version history (faster)
sekm-migrate --entity as-object --no-history

# Limit documents (trial run)
sekm-migrate --entity sample --limit 10000

# List all 21 entity names
sekm-migrate --list-entities

# Verbose logging
sekm-migrate --log-level DEBUG

Programmatic usage

from sysnet_sekm_model.config import MongoSettings
from sysnet_sekm_model.db import init_mongo, close_mongo
from sysnet_sekm_migrate.services.elasticsearch import es_client
from sysnet_sekm_migrate.services.migration import migrate_all

settings = MongoSettings()
client = await init_mongo(settings.mongo_uri, settings.mongo_db)
try:
    async with es_client() as es:
        results = await migrate_all(
            es,
            entity_names=["area-source", "user"],  # None = all entities
            import_history=True,
            dry_run=False,
        )
finally:
    await close_mongo(client)

for entity, result in results.items():
    result.log_summary()

Supported entities

Migration is idempotent — it can be run multiple times against the same MongoDB database. Already-migrated documents are detected and skipped.

Documentation

License

AGPL-3.0