energyml-utils

Installation

energyml-utils can be installed with pip :

pip install energyml-utils

or with poetry:

poetry add energyml-utils

Features

Supported packages versions

This package supports read/write in xml/json the following packages :

• EML (common) : 2.0, 2.1, 2.2, 2.3
• RESQML : 2.0.1, 2.2dev3, 2.2
• WITSMl : 2.0, 2.1
• PRODML : 2.0, 2.2

/!\ By default, these packages are not installed and are published independently. You can install only the versions you need by adding the following lines in the .toml file :

energyml-common2-0 = "^1.12.0"
energyml-common2-1 = "^1.12.0"
energyml-common2-2 = "^1.12.0"
energyml-common2-3 = "^1.12.0"
energyml-resqml2-0-1 = "^1.12.0"
energyml-resqml2-2-dev3 = "^1.12.0"
energyml-resqml2-2 = "^1.12.0"
energyml-witsml2-0 = "^1.12.0"
energyml-witsml2-1 = "^1.12.0"
energyml-prodml2-0 = "^1.12.0"
energyml-prodml2-2 = "^1.12.0"

Content of the package :

• Support EPC + h5 read and write
  • .rels files are automatically generated, but it is possible to add custom Relations.
  • You can add "raw files" such as PDF or anything else, in your EPC instance, and it will be package with other files in the ".epc" file when you call the "export" function.
  • You can work with local files, but also with IO (BytesIO). This is usefull to work with cloud application to avoid local storage.
• Supports xml / json read and write (for energyml objects)
• Work in progress : Supports the read of 3D data inside the "AbstractMesh" class (and sub-classes "PointSetMesh", "PolylineSetMesh", "SurfaceMesh"). This gives you a instance containing a list of point and a list of indices to easily re-create a 3D representation of the data.
  • These "mesh" classes provides .obj, .off, and .geojson export.
• Introspection : This package includes functions to ease the access of specific values inside energyml objects.
  • Functions to access to UUID, object Version, and more generic functions for any other attributes with regex like ".Citation.Title" or "Cit\.*.Title" (regular dots are used as in python object attribute access. To use dot in regex, you must escape them with a '\')
  • Functions to parse, or generate from an energyml object the "ContentType" or "QualifiedType"
  • Generation of random data : you can generate random values for a specific energyml object. For example, you can generate a WITSML Tubular object with random values in it.
• Objects correctness validation :
  • You can verify if your objects are valid following the energyml norm (a check is done on regex contraint attributes, maxCount, minCount, mandatory etc...)
  • The DOR validation is tested : check if the DOR has correct information (title, ContentType/QualifiedType, object version), and also if the referenced object exists in the context of the EPC instance (or a list of object).
• Abstractions done to ease use with ETP (Energistics Transfer Protocol) :
  • The "EnergymlWorkspace" class allows to abstract the access of numerical data like "ExternalArrays". This class can thus be extended to interact with ETP "GetDataArray" request etc...
• ETP URI support : the "Uri" class allows to parse/write an etp uri.

EPC Stream Reader

The EpcStreamReader provides memory-efficient handling of large EPC files through lazy loading and smart caching. Unlike the standard Epc class which loads all objects into memory, the stream reader loads objects on-demand, making it ideal for handling very large EPC files with thousands of objects.

Key Features

• Lazy Loading: Objects are loaded only when accessed, reducing memory footprint
• Smart Caching: LRU (Least Recently Used) cache with configurable size
• Automatic EPC Version Detection: Supports both CLASSIC and EXPANDED EPC formats
• Add/Remove/Update Operations: Full CRUD operations with automatic file structure maintenance
• Relationship Management: Automatic or manual .rels file updates with parallel processing support
• External Data Arrays: Read/write HDF5, Parquet, CSV arrays with intelligent file caching
• Context Management: Automatic resource cleanup with with statements
• Memory Monitoring: Track cache efficiency and memory usage statistics

Basic Usage

from energyml.utils.epc_stream import EpcStreamReader, RelsUpdateMode

# Open EPC file with context manager (recommended)
with EpcStreamReader('large_file.epc', 
                     cache_size=50,
                     rels_update_mode=RelsUpdateMode.UPDATE_ON_CLOSE) as reader:
    # List all objects without loading them
    print(f"Total objects: {len(reader)}")
    
    # Get object by identifier
    obj = reader.get_object("uuid.version")
    
    # List objects by type (returns metadata, not full objects)
    features = reader.list_objects(object_type="BoundaryFeature")
    print(f"Found {len(features)} features")
    
    # Get all objects with same UUID
    versions = reader.get_object_by_uuid("12345678-1234-1234-1234-123456789abc")

Adding Objects

from energyml.utils.epc_stream import EpcStreamReader
from energyml.utils.constants import gen_uuid
import energyml.resqml.v2_2.resqmlv2 as resqml
import energyml.eml.v2_3.commonv2 as eml

# Create a new EnergyML object
boundary_feature = resqml.BoundaryFeature()
boundary_feature.uuid = gen_uuid()
boundary_feature.citation = eml.Citation(title="My Feature")

with EpcStreamReader('my_file.epc') as reader:
    # Add object - path is automatically generated based on EPC version
    identifier = reader.add_object(boundary_feature)
    print(f"Added object with identifier: {identifier}")
    
    # Or specify custom path (optional)
    identifier = reader.add_object(boundary_feature, "custom/path/MyFeature.xml")

Removing Objects

with EpcStreamReader('my_file.epc') as reader:
    # Remove by full identifier
    success = reader.delete_object("uuid.version")
    
    # Or use the alias
    success = reader.remove_object("uuid.version")
    
    if success:
        print("Object removed successfully")

Updating Objects

from energyml.utils.epc_stream import EpcStreamReader
from energyml.utils.introspection import set_attribute_from_path

with EpcStreamReader('my_file.epc') as reader:
    # Get existing object
    obj = reader.get_object("uuid.version")
    
    # Modify the object
    set_attribute_from_path(obj, "citation.title", "Updated Title")
    
    # Update in EPC file
    new_identifier = reader.put_object(obj)
    print(f"Updated object: {new_identifier}")

Performance Monitoring

with EpcStreamReader('large_file.epc', cache_size=100) as reader:
    # Access some objects...
    for i in range(10):
        obj = reader.get_object_by_identifier(f"uuid-{i}.1")
    
    # Check performance statistics
    print(f"Cache hit rate: {reader.stats.cache_hit_rate:.1f}%")
    print(f"Memory efficiency: {reader.stats.memory_efficiency:.1f}%") 
    print(f"Objects in cache: {reader.stats.loaded_objects}/{reader.stats.total_objects}")

EPC Version Support

The EpcStreamReader automatically detects and handles both EPC packaging formats:

• CLASSIC Format: Flat file structure (e.g., obj_BoundaryFeature_{uuid}.xml)
• EXPANDED Format: Namespace structure (e.g., namespace_resqml201/version_{id}/obj_BoundaryFeature_{uuid}.xml or namespace_resqml201/obj_BoundaryFeature_{uuid}.xml)

with EpcStreamReader('my_file.epc') as reader:
    print(f"Detected EPC version: {reader.export_version}")
    # Objects added will use the same format as the existing EPC file

Relationship Management

from energyml.utils.epc_stream import EpcStreamReader, RelsUpdateMode

# Choose relationship update strategy
with EpcStreamReader('my_file.epc', 
                     rels_update_mode=RelsUpdateMode.UPDATE_ON_CLOSE,
                     enable_parallel_rels=True) as reader:
    
    # Add/modify objects - rels updated automatically based on mode
    reader.add_object(my_object)
    
    # Manual rebuild of all relationships (e.g., after bulk operations)
    stats = reader.rebuild_all_rels(clean_first=True)
    print(f"Rebuilt {stats['rels_files_created']} .rels files")

External Data Arrays

import numpy as np

with EpcStreamReader('my_file.epc') as reader:
    # Read array from HDF5/Parquet/CSV
    data = reader.read_array(
        proxy=my_representation,
        path_in_external="/geometry/points"
    )
    
    # Write array to external file
    new_data = np.array([[1, 2, 3], [4, 5, 6]])
    success = reader.write_array(
        proxy=my_representation,
        path_in_external="/geometry/points",
        array=new_data
    )
    
    # Get metadata without loading full array
    metadata = reader.get_array_metadata(my_representation)
    print(f"Array shape: {metadata.dimensions}, dtype: {metadata.array_type}")

Advanced Usage

# Initialize with persistent ZIP connection for better performance
reader = EpcStreamReader('huge_file.epc', 
                         keep_open=True,
                         cache_size=200,
                         enable_parallel_rels=True,
                         parallel_worker_ratio=10)

try:
    # Get object dependencies
    deps = reader.get_object_dependencies("uuid.version")
    
    # Batch processing with memory monitoring
    for obj_type in ["BoundaryFeature", "PropertyKind"]:
        obj_list = reader.list_objects(object_type=obj_type)
        print(f"Processing {len(obj_list)} {obj_type} objects")
        
        for metadata in obj_list:
            obj = reader.get_object(metadata.identifier)
            # Process object...
        
finally:
    reader.close()  # Manual cleanup if not using context manager

The EpcStreamReader is perfect for applications that need to work with large EPC files efficiently, such as data processing pipelines, web applications, or analysis tools where memory usage is a concern.

Command line scripts :

Installing the package (pip install energyml-utils) creates these executables. They live in energyml.utils.cli, so they work from an installed wheel as well as from a checkout.

• extract_3d : extract a representation into a 3D / GIS file (obj/off/stl/vtk/geojson)
• csv_to_dataset : translate csv data into h5 or parquet datasets (needs the parquet extra for the csv reader)
• generate_data : generate a random object from a qualified_type
• generate_multiple_data : same, for several types at once, optionally one file per object
• xml_to_json : translate an energyml xml file (or every object of an EPC) into json
• json_to_xml : translate an energyml json file into one xml file per object
• json_to_epc : package every object of an energyml json file into a single EPC
• loadNsave : read a file or a folder (json/xml/epc) and write it back as an EPC
• describe_as_csv : create a csv description of an EPC content
• validate : validate an energyml object or an EPC instance (or a folder containing energyml objects)

Every command accepts --help, and -v / -vv to raise the log level (-q to only report errors). They can also be called from python, passing the arguments explicitly:

from energyml.utils.cli import extract_representation_in_3d_file

extract_representation_in_3d_file(["--epc", "file.epc", "--output", "out", "-ff", "geojson"])

Installation to test poetry scripts :

poetry install

if you fail to run a script, you may have to add "src" to your PYTHONPATH environment variable. For example, in powershell :

$env:PYTHONPATH="src"

Poetry Script Examples :

Validation

Validate an EPC file:

poetry run validate --file "path/to/your/energyml/object.epc" *> output_logs.json

Validate an XML file:

poetry run validate --file "path/to/your/energyml/object.xml" *> output_logs.json

Validate a JSON file:

poetry run validate --file "path/to/your/energyml/object.json" *> output_logs.json

Validate a folder containing EPC/XML/JSON files:

poetry run validate --file "path/to/your/folder" *> output_logs.json

Ignore specific error types (e.g., INFO):

poetry run validate --file "path/to/file.epc" --ignore-err-type INFO *> output_logs.json

Group errors by their class for better organization:

poetry run validate --file "path/to/file.epc" --group-by-err-class *> output_logs.json

Include PRODML version errors in validation (by default they are ignored):

poetry run validate --file "path/to/file.epc" --ignore-prodml-version-errs *> output_logs.json

Combined example with multiple options:

poetry run validate --file "path/to/file.epc" -i INFO WARNING --group-by-err-class *> output_logs.json

Extract 3D Representations

Extract all representations from an EPC to OBJ files:

poetry run extract_3d --epc "path/to/file.epc" --output "output_folder"

Extract specific representations by UUID:

poetry run extract_3d --epc "path/to/file.epc" --output "output_folder" --uuid "uuid1" "uuid2"

Extract to OFF format without CRS displacement:

poetry run extract_3d --epc "path/to/file.epc" --output "output_folder" --file-format OFF --no-crs

Extract 3D Representations as GeoJSON

Export every exportable representation of the EPC to GeoJSON (one .geojson file per representation; a representation that cannot be read is logged and skipped):

poetry run extract_3d --epc "path/to/file.epc" --output "output_folder" --file-format geojson

Export only some representations:

poetry run extract_3d --epc "path/to/file.epc" --output "output_folder" --file-format geojson --uuid "uuid1" "uuid2"

Coordinates are reprojected to WGS84 by default, as required by RFC 7946. This needs the crs extra:

poetry install --extras crs        # or : pip install energyml-utils[crs]

Without it (or when no EPSG code can be found in the CRS), a warning is logged, the coordinates are left in their source CRS, and that CRS is advertised in the output through the crs (GeoJSON 2008, read by GDAL / QGIS) and coordRefSys (OGC JSON-FG) members.

Keep the coordinates in the source projected CRS:

poetry run extract_3d --epc "path/to/file.epc" --output "output_folder" --file-format geojson --no-wgs84

Allow PROJ to download the geoid grids used by the vertical datum transformation. Without them the height conversion is silently skipped, which can be off by tens of metres:

poetry run extract_3d --epc "path/to/file.epc" --output "output_folder" --file-format geojson --proj-network

Each feature carries the identification metadata of its source object: the energyml uuid in the RFC 7946 id member, and the uuid, qualified_type, content_type, ETP uri, Citation fields (title, originator, creation, last_update, …) and EPSG codes in properties:

{
  "type": "FeatureCollection",
  "name": "Bartonien Top",
  "bbox": [2.3675, 48.9129, 30.13, 2.4057, 48.9198, 37.15],
  "features": [
    {
      "type": "Feature",
      "id": "02cc9411-6b90-4619-a9fd-a39ac332b367",
      "properties": {
        "uuid": "02cc9411-6b90-4619-a9fd-a39ac332b367",
        "qualified_type": "resqml22.PointSetRepresentation",
        "uri": "eml:///resqml22.PointSetRepresentation(02cc9411-6b90-4619-a9fd-a39ac332b367)",
        "title": "Bartonien Top",
        "originator": "Geosiris",
        "creation": "2025-12-17T16:11:36Z",
        "last_update": "2025-12-17T16:11:36Z",
        "projected_epsg_code": 3949,
        "source_crs": "EPSG:3949",
        "coordinates_crs": "OGC:CRS84"
      },
      "geometry": { "type": "MultiPoint", "coordinates": [[2.4055336, 48.9140288, 37.15]] }
    }
  ]
}

The same options are available from python:

from energyml.utils.data.mesh import MeshFileFormat, export_multiple_data

export_multiple_data(
    epc_path="path/to/file.epc",
    uuid_list=["uuid1"],
    output_folder_path="output_folder",
    file_format=MeshFileFormat.GEOJSON,
    to_wgs84=True,       # default
    use_network=False,   # True to download the geoid grids
)

CSV to Dataset

Convert CSV to HDF5:

poetry run csv_to_dataset --csv "data.csv" --output "output.h5"

Convert CSV to Parquet with custom delimiter:

poetry run csv_to_dataset --csv "data.csv" --output "output.parquet" --csv-delimiter ";"

With dataset name prefix:

poetry run csv_to_dataset --csv "data.csv" --output "output.h5" --prefix "/my/path/"

With column mapping (JSON file):

poetry run csv_to_dataset --csv "data.csv" --output "output.h5" --mapping "mapping.json"

With inline column mapping:

poetry run csv_to_dataset --csv "data.csv" --output "output.h5" --mapping-line '{"DATASET_A": ["COL1", "COL2"], "DATASET_B": ["COL3"]}'

Generate Random Data

Generate a random RESQML object in JSON:

poetry run generate_data --type "energyml.resqml.v2_2.resqmlv2.TriangulatedSetRepresentation" --file-format json

Generate a random object in XML:

poetry run generate_data --type "energyml.resqml.v2_0_1.resqmlv2.Grid2dRepresentation" --file-format xml

Using qualified type:

poetry run generate_data --type "resqml22.WellboreFeature" --file-format json

Generate multiple data :

poetry run generate_multiple_data -o generated -ff xml -t eml23.AbstractObject --exclude witsml --exclude prodml

XML to JSON Conversion

Convert an XML file to JSON:

poetry run xml_to_json --file "path/to/object.xml"

Convert with custom output path:

poetry run xml_to_json --file "path/to/object.xml" --out "output.json"

Convert entire EPC to JSON array:

poetry run xml_to_json --file "path/to/file.epc" --out "output.json"

JSON to XML Conversion

Convert a JSON file to XML:

poetry run json_to_xml --file "path/to/object.json"

Convert with custom output directory:

poetry run json_to_xml --file "path/to/object.json" --out "output_folder/"

Describe as CSV

Generate a CSV description of all objects in a folder:

poetry run describe_as_csv --folder "path/to/folder"

With custom columns:

poetry run describe_as_csv --folder "path/to/folder" \
  --columnsNames "Title" "Type" "UUID" \
  --columnsValues "citation.title" "$qualifiedType" "Uuid"

Available special values for columnsValues:

• $type: Object Python type
• $qualifiedType: EnergyML qualified type
• $contentType: EnergyML content type
• $path: File path
• $dor: UUIDs of referenced objects