soil-etl 0.1.0a2

YAML-driven ETL mappers for ForSITE soil database imports.

SOIL - Structured Observation Ingestion Library

YAML-driven ETL mapper package for importing ForSITE soil datasets into the ForSITE PostGIS database model.

This package is being split out of forsite-soil-db-interface so the mapper layer can be versioned, tested, and published independently on PyPI.

Acknowledgements

Project Partners

This project was developed through joint funding provided by Agriculture and Agri-Food Canada (AAFC) under the CSBO program and Natural Resources Canada (NRCan) for the ForSITE-Soil degradation project.

The code developed as part of this project includes contributions from:

• Olivier Forgues-Nhan
• Xavier Malet
• Catlan Dallaire
• David Gagné

Target Public API

Basic Usage of the YAML Mapper

from pathlib import Path

from soil_etl.yaml_mapper import YamlDatasetMapper

mapper = YamlDatasetMapper(
    config_path=Path("examples/configs/example_config.yaml"),
    data_file_path=Path("data.xlsx"),
)
mapper.import_data()

For more details on the YAML format, refer to the YAML Mapper documentation.

Database Connection Configuration

Database Connection String

Database connection details can be provided when running the import. By default, the mapper still uses the existing environment/config lookup, but a call can now override it explicitly:

mapper.import_data(
    database_connection_string="postgresql+psycopg2://user:password@host:5432/database",
    database_engine_kwargs={"pool_size": 10, "pool_pre_ping": True},
)

Hydra Database Configuration Files

When using Hydra database configuration files instead of a direct connection string, pass the config directory and config name:

mapper.import_data(
    database_config_path="configs",
    database_config_name="configs.yaml",
)

The config directory can also be provided through the CONFIG_PATH environment variable:

export CONFIG_PATH=configs
export DB_USER=user
export DB_PASSWORD=password

and then called in the import:

mapper.import_data(database_config_name="configs.yaml")

or entierly in python:

import os

os.environ['CONFIG_PATH'] = "./path_to_config_dir"
os.environ['DB_USER'] = "user"
os.environ['DB_PASSWORD'] = "password"
mapper.import_data(database_config_name="configs.yaml")

YAML Mapper Configuration examples

The YAML mapper configuration is a YAML file that defines the mapping between the data file columns and the database table columns. Detailled documentation of the YAML format is available in README_yaml_format_en.md.

YAML mapper configuration examples are available in examples/configs/example_config.yaml. Packaged reference files are also kept in src/soil_etl/yaml_mapper/template.yaml and src/soil_etl/yaml_mapper/examples/on_master.yaml.

Package Status

The package is in alpha and the extractor implementation has been copied into src/soil_etl. The current tree includes the YAML mapper, binding helpers, dataclasses, enums, staging models, database interface, and ORM models needed by the migrated tests.

The remaining packaging work is to finish aligning the public import surface, tests, and documentation with the final package namespace. Some older project notes may still refer to the previous forsite_soil_extractors migration target.

Development

poetry install --with dev,test
poetry run pytest

The migrated data extractor tests are documented in docs/testing.md.

Canonical Imports

from soil_etl.yaml_mapper import YamlDatasetMapper
from soil_etl.bindings import Binding, FromColumn
from soil_etl.db import ImportationInterface

The canonical package namespace is soil_etl. Older migration notes may still refer to forsite_soil_extractors, but new code should use only soil_etl imports.

Build

python -m build
twine check dist/*