canonmap 0.1.99

CanonMap - A Python library for entity canonicalization and mapping

CanonMap

CanonMap is a Python library for generating and managing canonical entity artifacts from various data sources. It provides a streamlined interface for processing data files and generating standardized artifacts that can be used for entity matching and data integration.

Features

• Flexible Input Support: Process data from:

  • CSV/JSON files
  • Directories of data files
  • Pandas DataFrames
  • Python dictionaries

• Artifact Generation:

  • Generate canonical entity lists
  • Create database schemas (supports multiple database types)
  • Generate semantic embeddings for entities
  • Clean and standardize field names
  • Process metadata fields

• Database Support:

  • DuckDB (default)
  • SQLite
  • BigQuery
  • MariaDB
  • MySQL
  • PostgreSQL

Installation

pip install canonmap

Quick Start

from canonmap import CanonMap
from canonmap.models.artifact_generation_request import ArtifactGenerationRequest, EntityField

# Initialize CanonMap
canonmap = CanonMap()

# Configure artifact generation
config = ArtifactGenerationRequest(
    input_path="path/to/your/data.csv",
    output_path="path/to/output",
    source_name="my_source",
    table_name="my_table",
    entity_fields=[
        EntityField(table_name="my_table", field_name="name"),
        EntityField(table_name="my_table", field_name="id")
    ],
    semantic_fields=["description", "notes"],
    generate_schema=True,
    generate_embeddings=True
)

# Generate artifacts
results = canonmap.generate(config)

Configuration Options

The ArtifactGenerationRequest model provides extensive configuration options:

• Input/Output:

  • input_path: Path to data file/directory or DataFrame/dict
  • output_path: Directory for generated artifacts
  • source_name: Logical source name
  • table_name: Logical table name

• Directory Processing:

  • recursive: Process subdirectories
  • file_pattern: File matching pattern (e.g., "*.csv")
  • table_name_from_file: Use filename as table name

• Entity Processing:

  • entity_fields: List of fields to treat as entities
  • semantic_fields: Fields for semantic embeddings
  • use_other_fields_as_metadata: Include non-entity fields as metadata

• Generation Options:

  • generate_canonical_entities: Generate entity list
  • generate_schema: Generate database schema
  • generate_embeddings: Generate semantic embeddings
  • save_processed_data: Save cleaned data
  • schema_database_type: Target database type
  • clean_field_names: Standardize field names

Output

The generate() method returns a dictionary containing:

• Generated artifacts
• Paths to saved files
• Schema information (if requested)
• Embeddings (if requested)
• Processed data (if requested)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.