p360-interface-bundle

Interface between Persona360 app instance and a project with Daipe Feature store.

Many operations need to be done between a feature store is populated and an app is working properly based on it. We may want to adjust some nomenclature in metadata for the app, we surely need to ensure the latest snapshot of the feature store is propagated to the app's underlying Clickhouse database. For these operational services, p360-interface-bundle comes to hand.

Setup

Nothing more to do then to add p360-interface-bundle to your project:

poetry add p360-interface-bundle

Tools

Feature store post-actions

metadata_adjustment.CreateMetadataJsonFiles

• Get feature store metadata
• Rename some of their columns
• Introduce subcategory mapping
• Save the resulting metadata JSONs for an app to ADLS

Configuration

Add the following configuration to your project's config.yaml:

parameters:
  p360_interface_bundle:
    featurestore:
      post_actions:
        metadata_adjustment:
          export_path: "abfss://<container>@<adls domain>/<path to folder>"
          mapping:
            general:
              <original column name in FS metadata>: <replacement value>
              <another original column name in FS metadata>: <another replacement value>
            category:
              <category name>:
                - <first subcategory of this category>
                - <second subcategory of this category>
              <another category name>:
                - <first subcategory of this category>
                - <second subcategory of this category>

Latest Changes

• Added support for exporting array features. PR #39 by @guderkar.
• Import update ch. PR #36 by @fkrjakDatasentics.
• Added statistics to exported metadata. PR #33 by @guderkar.
• Added additional metadata on which persona app relies. PR #32 by @guderkar.
• Clickhouse export rewritten from skeleton. PR #31 by @guderkar.
• Add Pycharm files. PR #29 by @ds-lukaslangr.
• Update feature store. PR #28 by @ds-lukaslangr.
• 🐛 Extras conversion to strings suppressed. PR #26 by @matejoravec.
• 📝 Post-version-bump README update. PR #24 by @matejoravec.

0.1.1

• 🔖 Bump version to 0.1.1. PR #23 by @matejoravec.
• 📝 Documentation in README. PR #22 by @matejoravec.
• 🏗️ Default configuration removed. PR #21 by @matejoravec.
• ✅ MetadataJsonGetter test added. PR #20 by @matejoravec.

0.1.0

• 🐛 Metadata adjustment fix. PR #16 by @matejoravec.
• ⬆️ feature-store-bundle upgraded to 2.5.0. PR #12 by @matejoravec.
• 📄 License fix. PR #7 by @matejoravec.
• 👷 Release pipeline. PR #5 by @matejoravec.
• ✨ Metadata adjustment post-action. PR #4 by @matejoravec.
• 🧱 Basic bundle infrastructure. PR #3 by @matejoravec.
• 👷 Wrong main branch name fixed. PR #2 by @matejoravec.