utic-dev-tools 0.2.9

Dev tools for Unstructured.io ecosystem.

Introduction

A collection of Types and Schemas for use in various projects in the Unstructured ecosystem.

The project is intended to be very lightweight, and to play the role of the abstraction in "Concretions can depend on abstractions, but not vice-versa".

In other words, this project should exclusively define types and remain lightweight enough to encourage other projects to take it on as a dependency. Ideally, the project would itself have no dependencies. For rapid development in early phases, we take a dependency on Pydantic for easier modeling.

Right now, it is mostly utilized for defining unstructured's plugin architecture to enable developers to define their own plugins.

Note: This package is automatically published to PyPI on every merge to main.

Requirements

• Python 3.12
• uv: pip install uv

Quick start

• make install
• make test

unstructured-plugins command

Basic Usage

unstructured-plugins publish plugin_hello_world:PLUGIN_MANIFEST --registry http://localhost:8080/

CLI Arguments

Registry Configuration

• --registry: URL to registry to publish to
• --registry-project: Some registries require objects to be organized into projects
• --registry-username: Username for authentication
• --registry-password: Password for authentication
• --channel: Channel or tag to push or read from (default: 'dev')
• --from-channel: Used for promoting from one channel to another

Metadata Overrides (publish only)

• --override-name: Override the plugin name from the manifest
• --override-type: Override the plugin type from the manifest
• --override-subtype: Override the plugin subtype from the manifest
• --dry-run: Show what would be published without making any changes to the registry

Adding plugin icon (publish only)

• --icon-path: Path to the plugin icon (currently only .svg files are supported)

Environment Variables

Configs can also be provided as env vars:

• PLUGIN_REGISTRY
• PLUGIN_REGISTRY_USERNAME
• PLUGIN_REGISTRY_PASSWORD
• PLUGIN_REGISTRY_PROJECT
• PLUGIN_REGISTRY_VERIFY_SSL

Publishing Multiple Variants

You can publish multiple plugin variants from a single manifest using override flags:

unstructured-plugins publish s3_protocol:PLUGIN_MANIFEST \
  --override-type index \
  --override-subtype vast \
  --override-name "VAST Indexer" \
  --channel dev

Using Dry-Run

Preview what will be published before making changes:

unstructured-plugins publish s3_protocol:PLUGIN_MANIFEST \
  --override-type index \
  --override-subtype vast \
  --override-name "VAST Indexer" \
  --dry-run

This will show:

• Registry path where plugin will be published
• Tags that will be created
• Overrides that were applied
• Complete metadata payload
• Plugin index entry
• Command to run for real publish

Plugin icon

You can add an icon to the plugin by using --icon-path flag. Icon has to be an .svg file. It gets converted into an object that contains a base64 encoded string of the given svg. Icon gets added to the default presentation layer.

{
  ...,
  "presentation": {
    "default": {
      ...,
      "icon": {
        "value": "PHN2ZyB4bWxucz0i...",
        "original_file_extension": "svg",
        "encoding": "base64",
        "mime_type": "image/svg+xml"
      },
      ...
    }
  }
  ...
}