unstructured-platform-plugins 0.0.3

A utility library that provides tools around unstructured plugin development

Unstructured Platform Plugins

Information about how to build custom plugins to integrate with Unstructured Platform.

Plugin Development

Any plugin must be published in a dedicated docker image with all required dependencies that when run, exposes an api on port 8000 with the required endpoints to interact with the Unstructured Platform product:

• /invoke: A POST endpoint which gets all data to run the underlying logic in the request body and expects a json serializable response.
• /schema: A GET endpoint which publishes a json schema formatted response with the schema of the input and output expected by the plugin.
• /id: A GET endpoint which publishes a string unique identifier for this instance of the plugin. Will default to a hash of the schema response if one is not set explicitly.

Utility CLI

When installing this repo, it also installs the cli etl-uvicorn. This takes a pointer to any generic python function and wraps it in a FastApi application to conform to the patterns that are expected by the api hosting the plugin logic. This cli extends the existing uvicorn cli which takes in a pointer to a fastapi instance or factory but instead takes in a pointer to a python function/class which gets wrapped with a FastApi application.

Example usage

For all following commands, make sure you have the local repo in your PYTHONPATH:

export PYTHONPATH=.

Wrapping a basic function with a typed dict response

etl-uvicorn test.assets.typed_dict_response:sample_function

Wrapping a basic async function with a typed dict response

etl-uvicorn test.assets.async_typed_dict_response:async_sample_function

Wrapping a class. For this to work, the class must be self instantiating. When passing a class in, a method needs to be passed in as well, otherwise __call__ is used. The following example returns a pydantic BaseModel

etl-uvicorn test.assets.pydantic_response_class_method:SampleClass --method-name sample_method

Wrapping an instance of a class.

etl-uvicorn test.assets.pydantic_response_class_method:sample_class --method-name sample_method

The CLI does some validation on the wrapped function, which must have explicit inputs and outputs, meaning *args and **kwargs are not supported. These will cause the cli to fail fast.

etl-uvicorn test.assets.improper_function:sample_improper_function

/id requirements

All the the above examples caused the CLI to autogenerate the /id endpoint with a hash of the generated schema. However, you can also provide it a reference to use for the id value. This can be a reference to a concrete value (i.e. plugin_id="my_plugin_id") or a function in the same way that one was passed in to be wrapped above.

Will populate the response of /id with the static value of hash_value:

etl-uvicorn test.assets.typed_dict_response:sample_function --plugin-id test.assets.simple_hash_value:hash_value

Can populate it using a lambda:

etl-uvicorn test.assets.typed_dict_response:sample_function --plugin-id test.assets.simple_hash_lambda:hash_lambda_fn

Similar to the function being wrapped, can also use a class:

etl-uvicorn test.assets.typed_dict_response:sample_function --plugin-id test.assets.simple_hash_class:GetHash --plugin-id-method my_hash

Or the instantiated class:

etl-uvicorn test.assets.typed_dict_response:sample_function --plugin-id test.assets.simple_hash_class:get_hash_class_instance --plugin-id-method my_hash