Skip to main content

Microsoft Azure Form Recognizer Client Library for Python

Project description

Azure Form Recognizer client library for Python

Azure Cognitive Services Form Recognizer is a cloud service that uses machine learning to recognize text and table data from form documents. It includes the following main functionalities:

  • Custom models - Recognize field values and table data from forms. These models are trained with your own data, so they're tailored to your forms.
  • Content API - Recognize text, table structures, and selection marks, along with their bounding box coordinates, from documents. Corresponds to the REST service's Layout API.
  • Prebuilt models - Recognize data using the following prebuilt models
    • Receipt model - Recognize data from sales receipts using a prebuilt model.
    • Business card model - Recognize data from business cards using a prebuilt model.
    • Invoice model - Recognize data from invoices using a prebuilt model.

Source code | Package (PyPI) | API reference documentation| Product documentation | Samples

Getting started

Prerequisites

Install the package

Install the Azure Form Recognizer client library for Python with pip:

pip install azure-ai-formrecognizer --pre

Note: This version of the client library defaults to the v2.1-preview version of the service

This table shows the relationship between SDK versions and supported API versions of the service

SDK version Supported API version of service
3.0.0 - Latest GA release (can be installed by removing the --pre flag) 2.0
3.1.0b2 - Latest release (beta) 2.0, 2.1-preview

Create a Form Recognizer resource

Form Recognizer supports both multi-service and single-service access. Create a Cognitive Services resource if you plan to access multiple cognitive services under a single endpoint/key. For Form Recognizer access only, create a Form Recognizer resource.

You can create the resource using

Option 1: Azure Portal

Option 2: Azure CLI. Below is an example of how you can create a Form Recognizer resource using the CLI:

# Create a new resource group to hold the form recognizer resource -
# if using an existing resource group, skip this step
az group create --name my-resource-group --location westus2
# Create form recognizer
az cognitiveservices account create \
    --name form-recognizer-resource \
    --resource-group my-resource-group \
    --kind FormRecognizer \
    --sku F0 \
    --location westus2 \
    --yes

Authenticate the client

In order to interact with the Form Recognizer service, you will need to create an instance of a client. An endpoint and credential are necessary to instantiate the client object.

Looking up the endpoint

You can find the endpoint for your Form Recognizer resource using the Azure Portal or Azure CLI:

# Get the endpoint for the form recognizer resource
az cognitiveservices account show --name "resource-name" --resource-group "resource-group-name" --query "properties.endpoint"

Get the API key

The API key can be found in the Azure Portal or by running the following Azure CLI command:

az cognitiveservices account keys list --name "resource-name" --resource-group "resource-group-name"

Create the client with AzureKeyCredential

To use an API key as the credential parameter, pass the key as a string into an instance of AzureKeyCredential.

from azure.core.credentials import AzureKeyCredential
from azure.ai.formrecognizer import FormRecognizerClient

endpoint = "https://<region>.api.cognitive.microsoft.com/"
credential = AzureKeyCredential("<api_key>")
form_recognizer_client = FormRecognizerClient(endpoint, credential)

Create the client with an Azure Active Directory credential

AzureKeyCredential authentication is used in the examples in this getting started guide, but you can also authenticate with Azure Active Directory using the azure-identity library. Note that regional endpoints do not support AAD authentication. Create a custom subdomain name for your resource in order to use this type of authentication.

To use the DefaultAzureCredential type shown below, or other credential types provided with the Azure SDK, please install the azure-identity package:

pip install azure-identity

You will also need to register a new AAD application and grant access to Form Recognizer by assigning the "Cognitive Services User" role to your service principal.

Once completed, set the values of the client ID, tenant ID, and client secret of the AAD application as environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_CLIENT_SECRET.

from azure.identity import DefaultAzureCredential
from azure.ai.formrecognizer import FormRecognizerClient
credential = DefaultAzureCredential()

form_recognizer_client = FormRecognizerClient(
    endpoint="https://<my-custom-subdomain>.cognitiveservices.azure.com/",
    credential=credential
)

Key concepts

FormRecognizerClient

FormRecognizerClient provides operations for:

  • Recognizing form fields and content using custom models trained to recognize your custom forms. These values are returned in a collection of RecognizedForm objects.
  • Recognizing common fields from the following form types using prebuilt models. These fields and metadata are returned in a collection of RecognizedForm objects.
    • Sales receipts. See fields found on a receipt here.
    • Business cards. See fields found on a business card here.
    • Invoices. See fields found on an invoice here.
  • Recognizing form content, including tables, lines, words, and selection marks, without the need to train a model. Form content is returned in a collection of FormPage objects.

Sample code snippets are provided to illustrate using a FormRecognizerClient here.

FormTrainingClient

FormTrainingClient provides operations for:

  • Training custom models without labels to recognize all fields and values found in your custom forms. A CustomFormModel is returned indicating the form types the model will recognize, and the fields it will extract for each form type. See the service documentation for a more detailed explanation.
  • Training custom models with labels to recognize specific fields, selection marks, and values you specify by labeling your custom forms. A CustomFormModel is returned indicating the fields the model will extract, as well as the estimated accuracy for each field. See the service documentation for a more detailed explanation.
  • Managing models created in your account.
  • Copying a custom model from one Form Recognizer resource to another.
  • Creating a composed model from a collection of existing trained models with labels.

Please note that models can also be trained using a graphical user interface such as the Form Recognizer Labeling Tool.

Sample code snippets are provided to illustrate using a FormTrainingClient here.

Long-Running Operations

Long-running operations are operations which consist of an initial request sent to the service to start an operation, followed by polling the service at intervals to determine whether the operation has completed or failed, and if it has succeeded, to get the result.

Methods that train models, recognize values from forms, or copy/compose models are modeled as long-running operations. The client exposes a begin_<method-name> method that returns an LROPoller or AsyncLROPoller. Callers should wait for the operation to complete by calling result() on the poller object returned from the begin_<method-name> method. Sample code snippets are provided to illustrate using long-running operations below.

Examples

The following section provides several code snippets covering some of the most common Form Recognizer tasks, including:

Recognize Forms Using a Custom Model

Recognize name/value pairs and table data from forms. These models are trained with your own data, so they're tailored to your forms. For best results, you should only recognize forms of the same form type that the custom model was trained on.

from azure.ai.formrecognizer import FormRecognizerClient
from azure.core.credentials import AzureKeyCredential

endpoint = "https://<region>.api.cognitive.microsoft.com/"
credential = AzureKeyCredential("<api_key>")

form_recognizer_client = FormRecognizerClient(endpoint, credential)
model_id = "<your custom model id>"

with open("<path to your form>", "rb") as fd:
    form = fd.read()

poller = form_recognizer_client.begin_recognize_custom_forms(model_id=model_id, form=form)
result = poller.result()

for recognized_form in result:
    print("Form type: {}".format(recognized_form.form_type))
    print("Form type confidence: {}".format(recognized_form.form_type_confidence))
    print("Form was analyzed using model with ID: {}".format(recognized_form.model_id))
    for name, field in recognized_form.fields.items():
        print("Field '{}' has label '{}' with value '{}' and a confidence score of {}".format(
            name,
            field.label_data.text if field.label_data else name,
            field.value,
            field.confidence
        ))

Alternatively, a form URL can also be used to recognize custom forms using the begin_recognize_custom_forms_from_url method. The _from_url methods exist for all the recognize methods.

form_url = "<url_of_the_form>"
poller = form_recognizer_client.begin_recognize_custom_forms_from_url(model_id=model_id, form_url=form_url)
result = poller.result()

Recognize Content

Recognize text, selection marks, and table structures, along with their bounding box coordinates, from documents.

from azure.ai.formrecognizer import FormRecognizerClient
from azure.core.credentials import AzureKeyCredential

endpoint = "https://<region>.api.cognitive.microsoft.com/"
credential = AzureKeyCredential("<api_key>")

form_recognizer_client = FormRecognizerClient(endpoint, credential)

with open("<path to your form>", "rb") as fd:
    form = fd.read()

poller = form_recognizer_client.begin_recognize_content(form)
form_pages = poller.result()

for content in form_pages:
    for table in content.tables:
        print("Table found on page {}:".format(table.page_number))
        print("Table location {}:".format(table.bounding_box))
        for cell in table.cells:
            print("Cell text: {}".format(cell.text))
            print("Location: {}".format(cell.bounding_box))
            print("Confidence score: {}\n".format(cell.confidence))

    if content.selection_marks:
        print("Selection marks found on page {}:".format(content.page_number))
        for selection_mark in content.selection_marks:
            print("Selection mark is '{}' within bounding box '{}' and has a confidence of {}".format(
                selection_mark.state,
                selection_mark.bounding_box,
                selection_mark.confidence
            ))

Recognize Receipts

Recognize data from sales receipts using a prebuilt model. Receipt fields recognized by the service can be found here.

from azure.ai.formrecognizer import FormRecognizerClient
from azure.core.credentials import AzureKeyCredential

endpoint = "https://<region>.api.cognitive.microsoft.com/"
credential = AzureKeyCredential("<api_key>")

form_recognizer_client = FormRecognizerClient(endpoint, credential)

with open("<path to your receipt>", "rb") as fd:
    receipt = fd.read()

poller = form_recognizer_client.begin_recognize_receipts(receipt)
result = poller.result()

for receipt in result:
    for name, field in receipt.fields.items():
        if name == "Items":
            print("Receipt Items:")
            for idx, items in enumerate(field.value):
                print("...Item #{}".format(idx+1))
                for item_name, item in items.value.items():
                    print("......{}: {} has confidence {}".format(item_name, item.value, item.confidence))
        else:
            print("{}: {} has confidence {}".format(name, field.value, field.confidence))

Recognize Business Cards

Recognize data from business cards using a prebuilt model. Business card fields recognized by the service can be found here.

from azure.ai.formrecognizer import FormRecognizerClient
from azure.core.credentials import AzureKeyCredential

endpoint = "https://<region>.api.cognitive.microsoft.com/"
credential = AzureKeyCredential("<api_key>")

form_recognizer_client = FormRecognizerClient(endpoint, credential)

with open("<path to your business card>", "rb") as fd:
    business_card = fd.read()

poller = form_recognizer_client.begin_recognize_business_cards(business_card)
result = poller.result()

for business_card in result:
    for name, field in business_card.fields.items():
        if name == "ContactNames":
            print("ContactNames:")
            for items in field.value:
                for item_name, item in items.value.items():
                    print("...{}: {} has confidence {}".format(item_name, item.value, item.confidence))
        else:
            for item in field.value:
                print("{}: {} has confidence {}".format(item.name, item.value, item.confidence))

Recognize Invoices

Recognize data from invoices using a prebuilt model. Invoice fields recognized by the service can be found here.

from azure.ai.formrecognizer import FormRecognizerClient
from azure.core.credentials import AzureKeyCredential

endpoint = "https://<region>.api.cognitive.microsoft.com/"
credential = AzureKeyCredential("<api_key>")

form_recognizer_client = FormRecognizerClient(endpoint, credential)

with open("<path to your invoice>", "rb") as fd:
    invoice = fd.read()

poller = form_recognizer_client.begin_recognize_invoices(invoice)
result = poller.result()

for invoice in result:
    for name, field in invoice.fields.items():
        print("{}: {} has confidence {}".format(name, field.value, field.confidence))

Train a model

Train a custom model on your own form type. The resulting model can be used to recognize values from the types of forms it was trained on. Provide a container SAS URL to your Azure Storage Blob container where you're storing the training documents. If training files are within a subfolder in the container, use the prefix keyword argument to specify under which folder to train.

More details on setting up a container and required file structure can be found in the service documentation.

from azure.ai.formrecognizer import FormTrainingClient
from azure.core.credentials import AzureKeyCredential

endpoint = "https://<region>.api.cognitive.microsoft.com/"
credential = AzureKeyCredential("<api_key>")

form_training_client = FormTrainingClient(endpoint, credential)

container_sas_url = "<container-sas-url>"  # training documents uploaded to blob storage
poller = form_training_client.begin_training(
    container_sas_url, use_training_labels=False, model_name="my first model"
)
model = poller.result()

# Custom model information
print("Model ID: {}".format(model.model_id))
print("Model name: {}".format(model.model_name))
print("Is composed model?: {}".format(model.properties.is_composed_model))
print("Status: {}".format(model.status))
print("Training started on: {}".format(model.training_started_on))
print("Training completed on: {}".format(model.training_completed_on))

print("\nRecognized fields:")
for submodel in model.submodels:
    print(
        "The submodel with form type '{}' and model ID '{}' has recognized the following fields: {}".format(
            submodel.form_type, submodel.model_id,
            ", ".join(
                [
                    field.label if field.label else name
                    for name, field in submodel.fields.items()
                ]
            ),
        )
    )

# Training result information
for doc in model.training_documents:
    print("Document name: {}".format(doc.name))
    print("Document status: {}".format(doc.status))
    print("Document page count: {}".format(doc.page_count))
    print("Document errors: {}".format(doc.errors))

Manage Your Models

Manage the custom models attached to your account.

from azure.ai.formrecognizer import FormTrainingClient
from azure.core.credentials import AzureKeyCredential
from azure.core.exceptions import ResourceNotFoundError

endpoint = "https://<region>.api.cognitive.microsoft.com/"
credential = AzureKeyCredential("<api_key>")

form_training_client = FormTrainingClient(endpoint, credential)

account_properties = form_training_client.get_account_properties()
print("Our account has {} custom models, and we can have at most {} custom models".format(
    account_properties.custom_model_count, account_properties.custom_model_limit
))

# Here we get a paged list of all of our custom models
custom_models = form_training_client.list_custom_models()
print("We have models with the following ids: {}".format(
    ", ".join([m.model_id for m in custom_models])
))

# Replace with the custom model ID from the "Train a model" sample
model_id = "<model_id from the Train a Model sample>"

custom_model = form_training_client.get_custom_model(model_id=model_id)
print("Model ID: {}".format(custom_model.model_id))
print("Model name: {}".format(custom_model.model_name))
print("Is composed model?: {}".format(custom_model.properties.is_composed_model))
print("Status: {}".format(custom_model.status))
print("Training started on: {}".format(custom_model.training_started_on))
print("Training completed on: {}".format(custom_model.training_completed_on))

# Finally, we will delete this model by ID
form_training_client.delete_model(model_id=custom_model.model_id)

try:
    form_training_client.get_custom_model(model_id=custom_model.model_id)
except ResourceNotFoundError:
    print("Successfully deleted model with id {}".format(custom_model.model_id))

Troubleshooting

General

Form Recognizer client library will raise exceptions defined in Azure Core.

Logging

This library uses the standard logging library for logging.

Basic information about HTTP sessions (URLs, headers, etc.) is logged at INFO level.

Detailed DEBUG level logging, including request/response bodies and unredacted headers, can be enabled on the client or per-operation with the logging_enable keyword argument.

See full SDK logging documentation with examples here.

Optional Configuration

Optional keyword arguments can be passed in at the client and per-operation level. The azure-core reference documentation describes available configurations for retries, logging, transport protocols, and more.

Next steps

The following section provides several code snippets illustrating common patterns used in the Form Recognizer Python API.

More sample code

These code samples show common scenario operations with the Azure Form Recognizer client library.

Async APIs

This library also includes a complete async API supported on Python 3.5+. To use it, you must first install an async transport, such as aiohttp. Async clients are found under the azure.ai.formrecognizer.aio namespace.

Additional documentation

For more extensive documentation on Azure Cognitive Services Form Recognizer, see the Form Recognizer documentation on docs.microsoft.com.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Release History

3.1.0b2 (2021-01-12)

Bug Fixes

  • Package requires azure-core version 1.8.2 or greater

3.1.0b1 (2020-11-23)

This version of the SDK defaults to the latest supported API version, which currently is v2.1-preview.

New features

  • New methods begin_recognize_business_cards and begin_recognize_business_cards_from_url introduced to the SDK. Use these methods to recognize data from business cards
  • New methods begin_recognize_invoices and begin_recognize_invoices_from_url introduced to the SDK. Use these methods to recognize data from invoices
  • Recognize receipt methods now take keyword argument locale to optionally indicate the locale of the receipt for improved results
  • Added ability to create a composed model from the FormTrainingClient by calling method begin_create_composed_model()
  • Added support to train and recognize custom forms with selection marks such as check boxes and radio buttons. This functionality is only available for models trained with labels
  • Added property selection_marks to FormPage which contains a list of FormSelectionMark
  • When passing include_field_elements=True, the property field_elements on FieldData and FormTableCell will also be populated with any selection marks found on the page
  • Added the properties model_name and properties to types CustomFormModel and CustomFormModelInfo
  • Added keyword argument model_name to begin_training() and begin_create_composed_model()
  • Added model type CustomFormModelProperties that includes information like if a model is a composed model
  • Added property model_id to CustomFormSubmodel and TrainingDocumentInfo
  • Added properties model_id and form_type_confidence to RecognizedForm
  • appearance property added to FormLine to indicate the style of extracted text - like "handwriting" or "other"
  • Added keyword argument pages to begin_recognize_content and begin_recognize_content_from_url to specify the page numbers to analyze
  • Added property bounding_box to FormTable
  • Content-type image/bmp now supported by recognize content and prebuilt models
  • Added keyword argument language to begin_recognize_content and begin_recognize_content_from_url to specify which language to process document in

Dependency updates

3.0.0 (2020-08-20)

First stable release of the azure-ai-formrecognizer client library.

New features

  • Client-level, keyword argument api_version can be used to specify the service API version to use. Currently only v2.0 is supported. See the enum FormRecognizerApiVersion for supported API versions.
  • FormWord and FormLine now have attribute kind which specifies the kind of element it is, e.g. "word" or "line"

3.0.0b1 (2020-08-11)

The version of this package now targets the service's v2.0 API.

Breaking Changes

  • Client library version bumped to 3.0.0b1
  • Values are now capitalized for enums FormContentType, LengthUnit, TrainingStatus, and CustomFormModelStatus
  • document_name renamed to name on TrainingDocumentInfo
  • Keyword argument include_sub_folders renamed to include_subfolders on begin_training methods

New features

  • FormField now has attribute value_type which contains the semantic data type of the field value. The options for value_type are described in the enum FieldValueType

Fixes and improvements

  • Fixes a bug where error code and message weren't being returned on HttpResponseError if operation failed during polling
  • FormField property value_data is now set to None if no values are returned on its FieldData. Previously value_data returned a FieldData with all its attributes set to None in the above case.

1.0.0b4 (2020-07-07)

Breaking Changes

  • RecognizedReceipts class has been removed.
  • begin_recognize_receipts and begin_recognize_receipts_from_url now return RecognizedForm.
  • requested_on has been renamed to training_started_on and completed_on renamed to training_completed_on on CustomFormModel and CustomFormModelInfo
  • FieldText has been renamed to FieldData
  • FormContent has been renamed to FormElement
  • Parameter include_text_content has been renamed to include_field_elements for begin_recognize_receipts, begin_recognize_receipts_from_url, begin_recognize_custom_forms, and begin_recognize_custom_forms_from_url
  • text_content has been renamed to field_elements on FieldData and FormTableCell

Fixes and improvements

  • Fixes a bug where text_angle was being returned out of the specified interval (-180, 180]

1.0.0b3 (2020-06-10)

Breaking Changes

  • All asynchronous long running operation methods now return an instance of an AsyncLROPoller from azure-core
  • All asynchronous long running operation methods are renamed with the begin_ prefix to indicate that an AsyncLROPoller is returned:
    • train_model is renamed to begin_training
    • recognize_receipts is renamed to begin_recognize_receipts
    • recognize_receipts_from_url is renamed to begin_recognize_receipts_from_url
    • recognize_content is renamed to begin_recognize_content
    • recognize_content_from_url is renamed to begin_recognize_content_from_url
    • recognize_custom_forms is renamed to begin_recognize_custom_forms
    • recognize_custom_forms_from_url is renamed to begin_recognize_custom_forms_from_url
  • Sync method begin_train_model renamed to begin_training
  • training_files parameter of begin_training is renamed to training_files_url
  • use_labels parameter of begin_training is renamed to use_training_labels
  • list_model_infos method has been renamed to list_custom_models
  • Removed get_form_training_client from FormRecognizerClient
  • Added get_form_recognizer_client to FormTrainingClient
  • A HttpResponseError is now raised if a model with status=="invalid" is returned from the begin_training methods
  • PageRange is renamed to FormPageRange
  • first_page and last_page renamed to first_page_number and last_page_number, respectively on FormPageRange
  • FormField does not have a page_number
  • use_training_labels is now a required positional param in the begin_training APIs
  • stream and url parameters found on methods for FormRecognizerClient have been renamed to form and form_url, respectively
  • For begin_recognize_receipt methods, parameters have been renamed to receipt and receipt_url
  • created_on and last_modified are renamed to requested_on and completed_on in the CustomFormModel and CustomFormModelInfo models
  • models property of CustomFormModel is renamed to submodels
  • CustomFormSubModel is renamed to CustomFormSubmodel
  • begin_recognize_receipts APIs now return a list of RecognizedReceipt instead of USReceipt
  • Removed USReceipt. To see how to deal with the return value of begin_recognize_receipts, see the recognize receipt samples in the samples directory for details.
  • Removed USReceiptItem. To see how to access the individual items on a receipt, see the recognize receipt samples in the samples directory for details.
  • Removed USReceiptType and the receipt_type property from RecognizedReceipt. See the recognize receipt samples in the samples directory for details.

New features

  • Support to copy a custom model from one Form Recognizer resource to another
  • Authentication using azure-identity credentials now supported
  • page_number attribute has been added to FormTable
  • All long running operation methods now accept the keyword argument continuation_token to restart the poller from a saved state

Dependency updates

1.0.0b2 (2020-05-06)

Fixes and improvements

  • Bug fixed where confidence == 0.0 was erroneously getting set to 1.0
  • __repr__ has been added to all of the models

1.0.0b1 (2020-04-23)

Version (1.0.0b1) is the first preview of our efforts to create a user-friendly and Pythonic client library for Azure Form Recognizer. This library replaces the package found here: https://pypi.org/project/azure-cognitiveservices-formrecognizer/

For more information about this, and preview releases of other Azure SDK libraries, please visit https://azure.github.io/azure-sdk/releases/latest/python.html.

Breaking changes: New API design

  • New namespace/package name:
    • The namespace/package name for the Form Recognizer client library has changed from azure.cognitiveservices.formrecognizer to azure.ai.formrecognizer
  • Two client design:
    • FormRecognizerClient to analyze fields/values on custom forms, receipts, and form content/layout
    • FormTrainingClient to train custom models (with/without labels), and manage the custom models on your account
  • Different analyze methods based on input type: file stream or URL.
    • URL input should use the method with suffix from_url
    • Stream methods will automatically detect content-type of the input file
  • Asynchronous APIs added under azure.ai.formrecognizer.aio namespace
  • Authentication with API key supported using AzureKeyCredential("<api_key>") from azure.core.credentials
  • New underlying REST pipeline implementation based on the azure-core library
  • Client and pipeline configuration is now available via keyword arguments at both the client level, and per-operation. See README for a link to optional configuration arguments
  • New error hierarchy:
    • All service errors will now use the base type: azure.core.exceptions.HttpResponseError

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

azure-ai-formrecognizer-3.1.0b2.zip (300.1 kB view hashes)

Uploaded Source

Built Distribution

azure_ai_formrecognizer-3.1.0b2-py2.py3-none-any.whl (137.9 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page