NuMind SDK

Python SDK to interact with NuMind's models API: NuExtract and NuMarkdown.

Installation

pip install numind

Usage and code examples

Create a client

You must first get an API key on the NuExtract platform.

import os
from numind import NuMind

# Create a client object to interact with the API
# Providing the `api_key` is not required if the `NUMIND_API_KEY` environment variable
# is already set.
client = NuMind(api_key=os.environ["NUMIND_API_KEY"])

You can create an async client by using the NuMindAsync class. The methods and their usages are the same as for the sync NuMind client.

Extract structured data

If you want to extract structured information without a project, provide the template directly to extract_structured_data:

template = {
    "destination": {
        "name": "verbatim-string",
        "zip_code": "string",
        "country": "country",
    },
    "accommodation": "verbatim-string",
    "activities": ["verbatim-string"],
    "duration": "duration",
}
input_file_path = Path("to", "file.pdf")
response = client.extract_structured_data(
    template=template,
    input_file=input_file_path,
)
print(response.result)

{
    "destination": {
        "name": "Tahiti",
        "zip_code": "98730",
        "country": "FR"
    },
    "accommodation": "overwater bungalow in Bora Bora",
    "activities": [
        "snorkeling",
        "paddleboarding",
        "basking",
        "explore lush rainforests, hidden waterfalls, and the rich Polynesian culture"
    ],
    "duration": null
}

Create a good template

NuExtract uses JSON extraction templates which specify the information to retrieve and their types:

• string: a text, whose value can be abstract, i.e. totally free and can be deduced from calculations, reasoning, external knowledge;
• verbatim-string: a purely extractive text whose value must be present in the document. Some flexibility might be allowed on the formatting, e.g. new lines and escaped characters (e.g. \n) in a documents might be represented with a space;
• integer: an integer number;
• number: any number, that may be a floating point number or an integer;
• boolean: a boolean whose value should be either true or false;
• date-time: a date or time whose value should follow the ISO 8601 standard (YYYY-MM-DDThh:mm:ss). It may feature "reduced" accuracy, i.e. omitting certain date or time components not useful in specific cases. For examples, if the extracted value is a date, YYYY-MM-DD is a valid value format. The same applies to times with the hh:mm:ss format (without omitting the leading T symbol). Additionally, the "least significant" component might be omitted if it is not required or specified. For example, a specific month and year can be specified as YYYY-MM while omitting the day component DD. A specific hour can be specified as hh while omitting the minutes and seconds components. When combining dates and time, only the least significant time components can be omitted, e.g. YYYY-MM-DDThh:mm which is omitting the seconds.

Additionally, the value of a field can be:

• a nested dictionary, i.e. another branch, describing elements associated to their parent node (key);
• an array of items of the form ["type"], whose values are elements of a given "type", which can also be a dictionary of unspecified depth;
• an enum, i.e. a list of elements to choose from of the form ["choice1", "choice2", ...]. For values of this type, just set the value of the item to choose, e.g. "choice1", and do not set the value as an array containing the item such as ["choice1"];
• a multi-enum, i.e. a list from which multiple elements can be picked, of the form [["choice1", "choice2", ...]] (double square brackets).

Converting JSON schema / Pydantic models to NuExtract template

The SDK offers a method to convert JSON schemas to NuExtract templates:

from typing import Literal

from pydantic import Field, BaseModel
from numind.nuextract_utils import convert_json_schema_to_nuextract_template


class HotelBooking(BaseModel):
    city: str
    check_in_date: str = Field(description="date")
    check_out_date: str = Field(description="date")
    number_of_guests: int
    room_type: Literal["single", "double", "suite"]


conversion = convert_json_schema_to_nuextract_template(
    HotelBooking.model_json_schema(),
)
template = conversion["template"]
incompatibilities = conversion["incompatibilities"]

# {'check_in_date': 'date', 'check_out_date': 'date', 'city': 'string', 'number_of_guests': 'integer', 'room_type': ['single', 'double', 'suite']}

When an instance is provided, instance_status reports whether it is valid for both the source JSON Schema and converted template. adapted_instance contains the original value or a deterministically corrected value only when it is valid for both; otherwise it is None, and incompatibilities identifies the affected locations.

Inferring a template

Template generation is asynchronous. Submit a natural-language description, poll the job status, and retrieve the generated template once complete.

import time

from numind.models import TemplateRequest

description = "Create a template that extracts key information from an order confirmation email. The template should be able to pull details like the order ID, customer ID, date and time of the order, status, total amount, currency, item details (product ID, quantity, and unit price), shipping address, any customer requests or delivery preferences, and the estimated delivery date."
job_id = client.post_api_template_generation_jobs_text(
    TemplateRequest(description=description)
).job_id

job_status = client.get_api_jobs_jobid_status(job_id)
while job_status.completed_at is None:
    time.sleep(4)
    job_status = client.get_api_jobs_jobid_status(job_id)
if job_status.status != "completed":
    raise RuntimeError(f"Template generation failed with status {job_status.status}")

template = client.get_api_template_generation_jobs_templatejobid(job_id).result

Create a project

A project allows to define an information extraction task from a template and examples.

from numind.models import CreateStructuredProjectRequest

project_id = client.post_api_structured_extraction(
    CreateStructuredProjectRequest(
        name="vacation",
        description="Extraction of locations and activities",
        template=template,
        instructions="",
    )
).id

The project_id can also be found in the "API" tab of a project on the NuExtract website.

Add examples to a project to teach NuExtract via ICL (In-Context Learning)

from pathlib import Path

# Prepare examples, here a text and a file
example_1_input = "This is a text example"
example_1_expected_output = {
    "destination": {"name": None, "zip_code": None, "country": None}
}
example_2_input = Path("example_2.odt")
example_2_expected_output = {
    "destination": {"name": None, "zip_code": None, "country": None}
}
examples = [
    (example_1_input, example_1_expected_output),
    (example_2_input, example_2_expected_output),
]

# Add the examples to the project
client.add_examples_to_structured_extraction_project(project_id, examples)

Convert a document to a RAG-ready Markdown

from pathlib import Path

file_path = Path("document.pdf")
response = client.extract_content(file_path, job_status_polling_delay=2.0)
print(response.result)

extract_structured_data and extract_content poll job status every four seconds by default. Use job_status_polling_delay to select a different interval in seconds. Failed jobs raise RuntimeError by default. Set raise_on_job_fail=False to return a FailedJobStatusResponse containing the complete job status and its failure reason.

Documentation

Extracting Information from Documents

Once your structured extraction project is ready, submit a document to the endpoint:

https://nuextract.ai/api/structured-extraction/{structuredProjectId}/jobs

To use it, you need:

• To create an API key in the Account section
• To replace {structuredProjectId} by the project ID found in the API tab of the structured extraction project

This endpoint creates an asynchronous extraction job. Use the returned jobId to retrieve the result from:

https://nuextract.ai/api/structured-extraction/jobs/{structuredExtractionJobId}

You can test your extraction endpoint in your terminal using this command-line example with curl (make sure that you replace values of STRUCTURED_PROJECT_ID and NUEXTRACT_API_KEY):

NUEXTRACT_API_KEY=\"_your_api_key_here_\"; \\
STRUCTURED_PROJECT_ID=\"a24fd84a-44ab-4fd4-95a9-bebd46e4768b\"; \\
JOB_ID=$(curl \"https://nuextract.ai/api/structured-extraction/${STRUCTURED_PROJECT_ID}/jobs\" \\
  -X POST \\
  -H \"Authorization: Bearer ${NUEXTRACT_API_KEY}\" \\
  -F \"file=@${FILE_NAME}\" | jq -r '.jobId')

curl \"https://nuextract.ai/api/structured-extraction/jobs/${JOB_ID}\" \\
  -H \"Authorization: Bearer ${NUEXTRACT_API_KEY}\"

You can also use the Python SDK, by replacing the project_id, api_key and request values in the following code:

import asyncio
from numind import NuMindAsync

client = NuMindAsync(api_key=api_key)
requests = [{}]


async def main():
    return [
        await client.extract_structured_data(project_id, **request_kwargs)
        for request_kwargs in requests
    ]


responses = asyncio.run(main())

Using the Platform via API

Everything you can do on the web platform can be done via API - check the user guide to learn about how the platform works. This can be useful to create projects automatically, or to make your production more robust for example.

Main resources

• Structured Extraction Project - user project for structured extraction, identified by structuredProjectId
• Content Extraction Project - user project for content extraction, identified by contentProjectId
• File - uploaded file, identified by fileId, stored up to two weeks if not tied to an Example
• Document - internal representation of a document, identified by documentId, created from a File, stored up to two weeks if not tied to an Example
• Example - document-extraction pair given to teach NuExtract, identified by structuredExampleId, created from a Document
• Job - asynchronous extraction task, identified by structuredExtractionJobId

Most common API operations

Structured extraction

• Creating a Structured Extraction Project via POST /api/structured-extraction
• Changing the template of a Structured Extraction Project via PATCH /api/structured-extraction/{structuredProjectId}
• Changing settings of a Structured Extraction Project via PATCH /api/structured-extraction/{structuredProjectId}/settings
• Uploading a file to a File via POST /api/files (up to 2 weeks storage)
• Starting a structured extraction Job via POST /api/structured-extraction/{structuredProjectId}/jobs
• Reading a structured extraction Job result via GET /api/structured-extraction/jobs/{structuredExtractionJobId}
• Adding an Example to a Structured Extraction Project via POST /api/structured-extraction/{structuredProjectId}/examples

Content extraction

• Creating a Content Extraction Project via POST /api/content-extraction
• Changing settings of a Content Extraction Project via PATCH /api/content-extraction/{contentProjectId}/settings
• Uploading a file to a File via POST /api/files (up to 2 weeks storage)
• Starting a content extraction Job via POST /api/content-extraction/jobs
• Reading a content extraction Job result via GET /api/content-extraction/jobs/{contentExtractionJobId}

This Python package is automatically generated by the OpenAPI Generator project:

• API version:
• Package version: 1.0.0
• Generator version: 7.25.0
• Build package: org.openapitools.codegen.languages.PythonClientCodegen

Documentation for API Endpoints

All URIs are relative to https://nuextract.ai

Documentation For Models

• Content
• ContentExtractionResponse
• ContentProjectResponse
• ContentProjectSettingsResponse
• ConvertRequest
• CreateContentProjectRequest
• CreateOrUpdateStructuredExampleRequest
• CreateStructuredProjectRequest
• DocumentInfo
• DocumentResponse
• Error
• FileResponse
• HealthResponse
• ImageInfo
• ImportProjectResponse
• InferenceStatus
• InferenceValidationError
• InformationResponse
• InvalidInformation
• JobIdResponse
• JobStatusResponse
• PaginatedResponseT
• ServiceStatus
• Structured
• StructuredExampleResponse
• StructuredExtractionResponse
• StructuredInferenceExample
• StructuredProjectResponse
• StructuredProjectSettingsResponse
• TemplateRequest
• TemplateResponse
• TextInfo
• TextRequest
• UpdateContentProjectRequest
• UpdateContentProjectSettingsRequest
• UpdateStructuredProjectRequest
• UpdateStructuredProjectSettingsRequest
• ValidInformation
• VersionResponse

Documentation For Authorization

Authentication schemes defined for the API:

oauth2Auth

• Type: OAuth
• Flow: accessCode
• Authorization URL: https://users.numind.ai/realms/extract-platform/protocol/openid-connect/auth
• Scopes:
• openid: OpenID connect
• profile: view profile
• email: view email