Dataverse SDK For Python

Dataverse is a MLOPs platform for assisting in data selection, data visualization and model training in computer vision. Use Dataverse-SDK for Python to help you to interact with the Dataverse platform by Python. Currently, the library supports:

• Create Project with your input ontology and sensors
• Get Project by project-id
• Create Dataset from your AWS storage or local
• Get Dataset by dataset-id
• List models for your selected project-id
• Get and download your model

Package (PyPi) | Source code

Getting started

Install the package

pip install dataverse-sdk

Prerequisites: You must have an Dataverse Platform Account and Python 3.10+ to use this package.

Create the client

Interaction with the Dataverse site starts with an instance of the DataverseClient class. You need site url, an email-account and its password to instantiate the client object.

from dataverse_sdk import *
from dataverse_sdk.connections import get_connection
from dataverse_sdk.constants import DataverseHost

client = DataverseClient(
    host=DataverseHost.PRODUCTION.value, email="XXX", password="***", service_id="xxxx-xxxx-xx-xxx", alias="default", force = False
)
assert client is get_connection("default")

# Should provide different alias if you are trying to connect to different workspaces
client2 = DataverseClient(
    host=DataverseHost.PRODUCTION.value, email="account-2", password="***", service_id="xxxx-xxxx-xx-xxx", alias="client2", force = False
)
assert client2 is get_connection(client2.alias)

client3 = DataverseClient(
    host=DataverseHost.PRODUCTION.value, email="XXX", password="", service_id="xxxx-xxxx-xx-xxx", access_token="xxx", alias="client3", force = False
)
assert client3 is get_connection(client3.alias)

• Input arguments:

Key concepts

Once you've initialized a DataverseClient, you can interact with Dataverse from the initialized object.

Examples

The following sections provide examples for the most common DataVerse tasks including:

• Get User
• List Projects
• Create Project
• Get Project
• Edit Project
• Update Alias
• Create Dataset
• List Dataset
• List Dataslices
• Export Dataslice
• List Models
• Get and Download Model
• Create VQA Project
• Edit VQA Ontology
• Get Question List

Get User

The get_user method is to list the current user info. You can get the detail info, such as role, permission and user detail.

user = client.get_user()

List Projects

The list_projects method will list all projects of the given sites.

• Example Usage:

projects = client.list_projects(current_user = True,
                                exclude_sensor_type=SensorType.LIDAR,
                                image_type=OntologyImageType._2D_BOUNDING_BOX)

• Input arguments:

Create Project

The create_project method will create project on the connected site with the defined ontology and sensors.

• Example Usage:

# 1) Create ontology with ontologyclass object
ontology = Ontology(
    name="sample ontology",
    image_type=OntologyImageType._2D_BOUNDING_BOX,
    pcd_type = None,
    classes=[
        OntologyClass(name="Pedestrian", rank=1, color="#234567"),
        OntologyClass(name="Truck", rank=2, color="#345678"),
        OntologyClass(name="Car", rank=3, color="#456789"),
        OntologyClass(name="Cyclist", rank=4, color="#567890"),
        OntologyClass(name="DontCare", rank=5, color="#6789AB"),
        OntologyClass(name="Misc", rank=6, color="#789AB1"),
        OntologyClass(name="Van", rank=7, color="#89AB12"),
        OntologyClass(name="Tram", rank=8, color="#9AB123"),
        OntologyClass(name="Person_sitting", rank=9, color="#AB1234"),
    ],
)

For project with camera sensor, there would be only one image_type for one project. You could choose from [OntologyImageType._2D_BOUNDING_BOX, OntologyImageType.SEMANTIC_SEGMENTATION, OntologyImageType.INSTANCE_SEGMENTATION, OntologyImageType.CLASSIFICATION, OntologyImageType.POINT, OntologyImageType.POLYGON, OntologyImageType.POLYLINE, OntologyImageType.VQA].

For project with lidar sensor, your should assign pcd_type = OntologyPcdType.CUBOID for the ontology.

# 2) Create your sensor list with name / SensorType
sensors = [
    Sensor(name="camera1", type=SensorType.CAMERA),
    Sensor(name="lidar1", type=SensorType.LIDAR),
]

# 3) Create your project tag attributes (Optional)
project_tag = ProjectTag(
    attributes=[
        {"name": "year", "type": "number"},
        {
            "name": "unknown_object",
            "type": "option",
            "options": [{"value": "fire"}, {"value": "leaves"}, {"value": "water"}],
        },
    ]
)

# 4) Create your project with your ontology/sensors/project_tag
project = client.create_project(name="Sample project", ontology=ontology, sensors=sensors, project_tag=project_tag)

• Input arguments for creating project:

＊--: required argument without default

• Check https://linkervision.gitbook.io/dataverse/data-management/project-ontology for the detail of Project Ontology

Get Project

The get_project method retrieves the project from the connected site. The project_id parameter is the unique integer ID of the project, not its "name" property.

project = client.get_project(project_id= 1, client_alias=client.alias) # if client_alias is not provided, we'll get it from client

Edit Project

For editing project contents, we have four functions below for add/edit project tag and ontology classes.

Add New Project Tags

• Note: Can not create existing project tag!

tag = {
        "attributes": [
            {
                "name": "month",
                "type": "number"
            },
            {
                "name": "weather",
                "type": "option",
                "options": [{"value":"sunny"}, {"value":"rainy"}, {"value":"cloudy"}
                ]
            }]}
project_tag= ProjectTag(**tag)
#should provided client_alias if calling from client
client.add_project_tag(project_id = 10, project_tag=project_tag, client_alias=client.alias)
#OR
project.add_project_tag(project_tag=project_tag)

Edit Project Tags

** Note:

1. Can not edit project tag that does not exist
2. Can not modify the data type of existing project tags
3. Can not provide attributes with existing options

tag = {
        "attributes": [
            {
                "name": "weather",
                "type": "option",
                "options": [{"value":"unknown"}, {"value":"snowy"}
                ]
            }]}
project_tag= ProjectTag(**tag)
#should provided client_alias if calling from client
client.edit_project_tag(project_id = 10, project_tag=project_tag, client_alias=client.alias)
#OR
project.edit_project_tag(project_tag=project_tag)

Add New Ontology Classes

• Note: Can not add existing ontology class!

new_classes = [OntologyClass(name="obstruction",
                    rank=9,
                    color="#AB4321",
                    attributes=[{
                    "name":
                    "status",
                    "type":
                    "option",
                    "options": [{
                    "value": "static"}, {"value": "moving"
                    }]}])]
#should provided client_alias if calling from client
client.add_ontology_classes(project_id=24, ontology_classes=new_classes, client_alias=client.alias)
#OR
project.add_ontology_classes(ontology_classes=new_classes)

Edit Ontology Classes

** Note:

1. Can not edit ontology class that does not exist
2. Can not modify the data type of existing ontology class attributes
3. Can not provide attributes with existing options

edit_classes = [OntologyClass(name="obstruction",
                    color="#AB4321",
                    attributes=[{
                    "name":
                    "status",
                    "type":
                    "option",
                    "options": [{
                    "value": "unknown"}]}])]
#should provided client_alias if calling from client
client.edit_ontology_classes(project_id=24, ontology_classes=edit_classes, client_alias=client.alias)
#OR
project.edit_ontology_classes(ontology_classes=edit_classes)

Update Ontology Alias

1. Get the csv file of alias map for your project

client.generate_alias_map(project_id=123, alias_file_path="./alias.csv")

2. Fill the alias in the csv file and save (DO NOT modify other fields)

3. Update alias for your project with the alias file path

client.update_alias(project_id=123, alias_file_path= "/Users/Downloads/alias.csv" )

Create Dataset

Required fields by data_source:

＊--: required for this data_source · -: not used (can be omitted)

DataSource.DATA_GENERATION and DataSource.PRE_IMPORT require a data_source_search_body that the SDK does not currently send, so they are not supported via create_dataset.

DataSource.EXISTING_DATASETS / EXISTING_DATASLICES only appear on datasets you read back from the API; they are not inputs for create_dataset.

Use create_dataset to import dataset from cloud storage

dataset_data = {
    "name": "Dataset 1",
    "data_source": DataSource.AWS,
    "storage_url": "storage/url",
    "data_folder": "datafolder/to/vai_anno",
    "type": DatasetType.ANNOTATED_DATA,
    "annotation_format": AnnotationFormat.VISION_AI,
    "annotations": ["groundtruth"],
    "sequential": False,
    "render_pcd": False,
    "generate_metadata": False,
    "access_key_id": "aws s3 access key id",  # only for private s3 bucket, don't need to assign it in case of public s3 bucket
    "secret_access_key": "aws s3 secret access key",  # only for private s3 bucket, don't need to assign it in case of public s3 bucket
}
dataset = project.create_dataset(**dataset_data)

• Input arguments for creating dataset from cloud storage:

＊--: required for cloud storage

• Check https://linkervision.gitbook.io/dataverse/data-management/import-dataset for the detail of Import Dataset.

Use create_dataset to import dataset from LOCAL

dataset_data2 = {
    "name": "dataset-local-upload",
    "data_source": DataSource.LOCAL,
    "data_folder": "/YOUR/TARGET/LOCAL/FOLDER",
    "type": DatasetType.ANNOTATED_DATA, # or DatasetType.RAW_DATA for images
    "annotation_format": AnnotationFormat.VISION_AI,
    "annotations": ["groundtruth"],  # remove it when type is DatasetType.RAW_DATA
    "sequential": False,
    "generate_metadata": False,
}
dataset2 = project.create_dataset(**dataset_data2)

Your could also use the script for importing dataset from local

python tools/import_dataset_from_local.py -host {YOUR_HOST} -e {your-account-email} -p {PASSWORD} -s {service-id}  -project {project-id} --folder {/YOUR/TARGET/LOCAL/FOLDER} -name {dataset-name} -type {raw_data OR annotated_data} -anno {image OR vision_ai} --sequential

List and Get Dataset

The list_datasets method would return the list of dataset under the given project

project = client.get_project(project_id=1)
datasets:list = project.list_datasets()

OR

datasets:list = client.list_datasets(project_id=1, client_alias=client.alias )

The get_dataset method retrieves the dataset info from the connected site. The dataset_id parameter is the unique integer ID of the dataset, not its "name" property.

dataset = client.get_dataset(dataset_id=5)

List and Get Dataslices

# list dataslices with project_id
client.list_dataslices(project_id=101, client_alias=client.alias)

# Get target dataslice data
dataslice_data = client.get_dataslice(dataslice_id=504)

Export Dataslice and Download

# Trigger export and get export record id
export_record = client.export_dataslice(dataslice_id=504)
# Use export record id to download export data
client.download_export_dataslice_data(dataslice_id=504, export_record_id=export_record["export_record_id"])

List Models

The list_models method will list all the models in the given project. You can filter models by type using the type parameter.

Basic Usage

# Method 1: Using client
models = client.list_models(project_id=1, client_alias=client.alias)

# Method 2: Using project object
project = client.get_project(project_id=1)
models = project.list_models()

Filtering by Model Type

You can filter models by type using strings or lists of strings. The SDK supports multiple model types:

# Filter by single type using string
models = client.list_models(project_id=1, type="trained", client_alias=client.alias)

# Filter by single type using list
models = client.list_models(project_id=1, type=["trained"], client_alias=client.alias)

# Filter by multiple types using list
models = client.list_models(
    project_id=1,
    type=["trained", "byom", "uploaded"],
    client_alias=client.alias
)

Available Model Types

Input Arguments

＊--: required argument without default

Get Model

The get_model method will get the model detail info by the given model-id

model = client.get_model(model_id=30, client_alias=client.alias)
model = project.get_model(model_id=30)

From the given model, we could get the model convert records as below

model_record = client.get_convert_record(convert_record_id=1, client_alias=client.alias)
OR
model_record = model.get_convert_record(convert_record_id=1)

• If the converted model format is onnx, you could download the model as below.

# Get the target convert record, and download labels.txt and model.onnx
model_record = model.get_convert_record(convert_record_id=5)
status, label_file_path = model_record.get_label_file(save_path="./labels.txt", timeout=6000)
status, onnx_model_path = model_record.get_onnx_model_file(save_path="./model.onnx", timeout=6000)

Create VQA Project

The create_vqa_project method will create project on the connected site with the defined questions/answer_type.

• Example Usage:

# 1) Create question class with question and answer type pair
question_answer = [ QuestionClass(class_name="question1", rank=1, question="Is any person found in the picture?",
                    answer_type="boolean"),
                    QuestionClass(class_name="question2", rank=2, question="What is the blob color of traffic light?",   answer_type="option",answer_options=["red","yellow","green"])
                   ]

# 2) Create your VQA project as below
project = client.create_vqa_project(name="vqa-project", sensor_name="camera1", ontology_name="vqa-ontology", question_answer=question_answer)

• Input arguments for creating project:

＊--: required argument without default

Edit VQA Ontology

** Note:

1. Can not edit question answer type
2. Can not update with existing answer options
3. Can not add question with existing rank id

create_questions = [QuestionClass(class_name="question3", rank=3, question="Age?",answer_type="number")]
update_questions = [{"rank": 2, "question": "What is the blob color of traffic light?(the closet one)", "options":["black"] }]

# Through client
client.edit_vqa_ontology(project_id=24, ontology_name="ontology-new-name",
                         create=create_questions,
                         update=update_questions,
                         client_alias=client.alias)

# Through project object
project.edit_vqa_ontology(ontology_name="ontology-new-name",
                          create=create_questions,
                          update=update_questions)

Get Question List

The function below could help you get the question list of VQA project (which could help you to prepare the annotated data)

output = client.get_question_list(project_id=107, output_file_path="./question.json" )

Quick Tools

Import Your Local Dataset

python tools/import_dataset_from_local.py -host {YOUR_HOST} -e {your-account-email} -p {PASSWORD} -s {service-id}  -project {project-id} --folder {/YOUR/TARGET/LOCAL/FOLDER} -name {dataset-name} -type {raw_data OR annotated_data} -anno {image OR vision_ai} --sequential

Import VQA Local Dataset

python tools/import_vqa_dataset.py -host {YOUR_HOST} -e {your-account-email} -p {PASSWORD} -s {service-id} -project {project-id} --folder {/YOUR/TARGET/LOCAL/FOLDER} -type {raw_data OR annotated_data}

Export Dataslice and download files

python tools/export_dataslice.py -host {YOUR_HOST}  -e {your-account-email} -p {PASSWORD} -s {service-id} -dataslice {dataslice_id} -f {/YOUR/TARGET/LOCAL/file.zip}

Export Large Dataslice and download files

python tools/export_dataslice_large.py -host {YOUR_HOST} -e {your-account-email} -p {PASSWORD} -s {service-id} -dataslice {dataslice_id} --anno {export-model-name / groundtruth} --target_folder {folder path} --export-format {coco, visionai, yolo, vlm ...etc}

Upload videos to create session tasks

python tools/upload_videos_create_session.py -host {YOUR_HOST} -e {your-account-email} -p {PASSWORD} -s {service-id} -f {/YOUR/VIDEOS/LOCAL/FOLDER} -n {session-name}

• Advanced arguments for video curation (sequential data):

Links to language repos

Python Readme