Skip to main content

Synthetic dataset insights.

Project description

Dataset Insights

PyPI python PyPI version Downloads Tests License

Unity Dataset Insights is a python package for downloading, parsing and analyzing synthetic datasets generated using the Unity Perception package.

Installation

Dataset Insights maintains a pip package for easy installation. It can work in any standard Python environment using pip install datasetinsights command.

Getting Started

Dataset Statistics

We provide a sample notebook to help you load synthetic datasets generated using Perception package and visualize dataset statistics. We plan to support other sample Unity projects in the future.

Dataset Download

You can download the datasets from HTTP(s), GCS, and Unity simulation projects using the 'download' command from CLI or API.

CLI

datasetinsights download \
  --source-uri=<xxx> \
  --output=$HOME/data

Programmatically

UnitySimulationDownloader downloads a dataset from Unity Simulation.

from datasetinsights.io.downloader import UnitySimulationDownloader

source_uri=usim://<project_id>/<run_execution_id>
dest = "~/data"
access_token = "XXX"
downloader = UnitySimulationDownloader(access_token=access_token)
downloader.download(source_uri=source_uri, output=dest)

GCSDatasetDownloader downloads a dataset from GCS location.

from datasetinsights.io.downloader import GCSDatasetDownloader

source_uri=gs://url/to/file.zip or gs://url/to/folder
dest = "~/data"
downloader = GCSDatasetDownloader()
downloader.download(source_uri=source_uri, output=dest)

HTTPDatasetDownloader downloads a dataset from any HTTP(S) location.

from datasetinsights.io.downloader import HTTPDatasetDownloader

source_uri=http://url.to.file.zip
dest = "~/data"
downloader = HTTPDatasetDownloader()
downloader.download(source_uri=source_uri, output=dest)

Dataset Explore

You can explore the dataset schema by using following API:

Unity Perception

AnnotationDefinitions and MetricDefinitions loads synthetic dataset definition tables and return a dictionary containing the definitions.

from datasetinsights.datasets.unity_perception import AnnotationDefinitions,
MetricDefinitions
annotation_def = AnnotationDefinitions(data_root=dest, version="my_schema_version")
definition_dict = annotation_def.get_definition(def_id="my_definition_id")

metric_def = MetricDefinitions(data_root=dest, version="my_schema_version")
definition_dict = metric_def.get_definition(def_id="my_definition_id")

Captures loads synthetic dataset captures tables and return a pandas dataframe with captures and annotations columns.

from datasetinsights.datasets.unity_perception import Captures
captures = Captures(data_root=dest, version="my_schema_version")
captures_df = captures.filter(def_id="my_definition_id")

Metrics loads synthetic dataset metrics table which holds extra metadata that can be used to describe a particular sequence, capture or annotation and return a pandas dataframe with captures and metrics columns.

from datasetinsights.datasets.unity_perception import Metrics
metrics = Metrics(data_root=dest, version="my_schema_version")
metrics_df = metrics.filter_metrics(def_id="my_definition_id")

Docker

You can use the pre-build docker image unitytechnologies/datasetinsights to run similar commands.

Documentation

You can find the API documentation on readthedocs.

Contributing

Please let us know if you encounter a bug by filing an issue. To learn more about making a contribution to Dataset Insights, please see our Contribution page.

License

Dataset Insights is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

Citation

If you find this package useful, consider citing it using:

@misc{datasetinsights2020,
    title={Unity {D}ataset {I}nsights Package},
    author={{Unity Technologies}},
    howpublished={\url{https://github.com/Unity-Technologies/datasetinsights}},
    year={2020}
}

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for datasetinsights, version 1.1.1
Filename, size File type Python version Upload date Hashes
Filename, size datasetinsights-1.1.1-py3-none-any.whl (1.5 MB) File type Wheel Python version py3 Upload date Hashes View
Filename, size datasetinsights-1.1.1.tar.gz (1.5 MB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page