Synthetic dataset insights.
Project description
Dataset Insights
Unity Dataset Insights is a python package for downloading, parsing and analyzing synthetic datasets generated using the Unity Perception package.
Installation
Dataset Insights maintains a pip package for easy installation. It can work in any standard Python environment using pip install datasetinsights
command.
Getting Started
Dataset Statistics
We provide a sample notebook to help you load synthetic datasets generated using Perception package and visualize dataset statistics. We plan to support other sample Unity projects in the future.
Dataset Download
You can download the datasets from HTTP(s), GCS, and Unity simulation projects using the 'download' command from CLI or API.
datasetinsights download \
--source-uri=<xxx> \
--output=$HOME/data
UnitySimulationDownloader downloads a dataset from Unity Simulation.
from datasetinsights.io.downloader import UnitySimulationDownloader
source_uri=usim://<project_id>/<run_execution_id>
dest = "~/data"
access_token = "XXX"
downloader = UnitySimulationDownloader(access_token=access_token)
downloader.download(source_uri=source_uri, output=dest)
GCSDatasetDownloader downloads a dataset from GCS location.
from datasetinsights.io.downloader import GCSDatasetDownloader
source_uri=gs://url/to/file.zip or gs://url/to/folder
dest = "~/data"
downloader = GCSDatasetDownloader()
downloader.download(source_uri=source_uri, output=dest)
HTTPDatasetDownloader downloads a dataset from any HTTP(S) location.
from datasetinsights.io.downloader import HTTPDatasetDownloader
source_uri=http://url.to.file.zip
dest = "~/data"
downloader = HTTPDatasetDownloader()
downloader.download(source_uri=source_uri, output=dest)
Dataset Explore
You can explore the dataset schema by using following API:
AnnotationDefinitions and MetricDefinitions loads synthetic dataset definition tables and return a dictionary containing the definitions.
from datasetinsights.datasets.unity_perception import AnnotationDefinitions,
MetricDefinitions
annotation_def = AnnotationDefinitions(data_root=dest, version="my_schema_version")
definition_dict = annotation_def.get_definition(def_id="my_definition_id")
metric_def = MetricDefinitions(data_root=dest, version="my_schema_version")
definition_dict = metric_def.get_definition(def_id="my_definition_id")
Captures loads synthetic dataset captures tables and return a pandas dataframe with captures and annotations columns.
from datasetinsights.datasets.unity_perception import Captures
captures = Captures(data_root=dest, version="my_schema_version")
captures_df = captures.filter(def_id="my_definition_id")
Metrics loads synthetic dataset metrics table which holds extra metadata that can be used to describe a particular sequence, capture or annotation and return a pandas dataframe with captures and metrics columns.
from datasetinsights.datasets.unity_perception import Metrics
metrics = Metrics(data_root=dest, version="my_schema_version")
metrics_df = metrics.filter_metrics(def_id="my_definition_id")
Docker
You can use the pre-build docker image unitytechnologies/datasetinsights to run similar commands.
Documentation
You can find the API documentation on readthedocs.
Contributing
Please let us know if you encounter a bug by filing an issue. To learn more about making a contribution to Dataset Insights, please see our Contribution page.
License
Dataset Insights is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.
Citation
If you find this package useful, consider citing it using:
@misc{datasetinsights2020,
title={Unity {D}ataset {I}nsights Package},
author={{Unity Technologies}},
howpublished={\url{https://github.com/Unity-Technologies/datasetinsights}},
year={2020}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for datasetinsights-1.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7769afcf9f38e60f49bad47a955e0ce55a6ced3f9a41f8a6ec47d4f2d26c0db3 |
|
MD5 | e80295ac022a80050e9f83bf3bc24206 |
|
BLAKE2b-256 | 52fd94a0dc9324d3efce97424806d4cfc5fde563d2c2c7928483c8353dccb2e2 |