Python Dataware Toolkit

A Python toolkit for managing, retrieving, and processing data.

Installation

You can install the toolkit with:

$ pip3 install pydtk

If you want to install the toolkit with extra feature (e.g. support for PointCloud and ROS), you can install it with extra dependencies as follows:

$ pip3 install pydtk[pointcloud,ros]

Some PyDTK models require additional packages.
Please refer the following table and install them manually with command pip install ....

PyDTK model	Required packages
rosbag.*	ros_numpy (https://github.com/eric-wieser/ros_numpy.git)
pointcloud.PCDModel	pypcd (https://github.com/klintan/pypcd.git)

Usage

By using Pydtk, you can load a variety of types of data with a unified interface as shown below.

Load DBHandler for retrieving metadata

from pydtk.db import DBHandler

# Initialize handler (This will read all the metadata from DB on initialization)
handler = DBHandler(
    db_class='meta',
    db_host='./examples/example_db',
    base_dir_path='./test'
)

Read metadata from db with data selection.

# Select by timestamp
handler.read(pql='start_timestamp > 1420000000 and end_timestamp < 1500000000')
print(handler.data)

# Select by record-id
handler.read(pql='record_id == regex("test.*")')
print(handler.data)

Load data from files based on metadata.

from pydtk.io import BaseFileReader, NoModelMatchedError

reader = BaseFileReader()

try:
    for sample in handler:
        print('loading content "{0}" from file "{1}"'.format(sample['contents'], sample['path']))
        try:
            timestamps, data, columns = reader.read(sample)
            assert print(data)
        except NoModelMatchedError as e:
            print(str(e))
            continue
except EOFError:
    pass

Documentation

For more information about this toolkit, please refer the document.

Setup for contribution

To improve this toolkit, firstly clone this repository and then run the following command to prepare the environment.

$ git clone git@github.com:dataware-tools/pydtk.git --recurse-submodules
$ poetry install

Make sure that poetry is installed before executing the command.

If you want to install the toolkit with extra feature (e.g. support for ROS), please specify it with -E option.
Example (installation with pointcloud and ros extras):

$ poetry install -E pointcloud -E ros

Metadata

Release files for pydtk 0.3.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pydtk 0.3.2
File	Size	Uploaded
pydtk-0.3.2.tar.gz	102.1 kB	Sep 7, 2023	Details

Built distribution (wheel)

Table of built distributions (wheels) for pydtk 0.3.2
File	Interpreter	ABI	Platform	Reset
pydtk-0.3.2-py3-none-any.whl 163.4 kB Sep 7, 2023	Python 3	none	any	Details

Total release size: 265.5 kB

Release files / pydtk-0.3.2.tar.gz

Download URL	pydtk-0.3.2.tar.gz
Size	102.1 kB
Tags	Source
SHA-256 checksum How to use checksums	`fb44cb30f2a1ec08e2d68a403ca80e3abbd9cc4bb148393431bab427126d3cf0`
BLAKE2b-256 checksum How to use checksums	`110baaed671685c0d75c23321834c06f8f60e6e5110a55a52b9b58f15aaf0138`
Upload date	Sep 7, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.5.0 CPython/3.9.18 Linux/5.15.0-1045-azure`

Release files / pydtk-0.3.2-py3-none-any.whl

Download URL	pydtk-0.3.2-py3-none-any.whl
Size	163.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`df618835b309baa4c8b5bd74aed63e63a04a394c08ec04ea131b933360831a12`
BLAKE2b-256 checksum How to use checksums	`40e5405d25f4605dd8de18e3ee4d9be170aec34b6aab32fdb2776bb77b453cca`
Upload date	Sep 7, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.5.0 CPython/3.9.18 Linux/5.15.0-1045-azure`