tspace

Overview

tspace is an data pipleline framework for deep reinforcement learning with IO interface, processing and configuration

• Working in training and inferrence mode
  • logging and monitoring with cutelog or TUI interface
  • cascaded threading pool for well-structured Scheduling of ETL and ML pipelines
  • Customized Exception handling
  • Graceful shutdown
  • online and offline training
  • local and distributed training
• Support for multiple models
  • reinforcement learning models with DDPG
  • time sequence models with LSTM and Transformer
• Data pipeline compatible to both ETL and ML dataflow
  • Support for multiple data sources (local CAN or remote cloud object storage)
  • Support both NoSQL database and local or cloud data storage through Dask with Parquet and Avro interface
  • Full Pandas DataFrame support with raw json codecs
  • Configuration system for vehicles, drivers, data sites, neural network hyperparameters, database, HMI types, etc
  • Timezone aware time sequence data processing
  • Data object meta-info processing and storage linked to configuration system
  • Stateful time sequence processing with sequential model
  • Type hint for data processing and configuration
  • Pydantic integration

The diagram shows the basic architure of tspace. The main components are:

• Avatar: orchestrates the whole ETL and ML workflow.
• Data Object: The data object is a container for the data. It contains the raw data, the metainfo and the configuration. The data object is passed through the data pipeline and can be modified by the processing components.
• Data Pipeline: The data pipeline is a sequence of processing components. Each processing component takes a data object as input and returns a data object as output. The data pipeline is responsible for transforming the data object into a format that can be used by the machine learning model.
• Machine Learning Model: The machine learning model is responsible for training and inferencing. It takes the data object as input and returns a prediction as output. The machine learning model can be a reinforcement learning model, a time sequence model or any other type of model.
• Configuration System: The configuration system is responsible for storing and managing the configuration of the data pipeline, the machine learning model and other components. The configuration system is used to configure the data pipeline, the machine learning model and other components.
• Logging and Monitoring: The logging and monitoring component is responsible for logging and monitoring the data pipeline, the machine learning model and other components. It provides real-time feedback on the performance of the data pipeline and the machine learning model.
• Exception Handling: The exception handling component is responsible for handling exceptions that occur during the execution of the data pipeline, the machine learning model and other components. It provides a mechanism for gracefully handling errors and recovering from failures.

TODO

1. Add time sequence embedding database support with LanceDB for TimeGPT
2. Batch mode for large scale inference and training with Unit of Work pattern

How to use

Install

pip install tspace

Fill me in please! Don’t forget code examples:

1+1

2

# say_hi("world")
foo()

'foo'