space-datasets 0.0.4

Unified storage framework for machine learning datasets

Space: Unified Storage for Machine Learning

Unify data in your entire machine learning lifecycle with Space, a comprehensive storage solution that seamlessly handles data from ingestion to training.

Key Features:

• Ground Truth Database
  • Store and manage data in open source file formats, locally or in the cloud.
  • Ingest from various sources, including ML datasets, files, and labeling tools.
  • Support data manipulation (append, insert, update, delete) and version control.
• OLAP Database and Lakehouse
  • Iceberg style open table format.
  • Optimized for unstructued data via reference operations.
  • Quickly analyze data using SQL engines like DuckDB.
• Distributed Data Processing Pipelines
  • Integrate with processing frameworks like Ray for efficient data transformation.
  • Store processed results as Materialized Views (MVs); incrementally update MVs when the source is changed.
• Seamless Training Framework Integration
  • Access Space datasets and MVs directly via random access interfaces.
  • Convert to popular ML dataset formats (e.g., TFDS, HuggingFace, Ray).