DVCx
Project description
What is DVCx?
DVCx is a Python data manipulation library designed to work with unstructured AI datasets. It provides a dataframe-like interface which can automatically reference data stored as files (text, images, video) locally or in the cloud.
Why use DVCx?
Storage as a single source of truth. DVCx can organize unstructured data from storages and datalakes (local files, S3, GCS, Azure ADLS) into overlapping datasets without unnecessary file copies.
Compute. DVCx supports local parallelization and external compute workers for efficient data processing and AI metadata creation.
Large scale. In contrast to in-memory frameworks (like Pandas data frame), DVCx can work with datasets of millions and billions of records by using out-of-memory algorithms.
Persistence and versioning. Your datasets, your computed metadata, and paid API call results remain versioned and reusable.
Installation
You can install DVCx via pip from PyPI:
$ pip install dvcx
Usage
DVCx can be used as a CLI (from system terminal), or as a Python library.
TODO: CLI usage
To use it from Python code, import class dvcx.catalog.Catalog, which provides methods for all the same commands above, like ls(), get(), find(), du() and index().
from dvcx.catalog import Catalog
catalog = Catalog()
catalog.ls(["gcs://dvcx-datalakes/dogs-and-cats/"], update=True)
Contributing
Contributions are very welcome. To learn more, see the Contributor Guide.
License
Distributed under the terms of the Apache 2.0 license, DVCx is free and open source software.
Issues
If you encounter any problems, please file an issue along with a detailed description.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.