Skip to main content

Chroma Datasets

Making it easy to load data into Chroma since 2023

pip install chroma_datasets

Current Datasets

Dataset Size Contributor Python Class
State of the Union 51kb Chroma from chroma_datasets import StateOfTheUnion
Paul Graham Essay 1.3mb Chroma from chroma_datasets import PaulGrahamEssay
SciQ 2.8mb Hugging Face from chroma_datasets import SciQ
Huberman Podcasts 4.3mb Dexa AI from chroma_datasets import HubermanPodcasts

chroma_datasets is generally backed by hugging face datasets, but it is not a requirement.

How to use

The following will:

  1. Download the 2022 State of the Union
  2. Chunk it up for you
  3. Embed it using Chroma's default open-source embedding function
  4. Import it into Chroma
import chromadb
from chroma_datasets import StateOfTheUnion
from chroma_datasets.utils import import_into_chroma

chroma_client = chromadb.Client()
collection = import_into_chroma(chroma_client=chroma_client, dataset=StateOfTheUnion)
result = collection.query(query_texts=["The United States of America"])
print(result)

Adding a New Dataset

We welcome new datasets!

These datasets can be anything generally useful to developer education for processing and using embeddings.

Datasets can be:

  • raw text (like StateOfTheUnion)
  • pre-chunked data (like SciPy)
  • chunked and embedded (like PaulGrahamEssay)

See the examples/upload.ipynb for an example of how to create a dataset on Hugging Face (the default path)

Create a new dataset from a Chroma Collection

(more examples of this in examples/upload.ipynb and examples/upload_embeddings.ipynb)

Install dependencies

pip install datasets huggingface_hub chromadb

Login into Hugging Face

huggingface-cli login

Upload an existing collection to Hugging Face ** Hugging Face requires the data to have a "split name" - I suggest using a default of "data" **

import chromadb
from chroma_datasets.utils import export_collection_to_hf_dataset
client = chromadb.PersistantClient(path="./chroma_data")
dataset = export_collection_to_hf_dataset(
    client=client, 
    collection_name="paul_graham_essay", 
    license="MIT")
dataset.push_to_hub(
    repo_id="chromadb/paul_graham_essay", 
    split="data")

Create a Dataset Class

  • Set the string name of the embedding function you used to embed the data, this will make it possible for users to use the embeddings. Please also customize the helpful error message so if users pass in no embedding function or the wrong one, they get help.
  • raw_text is optional
  • chunked and to_chroma you can copy letter for letter if you uploaded with the method above.
class PaulGrahamEssay(Dataset):
    """
    http://www.paulgraham.com/worked.html

    Columns:
        - id: unique identifier for each chunk
        - document: the text of the chunk
        - embedding: the embedding of the chunk (OpenAI-ada-002)
        - metadata: metadata about the chunk
    """
    hf_data = None
    embedding_function = "OpenAIEmbeddingFunction" # name of embedding function inside Chroma
    embedding_function_instructions = """
        from chromadb.utils import embedding_functions
        openai_ef = embedding_functions.OpenAIEmbeddingFunction(
            api_key="YOUR_API_KEY",
            model_name="text-embedding-ada-002"
        )
    """

    @classmethod
    def load_data(cls):
        cls.hf_data = load_huggingface_dataset(
            "chromadb/paul_graham_essay",
            split_name="data"
        )

    @classmethod
    def raw_text(cls) -> str:
        if cls.hf_data is None:
            cls.load_data()
        return "\n".join(cls.hf_data["document"])
    
    @classmethod
    def chunked(cls) -> List[Datapoint]:
        if cls.hf_data is None:
            cls.load_data()
        return cls.hf_data
    
    @classmethod
    def to_chroma(cls) -> AddEmbedding:
        return to_chroma_schema(cls.chunked())

Add it to the manifest at chroma_datasets/__init__.py to make it easy for people to retrieve (optional)

Utility API Documentation

Many of these methods are purely conveneient. This makes it easy to save and load Chroma Collections to disk. See ./examples/example_export.ipynb for example use.

from chromadb.utils import (
    export_collection_to_hf_dataset,
    export_collection_to_hf_dataset_to_disk,
    import_chroma_exported_hf_dataset_from_disk,
    import_chroma_exported_hf_dataset
)

# Exports a Chroma collection to an in-memory HuggingFace Dataset
def export_collection_to_hf_dataset(chroma_client, collection_name, license="MIT"):

# Exports a Chroma collection to a HF dataset and saves to the path
def export_collection_to_hf_dataset_to_disk(chroma_client, collection_name, path, license="MIT"):

# Imports a HuggingFace Dataset into a Chroma Collection
def import_chroma_exported_hf_dataset(chroma_client, dataset, collection_name, embedding_function=None):

# Imports a HuggingFace Dataset from Disk and loads it into a Chroma Collection
def import_chroma_exported_hf_dataset_from_disk(chroma_client, path, collection_name, embedding_function=None):

Todo

  • Add test suite to test some of the critical paths
  • Add automated pypi release
  • Add loaders for other locations (remote like S3, local like CSV... etc)
  • Super easy streamlit/gradio wrapper to push up a collection to interact with

Metadata

Release files for chroma-datasets 0.1.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for chroma-datasets 0.1.5
File Size Uploaded Provenance Actions
chroma_datasets-0.1.5.tar.gz 160.4 kB Without provenance Details for chroma_datasets-0.1.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for chroma-datasets 0.1.5
File Interpreter ABI Platform Actions
chroma_datasets-0.1.5-py3-none-any.whl Python 3 none any Details for chroma_datasets-0.1.5-py3-none-any.whl

Total release size: 176.0 kB

Release files / chroma_datasets-0.1.5.tar.gz

Download URL chroma_datasets-0.1.5.tar.gz
Size 160.4 kB
Tags Source
SHA-256 checksum
How to use checksums
4724c501b5d40bbb3868648036d7957bcd086eae9cca41e7d775c6f53de67fa1
BLAKE2b-256 checksum
How to use checksums
fbe7a550e1da7b5ecbe94df310a5b2c8bb06a03ef533930b29728d568076a276
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.10

Release files / chroma_datasets-0.1.5-py3-none-any.whl

Download URL chroma_datasets-0.1.5-py3-none-any.whl
Size 15.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
386f68978f7ffe59b31a55d98d961fc205e459d55e2a57022e05ad668430d7dc
BLAKE2b-256 checksum
How to use checksums
048741b4383f64c870c7d7b501a7cb0a5220dffea1c9f587511191e2595e5c0b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.10

Security for chroma-datasets 0.1.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page