python-fastdata 0.0.1

Easiest and fastest way to 1B synthetic tokens

fastdata

Developer Guide

If you are new to using nbdev here are some useful pointers to get you started.

Install fastdata in Development mode

# make sure fastdata package is installed in development mode
$ pip install -e .

# make changes under nbs/ directory
# ...

# compile to have changes apply to fastdata
$ nbdev_prepare

Usage

Installation

Install latest from the GitHub repository:

$ pip install git+https://github.com/AnswerDotAI/fastdata.git

or from conda

$ conda install -c AnswerDotAI fastdata

or from pypi

$ pip install fastdata

Documentation

Documentation can be found hosted on this GitHub repository’s pages. Additionally you can find package manager specific guidelines on conda and pypi respectively.

How to use

First you need to define the structure of the data you want to generate. instructor, which is the library that fastdata uses to generate data, requires you to define the schema of the data you want to generate. This is done using pydantic models.

from pydantic import BaseModel, Field

class Translation(BaseModel):
    english: str = Field(description="An english phrase")
    german: str = Field(description="An equivalent german phrase that is a translation of the english phrase")

Next, you need to define the prompt that will be used to generate the data and any inputs you want to pass to the prompt.

prompt_template = """\
Generate English and German translations on the following topic:
{topic}
"""

inputs = [{"topic": "Otters are cute"}, {"topic": "I love programming"}]

Finally, we can generate some data with fastdata.

[!NOTE]

We only support Anthropic models at the moment. Therefore, make sure you have an API key for the model you want to use and the proper environment variables set or pass the api key to the FastData class FastData(api_key="sk-ant-api03-...").

from fastdata.core import FastData

import pprint

# Create a pretty printer object with custom settings
pp = pprint.PrettyPrinter(indent=4, width=100, compact=False)

fast_data = FastData()
translations = fast_data.generate(
    prompt_template=prompt_template,
    inputs=inputs,
    response_model=Translation,
    model="claude-3-haiku-20240307"
)

# Pretty print the translations
print("Translations:")
pp.pprint(translations)

100%|██████████| 2/2 [00:00<00:00,  2.21it/s]

Translations:
[   {'english': 'Otters are cute', 'german': 'Otter sind süß'},
    {'english': 'I love programming', 'german': 'Ich liebe das Programmieren'}]

If you’d like to see how best to generate data with fastdata, check out our blog post here and some of the examples in the examples directory.