Skip to main content

Imports: isort CI pre-commit

GPT3DataGen

GPT3DataGen is a python package that generates fake data for fine-tuning your openai models.

               _      ___      _         _
              ( )_  /'_  )    ( )       ( )_
   __   _ _   | ,_)(_)_) |   _| |   _ _ | ,_)   _ _    __     __    ___
 /'_ `\( '_`\ | |   _(_ <  /'_` | /'_` )| |   /'_` ) /'_ `\ /'__`\/' _ `\
( (_) || (_) )| |_ ( )_) |( (_| |( (_| || |_ ( (_| |( (_) |(  ___/| ( ) |
`\__  || ,__/'`\__)`\____)`\__,_)`\__,_)`\__)`\__,_)`\__  |`\____)(_) (_)v1.0.3
( )_) || |                                          ( )_) |
 \___/'(_)                                           \___/'

Install with pip. See Install & Usage Guide

pip install -U gpt3datagen

Alternatively, the following command will pull and install the latest commit from this repository, along with its Python dependencies:

pip install git+https://github.com/donwany/gpt3datagen.git --use-pep517

Or git clone repository:

git clone https://github.com/donwany/gpt3datagen.git
cd gpt3datagen
make install && pip install -e .

To update the package to the latest version of this repository, please run:

pip install --upgrade --no-deps --force-reinstall git+https://github.com/donwany/gpt3datagen.git

Command-Line Usage

Run the following to view all available options:

gpt3datagen --help
gpt3datagen --version

Output formats: jsonl, json, csv, tsv, xlsx

gpt3datagen \
    --num_samples 500 \
    --max_length 2048 \
    --sample_type "classification" \
    --output_format "jsonl" \
    --output_dir .

gpt3datagen \
    --num_samples 500 \
    --max_length 2048 \
    --sample_type completion \
    --output_format csv \
    --output_dir .

gpt3datagen \
    --sample_type completion \
    --output_format jsonl \
    --output_dir .

gpt3datagen --sample_type completion -o . -f jsonl
gpt3datagen --sample_type news -o . -f jsonl

Data Format

{"prompt": "<prompt text> \n\n###\n\n", "completion": " <ideal generated text> END"}
{"prompt": "<prompt text> \n\n###\n\n", "completion": " <ideal generated text> END"}
{"prompt": "<prompt text> \n\n###\n\n", "completion": " <ideal generated text> END"}
                                    ...

Basic Usage

Only useful if you clone the repository

python prepare.py \
    --num_samples 500 \
    --max_length 2048 \
    --sample_type "classification" \
    --output_format "jsonl" \
    --output_dir .

python prepare.py \
    --num_samples 500 \
    --max_length 2048 \
    --sample_type "completion" \
    --output_format "csv" \
    --output_dir .

python prepare.py \
    --num_samples 500 \
    --max_length 2048 \
    --sample_type "completion" \
    --output_format "json" \
    --output_dir /Users/<tsiameh>/Desktop

Validate Sample Data

pip install --upgrade openai

export OPENAI_API_KEY="<OPENAI_API_KEY>"

# validate sample datasets generated
openai tools fine_tunes.prepare_data -f <SAMPLE_DATA>.jsonl
openai tools fine_tunes.prepare_data -f <SAMPLE_DATA>.csv
openai tools fine_tunes.prepare_data -f <SAMPLE_DATA>.tsv
openai tools fine_tunes.prepare_data -f <SAMPLE_DATA>.json
openai tools fine_tunes.prepare_data -f <SAMPLE_DATA>.xlsx
openai tools fine_tunes.prepare_data -f /Users/<tsiameh>/Desktop/data_prepared.jsonl

# fine-tune
openai api fine_tunes.create \
  -t <DATA_PREPARED>.jsonl \
  -m <BASE_MODEL: davinci, curie, ada, babbage>

# List all created fine-tunes
openai api fine_tunes.list

Test Runs

# For multiclass classification
openai api fine_tunes.create \
  -t <TRAIN_FILE_ID_OR_PATH> \
  -v <VALIDATION_FILE_OR_PATH> \
  -m <MODEL> \
  --compute_classification_metrics \
  --classification_n_classes <N_CLASSES>

# For binary classification
openai api fine_tunes.create \
  -t <TRAIN_FILE_ID_OR_PATH> \
  -v <VALIDATION_FILE_OR_PATH> \
  -m <MODEL> \
  --compute_classification_metrics \
  --classification_n_classes 2 \
  --classification_positive_class <POSITIVE_CLASS_FROM_DATASET>

Contribute

Please see CONTRIBUTING.

License

GPT3DataGen is released under the MIT License. See the bundled LICENSE file for details.

Credits

  • Theophilus Siameh

Metadata

Release files for gpt3datagen 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for gpt3datagen 0.1.0
File Size Uploaded Provenance Actions
gpt3datagen-0.1.0.tar.gz 469.9 kB Without provenance Details for gpt3datagen-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for gpt3datagen 0.1.0
File Interpreter ABI Platform Actions
gpt3datagen-0.1.0-py3-none-any.whl Python 3 none any Details for gpt3datagen-0.1.0-py3-none-any.whl

Total release size: 941.0 kB

Release files / gpt3datagen-0.1.0.tar.gz

Download URL gpt3datagen-0.1.0.tar.gz
Size 469.9 kB
Tags Source
SHA-256 checksum
How to use checksums
662e0ef4e84bb360b1a47942897a90986294820cad27f70b835cd485ef1aacbd
BLAKE2b-256 checksum
How to use checksums
0ee5781b09c3e1d85953defb0953301e925eb97b871dd6fe5b8703f16fa5a800
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.1

Release files / gpt3datagen-0.1.0-py3-none-any.whl

Download URL gpt3datagen-0.1.0-py3-none-any.whl
Size 471.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8d2ae55b75b65f37fa7b16d80c3c7a62e131cfa59ae537289102069803e2b9e1
BLAKE2b-256 checksum
How to use checksums
1917e5110f73b80d45c3dd50a02e7a62fb19b247f5370729680aa09b3e7ce721
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.1

Security for gpt3datagen 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page