GPT3DataGen
GPT3DataGen is a python package that generates fake data for fine-tuning your openai models.
_ ___ _ _
( )_ /'_ ) ( ) ( )_
__ _ _ | ,_)(_)_) | _| | _ _ | ,_) _ _ __ __ ___
/'_ `\( '_`\ | | _(_ < /'_` | /'_` )| | /'_` ) /'_ `\ /'__`\/' _ `\
( (_) || (_) )| |_ ( )_) |( (_| |( (_| || |_ ( (_| |( (_) |( ___/| ( ) |
`\__ || ,__/'`\__)`\____)`\__,_)`\__,_)`\__)`\__,_)`\__ |`\____)(_) (_)v1.0.3
( )_) || | ( )_) |
\___/'(_) \___/'
Install with pip. See Install & Usage Guide
pip install -U gpt3datagen
Alternatively, the following command will pull and install the latest commit from this repository, along with its Python dependencies:
pip install git+https://github.com/donwany/gpt3datagen.git --use-pep517
Or git clone repository:
git clone https://github.com/donwany/gpt3datagen.git
cd gpt3datagen
make install && pip install -e .
To update the package to the latest version of this repository, please run:
pip install --upgrade --no-deps --force-reinstall git+https://github.com/donwany/gpt3datagen.git
Command-Line Usage
Run the following to view all available options:
gpt3datagen --help
gpt3datagen --version
Output formats: jsonl, json, csv, tsv, xlsx
gpt3datagen \
--num_samples 500 \
--max_length 2048 \
--sample_type "classification" \
--output_format "jsonl" \
--output_dir .
gpt3datagen \
--num_samples 500 \
--max_length 2048 \
--sample_type completion \
--output_format csv \
--output_dir .
gpt3datagen \
--sample_type completion \
--output_format jsonl \
--output_dir .
gpt3datagen --sample_type completion -o . -f jsonl
gpt3datagen --sample_type news -o . -f jsonl
Data Format
{"prompt": "<prompt text> \n\n###\n\n", "completion": " <ideal generated text> END"}
{"prompt": "<prompt text> \n\n###\n\n", "completion": " <ideal generated text> END"}
{"prompt": "<prompt text> \n\n###\n\n", "completion": " <ideal generated text> END"}
...
Basic Usage
Only useful if you clone the repository
python prepare.py \
--num_samples 500 \
--max_length 2048 \
--sample_type "classification" \
--output_format "jsonl" \
--output_dir .
python prepare.py \
--num_samples 500 \
--max_length 2048 \
--sample_type "completion" \
--output_format "csv" \
--output_dir .
python prepare.py \
--num_samples 500 \
--max_length 2048 \
--sample_type "completion" \
--output_format "json" \
--output_dir /Users/<tsiameh>/Desktop
Validate Sample Data
pip install --upgrade openai
export OPENAI_API_KEY="<OPENAI_API_KEY>"
# validate sample datasets generated
openai tools fine_tunes.prepare_data -f <SAMPLE_DATA>.jsonl
openai tools fine_tunes.prepare_data -f <SAMPLE_DATA>.csv
openai tools fine_tunes.prepare_data -f <SAMPLE_DATA>.tsv
openai tools fine_tunes.prepare_data -f <SAMPLE_DATA>.json
openai tools fine_tunes.prepare_data -f <SAMPLE_DATA>.xlsx
openai tools fine_tunes.prepare_data -f /Users/<tsiameh>/Desktop/data_prepared.jsonl
# fine-tune
openai api fine_tunes.create \
-t <DATA_PREPARED>.jsonl \
-m <BASE_MODEL: davinci, curie, ada, babbage>
# List all created fine-tunes
openai api fine_tunes.list
Test Runs
# For multiclass classification
openai api fine_tunes.create \
-t <TRAIN_FILE_ID_OR_PATH> \
-v <VALIDATION_FILE_OR_PATH> \
-m <MODEL> \
--compute_classification_metrics \
--classification_n_classes <N_CLASSES>
# For binary classification
openai api fine_tunes.create \
-t <TRAIN_FILE_ID_OR_PATH> \
-v <VALIDATION_FILE_OR_PATH> \
-m <MODEL> \
--compute_classification_metrics \
--classification_n_classes 2 \
--classification_positive_class <POSITIVE_CLASS_FROM_DATASET>
Contribute
Please see CONTRIBUTING.
License
GPT3DataGen is released under the MIT License. See the bundled LICENSE file for details.
Credits
Theophilus Siameh
Metadata
Release files for gpt3datagen 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| gpt3datagen-0.1.0.tar.gz | 469.9 kB | Without provenance | Details for gpt3datagen-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| gpt3datagen-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for gpt3datagen-0.1.0-py3-none-any.whl |
Total release size: 941.0 kB
Release files / gpt3datagen-0.1.0.tar.gz
| Download URL | gpt3datagen-0.1.0.tar.gz |
|---|---|
| Size | 469.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
662e0ef4e84bb360b1a47942897a90986294820cad27f70b835cd485ef1aacbd
|
|
BLAKE2b-256 checksum How to use checksums |
0ee5781b09c3e1d85953defb0953301e925eb97b871dd6fe5b8703f16fa5a800
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.1
|
Release files / gpt3datagen-0.1.0-py3-none-any.whl
| Download URL | gpt3datagen-0.1.0-py3-none-any.whl |
|---|---|
| Size | 471.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8d2ae55b75b65f37fa7b16d80c3c7a62e131cfa59ae537289102069803e2b9e1
|
|
BLAKE2b-256 checksum How to use checksums |
1917e5110f73b80d45c3dd50a02e7a62fb19b247f5370729680aa09b3e7ce721
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.1
|
Security for gpt3datagen 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.