Synthegrator
Synthegrator is a framework for code generation problems. It simplifies the process of loading common datasets and solving them with language models.
Installation
pip install synthegrator
Also, for execution you will need to install docker.
Example
Let's take a look at an example of how we can run a solver over the HumanEval dataset, which collects 164 function synthesis problems.
# Imports
from lmwrapper.openai_wrapper import get_open_ai_lm, OpenAiModelNames
from synthegrator.code_solver import LmCodeSolverAutoRegressive
from synthegrator.execution_threading import solve_and_evaluate_problems
from synthegrator.synthdatasets.human_eval import yield_human_eval
from synthegrator.df_converters import solution_evals_to_df
# Loading of a selection of AI4SE Datasets
problems = list(yield_human_eval())
# Create a solver that can solve a problem
lm = get_open_ai_lm(OpenAiModelNames.gpt_3_5_turbo_instruct)
# ^ Make sure to add your API key to OPENAI_API_KEY or a file.
# See https://github.com/DaiseyCode/lmwrapper for more.
solver = LmCodeSolverAutoRegressive(lm)
# Generate code and execute problems testcases
evals = list(solve_and_evaluate_problems(
solver=solver,
problems=problems,
max_threads_eval=4,
))
# Convert to a dataframe
df = solution_evals_to_df(
evals,
pickle_gzip_whole_solution_eval=True
)
print("Fraction Passing", df.main_metric__is_success.mean())
Architecture
Guiding Design Requirements
- DR-1 Support Diverse Datasets and Tasks. We want an architecture that can support a diverse tasks (including potentially complex, repository-level tasks).
- DR-2 Consistent & Efficient Execution. Experiments often involve running LLM-generated code. We want this to be fast, efficient, and reasonably secure.
- DR-3 Adaptable to State-of-the-Art Models. This includes models like those from OpenAI or on HuggingFace. Additionally be adaptable to models that might do complex retrieval or reasoning
- DR-4 Maintainable. Try to follow best practices around automated testing and continuous integration.
Diagram
1 maintainer
Metadata
1 maintainer
Release files for synthegrator 0.14.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| synthegrator-0.14.1.0.tar.gz | 3.2 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| synthegrator-0.14.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 6.5 MB
Release files / synthegrator-0.14.1.0.tar.gz
| Download URL | synthegrator-0.14.1.0.tar.gz |
|---|---|
| Size | 3.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
90d3ef154298698a6632e171d2f318a22e26b622ef187baf96b44221bdb7fe3d
|
|
BLAKE2b-256 checksum How to use checksums |
4ff3cfd95cefb5b064e062e380528f751af316aaa609a8adb4e4224a7bdf3deb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.4
|
Release files / synthegrator-0.14.1.0-py3-none-any.whl
| Download URL | synthegrator-0.14.1.0-py3-none-any.whl |
|---|---|
| Size | 3.3 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
aeecdc7fec6e3056eb3b455cdf3c0bacde927c1e7e56ffe93b41755da704c665
|
|
BLAKE2b-256 checksum How to use checksums |
c15f958ab5d8e1a7e769f05b6939f409650029239f47fc9a21177dec8d0525c6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.4
|
