omni-binding-api 0.0.1

LLM-powered omni api

OMNI

Generate an API on-the-fly with LLMs.

---

Usage

Create an omni context:

from omni import Omni

o = Omni() # default is anthropic/claude-3-5-sonnet-20241022

or specify a particular model

o = Omni(model='openai/gpt-4o')

then execute any code:

o.execute('''
# any code here
''')

Make sure the associated API keys are set for the model you specify. See a list of available models here: https://docs.litellm.ai/docs/providers.

Example 1

Suppose we want to draw some shapes in Jupyter notebook, but we don't know the right API. Let's just define our own on the fly!

Create an omni context:

from omni import Omni

o = Omni()

Invoke any Python code inside the omni context:

o.execute('''
r = Rect(width=4, height=6)
r.set_origin(5, 4)
r.set_color('green')
r.set_rotation(deg=45)

c = Canvas()
c.add(r)
c.draw()
''')

Neither Rect or Canvas is defined, so omni will query an LLM to generate binding code that does something plausible (here it decides to use matplotlib). Then it will rerun this code in that context, in this case successfully drawing and rendering a rectangle:

Now that we have an implementation of Rect and Canvas, we can continue to use them:

o.execute('''
r = Rect(width=4, height=6)
r.set_origin(5, 4)
r.set_color('green')
r.set_rotation(deg=45)

r2 = Rect(width=3, height=3)
r2.set_origin(2, 2)
r2.set_color('red')
r2.set_rotation(deg=10)

c = Canvas()
c.add(r)
c.add(r2)
c.draw()
''')

In this case, the new code did not throw an error. Since we already have a context for Rect and Canvas, we immediately get the following figure without having to query the LLM again:

Let's draw a triangle now:

o.execute('''
r = Rect(width=4, height=6)
r.set_origin(5, 4)
r.set_color('green')
r.set_rotation(deg=45)

r2 = Rect(width=3, height=3)
r2.set_origin(2, 2)
r2.set_color('red')
r2.set_rotation(deg=10)

t = Triangle(width=3, height=5)
t.set_origin(6,6)
t.set_color('blue')

c = Canvas()
c.add(r)
c.add(r2)
c.add(t)
c.draw()
''')

This invocation errors (Triangle is not defined), so we query the LLM to extend our API. Then it reruns and we get a nice triangle:

Now lets add random shapes by using the function add_random_shapes (which doesn't currently exist):

o.execute('''
c = Canvas()
c.add_random_shapes(num=5)
c.draw()
''')

This hits the LLM to figure out what to do and we get:

Of course now we can scale up without needing to query the LLM:

o.execute('''
c = Canvas()
c.add_random_shapes(num=100)
c.draw()
''')

Now let's make an animation and save it as a GIF. I have no idea how to do this so lets just create a new Gif object which we can add frames to, and assume the Canvas has some way to .render into something that can be added to a Gif:

o.execute('''
g = Gif()

for i in range(10):
    c = Canvas()
    c.add_random_shapes(num=100)
    r = c.render()
    g.add_frame(r, ms=20)

g.save('./out.gif')
''')

This queries an LLM and we get back an implementation of these APIs that successfully does the thing:

Example 2

Suppose we want to do some github exploration. I want to look at all the github repos I have and explore the data.

We'll create a new omni context:

o2 = Omni()

Then use the api fetch_github_projects to get my projects:

o2.execute('''
projects = fetch_github_projects(user='hgarrereyn')

for p in projects:
    print(f'{p.name} :: {p.stars}')
''')

2018submissions :: 0
AudioScroll-Extension :: 9
beam :: 0
bn-fish-disassembler :: 10
bn-pokemon-mini :: 9
bn-riscv-disassembler :: 2
bn-uxn :: 1
bn-wasm :: 9
BRCA1-BioAssay-Review :: 0
ChocolateFixGame :: 0
chromium :: 0
Clairvoyance :: 25
ctfblog :: 0
ctfdocker :: 0
curl :: 0
DataSort :: 0
dice-is-you :: 7
dicecraft :: 0
dicectf2022-breach :: 4
dicectf22-taxes :: 12
doppler :: 0
EasyCTF-2017-Write-ups :: 0
easyctf-2017-writeups :: 0
EconGame :: 0
format-string-attacks :: 0
fossasia.github.io :: 0
GeneralsIO-Bot-Controller :: 2
ghidra :: 0
gitbook-plugin-collapsible-chapters :: 0
GLaDOS :: 0

Very nice! But it looks like it didn't actually fetch them all (maybe due to pagination?). Let's explicitly add a fetch_all=True to our call:

o2.execute('''
projects = fetch_github_projects(user='hgarrereyn', fetch_all=True)

for p in projects:
    print(f'{p.name} :: {p.stars}')
''')

2018submissions :: 0
AudioScroll-Extension :: 9
beam :: 0
bn-fish-disassembler :: 10
bn-pokemon-mini :: 9
bn-riscv-disassembler :: 2
bn-uxn :: 1
bn-wasm :: 9
BRCA1-BioAssay-Review :: 0
ChocolateFixGame :: 0
chromium :: 0
Clairvoyance :: 25
ctfblog :: 0
...
OCRaaP :: 123
omni :: 0
phenny :: 0
PittAPI :: 0
pwndbg :: 0
pyalgotrade :: 0
reddit :: 1
redpandacoin :: 0
SBVA :: 31
STRIDE :: 98
sugar :: 0
tfx-bsl :: 0
Th3g3ntl3man-CTF-Writeups :: 21
TIFF :: 0

Let's visualize this data:

o2.execute('''
projects = fetch_github_projects(user='hgarrereyn', fetch_all=True)
plot_bar_chart(projects)
''')

Very nice, but a bit unreadable. Let's make it horizontal and also filter to the top 25, sorted by stars:

o2.execute('''
projects = fetch_github_projects(user='hgarrereyn', fetch_all=True)
top = top_sorted(projects, num=25)
plot_bar_chart(top, vertical=False)
''')

Let's flip it the other way (adding [::-1]) and ask for a bit more pizazz with with_pizazz=True:

o2.execute('''
projects = fetch_github_projects(user='hgarrereyn', fetch_all=True)
top = top_sorted(projects, num=25)[::-1]
plot_bar_chart(top, vertical=False, with_pizazz=True)
''')

Very nice! But suppose, we actually want to color the bars by the main language in the repo. Let's use the flag colored_by_primary_language=True:

o2.execute('''
projects = fetch_github_projects(user='hgarrereyn', fetch_all=True)
top = top_sorted(projects, num=25)[::-1]
plot_bar_chart(top, vertical=False, colored_by_primary_language=True)
''')