fleet-context 1.0.3

A chat interface over up-to-date Python library documentation.

🛩️ Fleet Context

Code generation with up-to-date Python libraries.

WIP View the site‎ ‎ |‎ ‎ WIP API waitlist

Quick Start

Install the package and run context to ask questions about the most up-to-date Python libraries. You will have to provide your OpenAI key to start a session.

pip install fleet-context

context

Limit libraries

You can use the -l or --libraries followed by a list of libraries to limit your session to a certain number of libraries. Defaults to all.

context -l langchain pydantic openai

Use a different OpenAI model

You can select a different OpenAI model by using -m or --model. Defaults to gpt-4. You can set your model to gpt-4-32k (if your organization has access), gpt-3.5-turbo, or gpt-3.5-turbo-16k.

context -m gpt-3.5-turbo

Advanced settings

You can control the number of retrieved chunks by using -k or --k_value (defaulted to 10), and you can toggle whether the model cites its source by using -c or --cite_sources (defaults to true).

context -k 15 -c false

Evaluations

Results

Sampled libraries

We saw a 37-point improvement for gpt-4 generation scores across the board. We attribute this to a lack of knowledge for the most up-to-date versions of libraries.

Langchain

We saw a 48-point improvement for gpt-3.5 and a 58-point improvement for gpt-4. We hypothesize that the reason the "before" score for gpt-4 is lower is because it's better at mentioning what it doesn't know.

The drastic jump makes sense, given the entire Langchain documentation was built after gpt-4's knowledge cutoff.

Pydantic

We saw a 34-point improvement for gpt-3.5 and a 38-point improvement for gpt-4. This is because Pydantic v1 was launched before gpt-4's knowledge cutoff, but Pydantic v2 was launched in 2022. The improvement was not as sharp, but it was still significant.