spark-llm 0.1.3

LLM assistant for the development of Spark applications

LLM Assistant for Apache Spark

Installation

pip install spark-llm

Usage

Initialization

from langchain.chat_models import ChatOpenAI
from spark_llm import SparkLLMAssistant

llm = ChatOpenAI(model_name='gpt-4') # using gpt-4 can achieve better results
assistant=SparkLLMAssistant(llm=llm)
assistant.activate() # active partial functions for Spark DataFrame

Data Ingestion

auto_df=assistant.create_df("2022 USA national auto sales by brand")
auto_df.show(n=5)

rank	brand	us_sales_2022	sales_change_vs_2021
1	Toyota	1849751	-9
2	Ford	1767439	-2
3	Chevrolet	1502389	6
4	Honda	881201	-33
5	Hyundai	724265	-2

DataFrame Transformation

auto_top_growth_df=auto_df.llm_transform("top brand with the highest growth")
auto_top_growth_df.show()

brand	us_sales_2022	sales_change_vs_2021
Cadillac	134726	14

DataFrame Explanation

auto_top_growth_df.llm_explain()

In summary, this dataframe is retrieving the brand with the highest sales change in 2022 compared to 2021. It presents the results sorted by sales change in descending order and only returns the top result.

Refer to example.ipynb for more detailed usage examples.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

Licensed under the Apache License 2.0.