scrapy-llm 0.1.7

LLM integration for Scrapy

Scrapy-LLM

LLM integration for scrapy as a middleware.

   

Installation

pip install scrapy-llm

Usage

# settings.py
DOWNLOADER_MIDDLEWARES = {
    'scrapy_llm.handler.LlmExtractorMiddleware': 543,
    ...
}

then access extracted data from response object

# spider.py
def parse(self, response):
    extracted_data: Dict[str, Any] = response.request.meta.get('llm_extracted_data')
    ...

Configuration

All aspects of the middleware can be configured using the settings.py file except the API key which should be set as the environment variable OPENAI_API_KEY according to the openai api documentation here.

LLM_RESPONSE_MODEL

• type: str
• required: True

the response model to use for extracting data from the web page text.

RESPONSE_MODEL = 'scraper.models.ResponseModel'

this setting can also be set as an attribute on the spider class itself, in that case the class should be used directly instead of a string path to the class.

class MySpider(scrapy.Spider):
    response_model = ResponseModel
    ...

LLM_UNWRAP_NESTED

• type: bool
• required: False
• default: True

whether to unwrap nested models in the extracted data.

LLM_UNWRAP_NESTED = True

for example if the following model is used

class ContactInfo(BaseModel):
    phone: str

class Person(BaseModel):
    name: str
    contact_info: ContactInfo

the extracted data will be unwrapped to

{
    "name": "John Doe",
    "phone": "1234567890"
}

without unwrapping the data will be

{
    "name": "John Doe",
    "contact_info": {
        "phone": "1234567890"
    }
}

LLM_API_BASE

• type: str
• required: False
• default: https://api.openai.com/v1

base url for the openai compatible api.

LLM_API_BASE = 'https://api.openai.com/v1'

LLM_MODEL

• type: str
• required: False
• default: "gpt-4-turbo"

the language model to use for extracting data from the web page text.

LLM_MODEL = 'gpt-4-turbo'

LLM_MODEL_TEMPERATURE

• type: float
• required: False
• default: 0.0001

the temperature to use for the language model.

LLM_MODEL_TEMPERATURE = 0.0001

LLM_SYSTEM_MESSAGE

• type: str
• required: False
• default: You are a data extraction expert, your role is to extract data from the given text according to the provided schema. make sure your output is a valid JSON object.

the system message to use for the language model.

LLM_SYSTEM_MESSAGE = '...'