langsearch 0.1.1

Easily create semantic search based LLM applications on your own data

LangSearch: Easily create semantic search based LLM applications for your own data

Spiders

Webspider

Usage

from langsearch.spiders.webspider import WebSpider


class MyWebSpider(WebSpider):
    name = "my_web_spider"

Settings for WebSpider

1. LANGSEARCH_WEB_SPIDER_START_URLS: list of seed URLs
2. LANGSEARCH_WEB_SPIDER_LINK_EXTRACTOR_ALLOW: list of regex patterns that absolute URLs must match to be extracted and followed
3. LANGSEARCH_WEB_SPIDER_LINK_EXTRACTOR_DENY: list of regex patterns. Matching links will not be extracted and followed. Has precedence over LANGSEARCH_WEB_SPIDER_LINK_EXTRACTOR_ALLOW

If you already have a list of target URLs, use the following settings in settings.py.

# You can also write code in settings.py e.g. to load START_URLS from a file
LANGSEARCH_WEB_SPIDER_START_URLS = ["<first_link>", "<second_link>", ... ]
LANGSEARCH_WEB_SPIDER_LINK_EXTRACTOR_ALLOW = []
# We use an all-matching regex to ensure that only links in START_URLS are downloaded and no further links are followed
LANGSEARCH_WEB_SPIDER_LINK_EXTRACTOR_DENY = [".*"]

Here is an example for the second use case, where you don't have a list of target URLs but want to auto-discover links by crawling from some seed URLs.

LANGSEARCH_WEB_SPIDER_START_URLS = ["https://docs.python.org"]
# Follow links to docs.python.org only. Links to wiki.python.org will not be extracted or followed, for example.
LANGSEARCH_WEB_SPIDER_LINK_EXTRACTOR_ALLOW = ["docs.\python.\org"]
# Deny old documentation versions and non-english pages
LANGSEARCH_WEB_SPIDER_LINK_EXTRACTOR_DENY = ["docs\.python\.org(?!/3/)",
                                        "/es|fr|ja|ko|pt-br|tr|zh-cn|zh-tw/",
                                        ]

Filespider

Usage

from langsearch.spiders.filespider import FileSpider


class MyFileSpider(FileSpider):
    name = "my_file_spider"

Settings for FileSpider

1. LANGSEARCH_FILE_SPIDER_START_FOLDERS: list of folders to start from
2. LANGSEARCH_FILE_SPIDER_FOLLOW_SUBFOLDERS: boolean that indicates whether documents in subfolders will be fetched
3. LANGSEARCH_FILE_SPIDER_FOLLOW_SYMLINKS: boolean that indicates whether symbolic links will be followed
4. LANGSEARCH_FILE_SPIDER_ALLOW: list of regex that the absolute filepath (including extension) must match to be fetched
5. LANGSEARCH_FILE_SPIDER_DENY: absolute filepaths (including extension) matching any regex in this list will not be fetched. Has precedence over LANGSEARCH_FILE_SPIDER_ALLOW.

Middlewares

Spider Middlewares

RegexFilterMiddleware

Usage

Include the following in your settings.py

SPIDER_MIDDLEWARES = {
    'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': None,   # Disable scrapy's OffsiteMiddleware
    'langsearch.middlewares.spider_middlewares.RegexFilterMiddleware': 500,    # Use langsearch's RegexFilterMiddleware instead
}

Settings

1. LANGSEARCH_REGEX_FILTER_MIDDLEWARE_ALLOW
2. LANGSEARCH_REGEX_FILTER_MIDDLEWARE_DENY

Pipelines

Include the following in your settings.py

ITEM_PIPELINES = {
   # The item that you send down the pipeline must have the fields "body", "text" and "url"
   # this pipeline detects the item type and sends it down the processors that handle that type
   # must be the first one in the list
   "langsearch.pipelines.DetectItemTypePipeline": 100,   
   # this builds a pipeline placing all components in the correct order 
   **assemble(pipelines=[TextPipeline,    # Add the pipelines for the types you want extracted
                         AudioVideoPipeline,   # This one extracts text from audio and video
                         OtherPipeline
                         ],
               )                          
   # attributes in item are transparently passed though
}

Available settings

LANGSEARCH_TRAFILATURA_PIPELINE_EXTRACT_ARGUMENTS = {...}


LANGSEARCH_WHISPER_PIPELINE_ALLOWED_LANGUAGES=[...]
LANGSEARCH_WHISPER_PIPELINE_MODEL=...

LANGSEARCH_TEXT_LANGUAGE_FILTER_PIPELINE_ALLOWED_LANGUAGES=[...]

LANGSEARCH_STORE_ITEM_PIPELINE_EMBEDDING_MODEL=langsearch.EmbeddingModel.GPT3  # None will lead to no embedding and Weaviate's automatic embedding being used
LANGSEARCH_STORE_ITEM_PIPELINE_WEAVIATE_BASE_URL=http://localhost:...   # If not specified, will look for env variable
LANGSEARCH_STORE_ITEM_PIPELINE_DATABASE_URL=http://localhost:...   # If not specified, will look for env variable
LANGSEARCH_STORE_ITEM_PIPELINE_WEAVIATE_CLASS = ...  # If not specified, will use BOT_NAME setting
LANGSEARCH_STORE_ITEM_PIPELINE_DUPLICATE_CUTOFF = ...  # Default: 95

DryRunPipeline

We additionally make a DryRunPipeline available that simply dumps the URLs to a file. This is useful to check that your allow/deny rules are working as expected.

Philosophy for pipeline classes

1. Some pipelines have generic components that could have many implementations. For example, we use LLMs to generate answers etc., and there are many LLMs. If there is a standard interface already available that abstracts out the various implementations, and this standard interface has everything we need to implement the pipeline, then we will use that standard interface to write flexible pipeline classes. The exact implementation is then specified using settings or environment variables.