Apify-Haystack integration

The Apify-Haystack integration allows easy interaction between the Apify platform and Haystack.

Apify is a platform for web scraping, data extraction, and web automation tasks. It provides serverless applications called Actors for tasks like crawling websites and scraping Facebook, Instagram, and Google results.

Haystack offers an ecosystem of tools for building, managing, and deploying search engines and LLM applications.

Installation

Apify-haystack is available as the apify-haystack package on PyPI. It requires Python 3.11 or newer and Haystack 3.

pip install apify-haystack

Components

• ApifyDatasetFromActorCall runs an Actor, waits for it to finish, and loads its default dataset.
• ApifyDatasetFromTaskCall does the same for a saved Actor task.
• ApifyDatasetLoader loads an existing dataset by its ID.
• ApifyWebSearch searches the web with the RAG Web Browser Actor and returns the content of the top result pages.
• ApifyActorTool is a tool for Haystack Agents that runs any Actor and returns its dataset items. It takes its parameters from the Actor's input schema, and it marks the items it hands to the LLM as untrusted external content to guard against prompt injection.

The dataset components turn dataset items into Haystack Document objects with a mapping function. Without one, they use default_dataset_mapping_function, which takes the markdown or text of website and search Actors with the page url, title, and description as meta, and the whole item as JSON for other Actors. Every component also has run_async, which Pipeline.run_async uses to run it without blocking the event loop. When an Actor or task run finishes with any status other than SUCCEEDED, the component raises a RuntimeError instead of loading a partial dataset.

The components read your Apify API token from the APIFY_TOKEN environment variable, or from APIFY_API_TOKEN if APIFY_TOKEN isn't set. To pass the token differently, use a Haystack Secret, for example apify_token=Secret.from_env_var('MY_APIFY_TOKEN').

Examples

Crawl a website using Apify's Website Content Crawler and convert it to Haystack Documents

You need an Apify account and API token to run this example. You can start with a free account at Apify and get your API token.

In the example below, set the APIFY_TOKEN environment variable and run the script:

from haystack import Document

from apify_haystack import ApifyDatasetFromActorCall

actor_id = 'apify/website-content-crawler'
run_input = {
    'maxCrawlPages': 3,  # limit the number of pages to crawl
    'startUrls': [{'url': 'https://haystack.deepset.ai/'}],
}


def dataset_mapping_function(dataset_item: dict) -> Document:
    return Document(content=dataset_item.get('text'), meta={'url': dataset_item.get('url')})


actor = ApifyDatasetFromActorCall(
    actor_id=actor_id, run_input=run_input, dataset_mapping_function=dataset_mapping_function
)
print(f'Calling the Apify Actor {actor_id}. Crawling will take some time.')
print('You can monitor the progress at: https://console.apify.com/actors/runs')

dataset = actor.run()['documents']

print(f'Loaded {len(dataset)} documents from the Apify Actor {actor_id}:')
for d in dataset:
    print(d)

Search the web

ApifyWebSearch returns the top result pages as Markdown Document objects, with the page url, title, and description in meta, and their URLs in links:

from apify_haystack import ApifyWebSearch

search = ApifyWebSearch(top_k=3)
result = search.run(query='What is Haystack?')

for document in result['documents']:
    print(document.meta['url'], document.meta['title'])

Give an Agent access to an Actor

ApifyActorTool lets a Haystack Agent run an Actor. Limit dataset_items_limit to keep the tool output within the model's context window. For pay-per-event Actors, max_total_charge_usd caps what each run the Agent starts can cost:

from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

from apify_haystack import ApifyActorTool

tool = ApifyActorTool('apify/rag-web-browser', dataset_items_limit=3, max_total_charge_usd=0.5)
agent = Agent(chat_generator=OpenAIChatGenerator(), tools=[tool])

result = agent.run([ChatMessage.from_user('What is new in Haystack 3?')])
print(result['last_message'].text)

Save and load a pipeline

The components and the tool support Haystack serialization. The token is saved as the name of its environment variable, never as its value. Haystack 3 only loads classes and functions from trusted modules, so allow apify_haystack and the module with your mapping function:

from haystack import Pipeline

yaml = pipeline.dumps()
restored = Pipeline.loads(yaml, allowed_modules=['apify_haystack', 'my_project.mapping'])

More examples

The examples directory has more examples:

• Load a dataset from Apify and convert it to a Haystack Document
• Call Website Content Crawler and convert the data into the Haystack Documents
• Crawl websites, retrieve text content, and store it in the InMemoryDocumentStore
• Retrieval-Augmented Generation (RAG): Extracting text from a website & question answering
• Analyze your Instagram comments' vibe with Apify and Haystack

Support

If you find any bug or issue, please submit an issue on GitHub. For questions, you can ask on Stack Overflow, in GitHub Discussions or you can join our Discord server.

Contributing

Code contributions are welcome. If you have ideas for improvements, submit an issue or create a pull request. For contribution guidelines, see CONTRIBUTING.md.

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.