firecrawl-toolkit 0.0.24

A high-performance, asynchronous MCP server for Firecrawl Search, featuring connection pooling, request retries, and intelligent input parsing.

Firecrawl MCP Toolkit

A high-performance, asynchronous MCP server that provides comprehensive Google search and web content scraping capabilities through the Firecrawl API (excluding some rarely used interfaces).

This project is built on httpx, utilizing asynchronous clients and connection pool management to offer LLMs a stable and efficient external information retrieval tool.

PyPI Package

firecrawl-toolkit: https://pypi.org/project/firecrawl-toolkit/

Key Features

• Asynchronous Architecture: Fully based on asyncio and httpx, ensuring high throughput and non-blocking I/O operations.
• HTTP Connection Pool: Manages and reuses TCP connections through a global httpx.AsyncClient instance, significantly improving performance under high concurrency.
• Concurrency Control: Built-in global and per-API endpoint concurrency semaphores effectively manage API request rates to prevent exceeding rate limits.
• Automatic Retry Mechanism: Integrated request retry functionality with exponential backoff strategy automatically handles temporary network fluctuations or server errors, enhancing service stability.
• Intelligent Country Code Parsing: Includes a comprehensive country name dictionary supporting inputs in Chinese, English, ISO Alpha-2/3, and other formats, with automatic normalization.
• Response Field Mapping: Search/Scrape responses are normalized into minimal, client-facing JSON schemas instead of upstream passthrough payloads.
• Noise Reduction for Scrape: Built-in excludeTags selector filtering removes common non-content blocks (navigation, ads, sidebars, comments, etc.) to improve signal quality. Supports returning a specified Markdown character window with startIndex and maxCharacters.
• Flexible Environment Variable Configuration: Supports fine-tuned service configuration via environment variables.
• The Search and Scrape Endpoints perform some request pre-processing and post-processing, which can save quite a few tokens.

Available Tools

This service provides the following tools:

Tool Name	Description
firecrawl-aggregated-search	Aggregated Search Interface, Combining Webpage, News, And Image Search Results.
firecrawl-web-search	Web Search Interface.
firecrawl-news-search	News Search Interface.
firecrawl-image-search	Image Search Interface.
firecrawl-scrape	Scrapes and returns the content of a specified URL.

Installation Guide

It is recommended to install using pip or uv.

# Using pip
pip install firecrawl-toolkit

# Or using uv
uv pip install firecrawl-toolkit

Quick Start

Set Environment Variables

Create a .env file in the project root directory and enter your Firecrawl API key:

Environment Variables	Default value	Description
FIRECRAWL_API_KEY	fc-xxx	Your Firecrawl API key. Multiple keys can be separated by commas, and one will be selected randomly for each request.
FIRECRAWL_HTTP2	0	Disable or enable HTTP2, <0/1>
FIRECRAWL_MAX_WORKERS	10	Number of processes
FIRECRAWL_MAX_CONNECTIONS	200	Maximum number of connections
FIRECRAWL_MAX_CONCURRENT_REQUESTS	200	Maximum number of concurrent requests
FIRECRAWL_KEEPALIVE	20	Maximum number of concurrent connections
FIRECRAWL_RETRY_COUNT	3	Maximum number of retries
FIRECRAWL_RETRY_BASE_DELAY	0.5	Base delay time for retries in seconds
FIRECRAWL_ENDPOINT_CONCURRENCY	{"search":10,"scrape":2}	Set concurrency per endpoint (JSON format)
FIRECRAWL_ENDPOINT_RETRYABLE	{"scrape": false}	Set retry allowance per endpoint (JSON format)
FIRECRAWL_MCP_ENABLE_STDIO	0	Disable or enable STDIO, <0/1>
FIRECRAWL_MCP_ENABLE_HTTP	0	Disable or enable HTTP, <0/1>
FIRECRAWL_MCP_ENABLE_SSE	0	Disable or enable SSE, <0/1>
FIRECRAWL_MCP_HTTP_HOST	127.0.0.1	HTTP host address
FIRECRAWL_MCP_HTTP_PORT	7001	HTTP host port
FIRECRAWL_MCP_SSE_HOST	127.0.0.1	SSE host address
FIRECRAWL_MCP_SSE_PORT	7001	SSE host port
FIRECRAWL_MCP_LOCK_FILE	/tmp/firecrawl_mcp.lock	Lock file path

• STDIO, HTTP, and SSE can only be used one at a time. If you need to use multiple protocols, please start separate services for each.
• When using multiple services, please specify different lock files for each.

Configure MCP Client

Add the following server configuration in the MCP client configuration file:

{
  "mcpServers": {
    "firecrawl": {
      "command": "python3",
      "args": ["-m", "firecrawl-toolkit"],
      "env": {
        "FIRECRAWL_API_KEY": "<Your Firecrawl API key>"
      }
    }
  }
}

{
  "mcpServers": {
    "firecrawl": {
      "command": "uvx",
      "args": ["firecrawl-toolkit"],
      "env": {
        "FIRECRAWL_API_KEY": "<Your Firecrawl API key>"
      }
    }
  }
}

Tool Parameters and Usage Examples

firecrawl Search: Perform aggregated / web / news / images search

Parameters:

• query (str, required): Keywords to search.
• country (str, optional): Specify the country/region for search results. Supports Chinese names (e.g., "China"), English names (e.g., "United States"), or ISO codes (e.g., "US"). Default is "US".
• search_num (int, optional): Number of results to return, range 1-100. Default is 20.
• search_time (str, optional): Filter results by time range. Available values: "hour", "day", "week", "month", "year".

Example:

result_json = firecrawl_web_search(
    query="AI advancements 2024",
    country="United States",
    search_num=5,
    search_time="month"
)

Response (mapped):

• Top-level fields: success, data, creditsUsed
• data.web[]: title, description, url
• data.news[]: title, snippet, url, date
• data.images[]: title, imageUrl, url
• web / news / images remain arrays and may be empty ([])
• Missing mapped fields are preserved as null
• Output is compact single-line JSON (no extra spaces)

Example response:

{"success":true,"data":{"web":[{"title":"Example Web","description":"Example description","url":"https://example.com"}],"news":[],"images":[]},"creditsUsed":1}

firecrawl-scrape: Scrape webpage content

Parameters:

• url (str, required): URL of the target webpage.
• excludeTags (list[str], optional, default []): Additional CSS selectors to exclude; merged with built-in noise-filter selectors after normalization and deduplication unless emptyTags=True.
• includeTags (list[str], optional, default None): Additional CSS selectors to include; no built-in defaults are applied, and the cleaned list is forwarded only when this parameter is provided.
• maxCharacters (int, optional, default None): Truncate only the returned markdown to N characters starting at startIndex. Invalid values (non-int, <= 0) are ignored and treated as not provided.
• startIndex (int, optional, default 0): Start offset used with maxCharacters when slicing returned markdown. Invalid values (non-int, < 0) are treated as 0.
• emptyTags (bool, optional, default False): Clear the built-in exclude selector list for this request, while still keeping any user-provided excludeTags.
• headers (dict[str, str], optional, default None): Root-level request headers passed through to the upstream scrape request only when a non-empty object is provided.

Example:

result_json = firecrawl_scrape(
    url="https://www.example.com",
    includeTags=["article", ".content"],
    excludeTags=["[class^=\"skip\"]", "[id*=\"disqus\"]"],
    startIndex=0,
    maxCharacters=1200,
    headers={"Authorization": "Bearer token", "X-Trace-Id": "abc123"}
)

This returns at most 1200 characters in markdown, starting at character index 0.

To explicitly send an empty include selector list:

result_json = firecrawl_scrape(
    url="https://www.example.com",
    includeTags=[]
)

To disable only the built-in exclude selectors for one request:

result_json = firecrawl_scrape(
    url="https://www.example.com",
    emptyTags=True
)

To disable the built-in exclude selectors but keep your own:

result_json = firecrawl_scrape(
    url="https://www.example.com",
    excludeTags=[".nav"],
    emptyTags=True
)

Built-in noise filtering:

• The tool uses an internal excludeTags selector set to suppress noisy DOM regions and prioritize main content quality.
• includeTags has no built-in defaults and is only forwarded when explicitly provided.
• Passing emptyTags=True clears only the built-in exclude selector set for that request.
• If the first scrape returns data.markdown == "", the tool automatically retries once without includeTags/excludeTags as a fallback.
• startIndex / maxCharacters slicing is applied locally in this toolkit post-processing and is not forwarded to upstream Firecrawl payloads.

Response (mapped):

• Top-level fields: success, proxyUsed, title, description, language, markdown, creditsUsed
• markdown is URL-decoded before returning to the client
• When a valid maxCharacters is provided, markdown length is capped at that value after applying startIndex
• Missing mapped fields are preserved as null
• Output is compact single-line JSON (no extra spaces)

Example response:

{"success":true,"proxyUsed":"auto","title":"Example Page","description":"Example summary","language":"en","markdown":"Hello world!","creditsUsed":1}

Response Contract Notes

• firecrawl-search and firecrawl-scrape success payloads are mapped to stable minimal schemas.
• Missing mapped fields are preserved as null (arrays remain arrays, and may be empty).
• Both success and error responses are compact single-line JSON.

License Agreement

This project is licensed under the MIT License.