website_firecrawl_service - MCP Server
🔍 Internet research just got smarter! Built an MCP server that turns any website into structured, relevant content based on your queries!
Using @Firecrawl's powerful features (mapping, selection, scraping), combined with GPT-4o for smart URL filtering, it's like having an AI research assistant that knows exactly what you're looking for! Works seamlessly with Claude, or any MCP-compatible client!
An agentic web scraping system powered by Firecrawl: Map → Select → Scrape → Extract
Features
- Efficient Web Crawling: Crawls websites using the Firecrawl API with customizable link limits and intelligent URL selection
- Intelligent URL Selection: Uses GPT-4 to select the most relevant URLs based on user queries
- Smart Content Processing: Extracts and cleans HTML content, providing readable text output
Setup
- Create a
.envfile with the following required environment variables:FIRECRAWL_API_KEY=your_firecrawl_api_key OPENAI_API_KEY=your_openai_api_key
Usage
The server exposes a single tool:
website_firecrawl
Description: Crawls a website and returns relevant content based on a query.
Parameters:
query(string): The search query to filter relevant contentbase_url(string): The target website URL to crawlmax_links(integer, optional): Maximum number of links to process (default: 100)
Technical Details
- Built using the MCP (Model Control Protocol) server framework
- Implements retry logic with exponential backoff for API calls
- Integrates with LangSmith for tracing and monitoring
- Implements singleton patterns for API clients to manage resources efficiently
- Uses Pydantic for robust data validation and serialization:
WebsiteCrawlArgs: Validates input parameters for the crawling serviceCrawlerModel: Handles URL selection and justificationPage: Structures metadata and content from crawled pages
- Structured OpenAI Integration:
- Uses OpenAI's beta chat completions with parsing
- Automatically validates and converts JSON responses to Pydantic models
- Ensures type safety and data validation for AI-generated content
- Jinja2 Template System:
- Modular prompt management using template inheritance
- Dynamic prompt generation based on user queries and context
- Separate system and user prompt templates for clear separation of concerns
- Easy maintenance and updates of prompt structures
Metadata
Release files for iflow-mcp_lgesuellip-website-firecrawl-service 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| iflow_mcp_lgesuellip_website_firecrawl_service-0.1.0.tar.gz | 256.0 kB | Without provenance | Details for iflow_mcp_lgesuellip_website_firecrawl_service-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| iflow_mcp_lgesuellip_website_firecrawl_service-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for iflow_mcp_lgesuellip_website_firecrawl_service-0.1.0-py3-none-any.whl |
Total release size: 265.0 kB
Release files / iflow_mcp_lgesuellip_website_firecrawl_service-0.1.0.tar.gz
| Download URL | iflow_mcp_lgesuellip_website_firecrawl_service-0.1.0.tar.gz |
|---|---|
| Size | 256.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6d1816cc7f477e572078310d7d51ad5a6978fb817eb1f672fe812d9e3be607dc
|
|
BLAKE2b-256 checksum How to use checksums |
a4954ea528a275b558119c2dd05efeb79999753b123d086065980600664d42e5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.10.0 {"installer":{"name":"uv","version":"0.10.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"13","id":"trixie","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / iflow_mcp_lgesuellip_website_firecrawl_service-0.1.0-py3-none-any.whl
| Download URL | iflow_mcp_lgesuellip_website_firecrawl_service-0.1.0-py3-none-any.whl |
|---|---|
| Size | 9.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b5eb5449dde2ef875017dab039a4698ea3cc0b7c0d3b9a9ca4488436c1cf7d46
|
|
BLAKE2b-256 checksum How to use checksums |
e7b4e698aee380bb6701ad3caff6471627b0da2c18e845862d923a2a329c2bc9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.10.0 {"installer":{"name":"uv","version":"0.10.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"13","id":"trixie","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for iflow-mcp_lgesuellip-website-firecrawl-service 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.