ibm-watsonx-data-integration-mcp 1.0.0rc1

MCP server for IBM watsonx.data Integration

IBM watsonx.data Integration MCP Server

An MCP server built on top of the ibm-watsonx-data-integration SDK with intelligent documentation retrieval using vector search.

---

Requirements

• Python: 3.11-3.12 (3.12 recommended)

---

Setup Instructions

There are three ways to run this MCP server:

Option 1: Using uvx (Recommended)

The fastest way to run the server without installation.

Verify Installation:

uvx --python 3.12 --from git+ssh://[e-mail]/watsonx-data-integration/di-mcp.git data-intg-mcp --help

MCP Server Configuration:

{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "uvx",
      "args": [
        "--python",
        "3.12",
        "--from",
        "git+ssh://[e-mail]/watsonx-data-integration/di-mcp.git",
        "data-intg-mcp"
      ],
      "env": {
        "WATSONX_API_KEY": "API_KEY"
      }
    }
  }
}

---

Option 2: Using uv as a Tool

Install and run as a uv tool.

Setup:

uv tool install --python 3.12 git+ssh://[e-mail]/watsonx-data-integration/di-mcp.git

Verify Installation:

uv tool run --python 3.12 data-intg-mcp --help

MCP Server Configuration:

{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "uv",
      "args": [
        "tool",
        "run",
        "data-intg-mcp"
      ],
      "env": {
        "WATSONX_API_KEY": "API_KEY"
      }
    }
  }
}

---

Option 3: Using pip install with Virtual Environment

Install the package in a virtual environment.

Setup:

python3.12 -m venv .venv
source .venv/bin/activate
pip install git+ssh://[e-mail]/watsonx-data-integration/di-mcp.git

Verify Installation:

data-intg-mcp --help

MCP Server Configuration:

{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "<Path to .venv>/bin/data-intg-mcp",
      "args": [],
      "env": {
        "WATSONX_API_KEY": "API_KEY"
      }
    }
  }
}

---

MCP Client Integration

🤖 Claude Desktop Integration

To connect this MCP server to Claude Desktop:

Navigate to: Claude → Settings → Developer → Edit Config

Add one of the MCP server configurations above to your claude_desktop_config.json file.

---

🤖 Bob (VS Code) Integration

To connect this MCP server to Bob via VS Code:

Navigate to: Open VS Code → Open Bob → Click the ⚙️ Settings icon → Navigate to the MCP section → Click Open next to Global MCPs

Add one of the MCP server configurations above to your Bob MCP config file.

---

Configuration Notes

Windows UTF-8 Configuration

Important for Windows Users: Due to a limitation in fastmcp 3.2.4, the server requires UTF-8 encoding to properly load skill files that contain emoji and special characters. On Windows, you must set the PYTHONUTF8 environment variable:

{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "...",
      "args": [...],
      "env": {
        "PYTHONUTF8": "1",
        "WATSONX_API_KEY": "your-api-key"
      }
    }
  }
}

Why is this needed?

• Windows defaults to cp1252 encoding, which cannot decode UTF-8 emoji (✅, ❌, →) in SKILL.md files
• Setting PYTHONUTF8=1 forces Python to use UTF-8 encoding for all file operations
• Without this setting, the server will fail to start on Windows with a UnicodeDecodeError

This applies to all installation methods (uvx, uv tool, pip install) when running on Windows.

---

Authentication

The server supports both SaaS (IBM Cloud) and On-Premises (Cloud Pak for Data) authentication. It automatically detects which authentication method to use based on the environment variables you set.

SaaS Authentication (IBM Cloud)

For SaaS deployments, set the WATSONX_API_KEY environment variable:

{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "...",
      "args": [...],
      "env": {
        "WATSONX_API_KEY": "your-ibm-cloud-api-key"
      }
    }
  }
}

Region Selection:

When using SaaS authentication, the AI assistant should ask you to specify your IBM Cloud region when needed. Generated examples should not hardcode a specific region; templates should use a neutral placeholder such as YOUR_REGION. Available regions (from the IBMCloudRegion enum):

• TORONTO - Toronto, Canada
• DALLAS - Dallas, USA
• FRANKFURT - Frankfurt, Germany
• LONDON - London, United Kingdom
• TOKYO - Tokyo, Japan
• SYDNEY - Sydney, Australia

The region is passed as a parameter to the tools that require it, ensuring you're connecting to the correct regional endpoint.

On-Premises Authentication (Cloud Pak for Data)

For On-Premises deployments, you have two options:

Option 1: Username + Password

{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "...",
      "args": [...],
      "env": {
        "CP4D_USERNAME": "your-username",
        "CP4D_PASSWORD": "your-password",
        "CP4D_URL": "https://your-cp4d-cluster.com",
        "CP4D_DISABLE_SSL_VERIFICATION": "false"
      }
    }
  }
}

Option 2: Username + Zen API Key

{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "...",
      "args": [...],
      "env": {
        "CP4D_USERNAME": "your-username",
        "ZEN_API_KEY": "your-zen-api-key",
        "CP4D_URL": "https://your-cp4d-cluster.com",
        "CP4D_DISABLE_SSL_VERIFICATION": "false"
      }
    }
  }
}

Note on SSL Verification:

• CP4D_DISABLE_SSL_VERIFICATION: Set to "true" to disable SSL certificate verification for On-Premises deployments (useful for self-signed certificates). Default is "false".
• Security Warning: Only disable SSL verification in development/testing environments. Always use proper SSL certificates in production.

Configuring TLS for On-Premises Deployments

If your On-Premises cluster uses self-signed certificates or custom Certificate Authorities (CA), you may need to configure TLS properly instead of disabling SSL verification. Here's how to set up TLS:

Step 1: Log in to your OpenShift cluster

oc login <CLUSTER_URL>

Step 2: Extract the cluster's CA certificate

openssl s_client -showcerts \
  -connect <CLUSTER>:443 </dev/null > ca.crt

Replace <CLUSTER> with your cluster hostname (e.g., your-cp4d-cluster.com).

Step 3: Clean up the certificate file

Open the ca.crt file and remove all text except the certificate blocks. The file should only contain lines like:

-----BEGIN CERTIFICATE-----
[certificate content]
-----END CERTIFICATE-----

Remove any connection information, verification messages, or other output from the openssl command.

Step 4: Configure the MCP server to use the CA certificate

Add the REQUESTS_CA_BUNDLE environment variable to your MCP server configuration, pointing to the cleaned ca.crt file:

{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "...",
      "args": [...],
      "env": {
        "CP4D_USERNAME": "your-username",
        "CP4D_PASSWORD": "your-password",
        "CP4D_URL": "https://your-cp4d-cluster.com",
        "CP4D_DISABLE_SSL_VERIFICATION": "false",
        "REQUESTS_CA_BUNDLE": "/path/to/ca.crt"
      }
    }
  }
}

Replace /path/to/ca.crt with the absolute path to your cleaned certificate file.

Authentication Detection Priority:

1. SaaS (IAMAuthenticator) - if WATSONX_API_KEY is set
2. On-Premises (ICP4DAuthenticator) - if CP4D_USERNAME and CP4D_PASSWORD are set
3. On-Premises (ZenApiKeyAuthenticator) - if CP4D_USERNAME and ZEN_API_KEY are set

The server validates authentication credentials on startup and will display which authentication method was detected.

---

Usage

Offline Documentation Mode

If your environment cannot access GitHub Pages, start the server with --use-local-docs. In this mode, the server reads packaged HTML files from src/di_mcp/bundled_docs, converts them into cached markdown files, and then builds the lanceDB index in the normal runtime cache.

Example:

data-intg-mcp --use-local-docs

Important:

• Offline docs may not be the most up to date because they do not use the live documentation site.
• Refresh the packaged docs periodically before publishing or deploying if you want the bundled copy to stay current.

To refresh the packaged HTML docs, run:

python tools/download_docs.py

This downloads the latest SDK HTML pages and stores them under src/di_mcp/bundled_docs/html so they can be packaged with the server.

Available Tools

The MCP server provides the following tools:

Documentation Tools

1. search_sdk_documentation: Search SDK documentation using semantic similarity

   • Use for conceptual questions and "how to" guidance
   • Searches across all indexed SDK documentation
   • Returns relevant sections with source information
   • Example: "How do I configure a Kafka Consumer?"

2. get_model_reference: Look up exact model definitions

   • Use for exact field lists, enum/Literal values, and type signatures
   • Returns authoritative reference for a specific SDK model class
   • Includes complete field definitions with types and defaults
   • Strips Pydantic boilerplate methods for clarity
   • Provides fuzzy suggestions if class name is not found
   • Example: get_model_reference(class_name='Schedule') returns all valid values for repeat_mode field

When to use which tool:

• Use search_sdk_documentation for: "How do I...", conceptual explanations, examples, tutorials
• Use get_model_reference for: "What are the valid values for...", "What fields does X have?", exact type signatures

Version Tools

3. get_mcp_version: Get the current version of the MCP server
4. get_sdk_version: Get the version of the IBM watsonx.data Integration SDK

Resource Tools

5. list_resources: List all available MCP resources (via ResourcesAsTools transform)
6. read_resource: Read a specific resource by URI (via ResourcesAsTools transform)

Execution Tools

7. execute_script: Execute generated Python scripts for validation (stdio only, requires auth)

Stage Discovery Tools (Streaming)

8. list_available_streaming_stages: List available streaming stages (stdio only, requires auth)

   • Returns all available stage information for a streaming flow
   • Critical for determining available stages when creating streaming flows

9. list_all_available_stage_configurations_streaming: Get streaming stage configurations (stdio only, requires auth)

   • Returns comprehensive configuration information for specified streaming stage types
   • Includes all available properties, their types, and default values
   • Critical for determining available properties for streaming flow stages

Stage Discovery Tools (Batch)

10. list_available_batch_stages: List available batch stages (stdio only, requires auth)

    • Returns all available stage information for a batch flow
    • Critical for determining available stages when creating batch flows

11. list_all_available_stage_configurations_batch: Get batch stage configurations (stdio only, requires auth)

    • Returns comprehensive configuration information for specified batch stage types
    • Includes all available properties, their types, and default values
    • Critical for determining available properties for batch flow stages

---

Available Resources

The MCP server provides several resources that contain best practices, templates, and guidance:

Best Practices Resource

• watsonx://best_practices: Canonical usage rules, sequencing patterns, and best practices for the IBM watsonx.data Integration SDK
  • Core concepts and platform-centric architecture
  • Authentication patterns
  • Configuration management
  • Streaming and batch flow best practices
  • Common invalid patterns to avoid
  • Code generation best practices

Intent Document Resources

Intent documents are MANDATORY when creating batch or streaming flows. They ensure complete requirements capture before code generation.

• intent://overview: Overview of the intent document generation approach
• intent://batch-flow: Template and instructions for batch flow intent documents
• intent://streaming-flow: Template and instructions for streaming flow intent documents

Workflow for flow creation:

1. Read the appropriate intent resource
2. Generate a complete intent document (must include Mermaid diagram)
3. Get user approval
4. Generate and execute Python code

Skills Resources

Skills provide authoritative patterns and complete working examples for specific operations:

• skill://platform/SKILL.md: Platform operations and authentication
• skill://project/SKILL.md: Project creation, listing, and management
• skill://batch-flows/SKILL.md: Batch flow operations and configuration
• skill://streaming-flows/SKILL.md: Streaming flow operations and configuration
• skill://jobs-and-job-runs/SKILL.md: Job creation, execution, and monitoring

---

Testing

Running Tests

The project includes comprehensive unit tests for all modules. To run the tests:

# Install test dependencies
uv pip install -e ".[dev]"

# Run all unit tests with coverage (using poe tasks)
uv run poe unit

# Run unit tests serially (useful for debugging)
uv run poe unit-serial

# Run specific test file or pattern
uv run poe test tests/unit/test_crawler.py
uv run poe test tests/unit/test_main.py -k test_name

# Run tests for specific Python versions
uv run poe unit311  # Python 3.11
uv run poe unit312  # Python 3.12

# Or use pytest directly
pytest tests/unit/
pytest --cov=src/di_mcp --cov-report=html --cov-report=term-missing tests/unit/

Running Integration Tests

Option 1: Ollama

ollama run qwen3-coder:30b

uv run --group integration-local poe integration

Option 2: Claude

export AWS_ACCESS_KEY_ID=<key_id>
export AWS_SECRET_ACCESS_KEY=<access_key>

uv run --group integration-remote poe integration-claude

Coverage

After running tests with coverage, open htmlcov/index.html in your browser to view detailed coverage reports.

Advanced Usage

Setting PYTHONPATH for Local SDK Development

If you're developing or testing with a local version of the IBM watsonx.data Integration SDK, you can set the PYTHONPATH environment variable to point to your local SDK directory:

{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "...",
      "args": [...],
      "env": {
        "PYTHONPATH": "<Path to SDK>",
        "WATSONX_API_KEY": "API_KEY"
      }
    }
  }
}

Replace <Path to SDK> with the absolute path to your local SDK directory.

Cache Management

The server caches documentation and creates a vector database for efficient retrieval. To clear all cached data:

data-intg-mcp --clear-cache

This removes:

• resources_cache/ - Cached documentation files
• raw_resource_html/ - Raw HTML files
• lance_db/ - Vector database (LanceDB)

Use this command when you need to refresh the documentation cache or troubleshoot issues with the vector database.

HTTP/SSE Transport

By default, the MCP server runs over stdio transport (for local MCP clients like Claude Desktop or Bob). You can also run it as an HTTP server using Server-Sent Events (SSE) transport for remote access:

# Run as HTTP/SSE server (API key is optional for SSE)
data-intg-mcp --transport sse --host 0.0.0.0 --port 8000

# If you need SDK operations that require authentication, set the API key:
export WATSONX_API_KEY="your-api-key" # pragma: allowlist secret
data-intg-mcp --transport sse --host 0.0.0.0 --port 8000

Tool Availability by Transport:

Tool	stdio	SSE	Notes
search_sdk_documentation	✓	✓	Semantic search across SDK docs
get_model_reference	✓	✓	Look up exact model definitions
get_mcp_version	✓	✓	Get MCP server version
get_sdk_version	✓	✓	Get SDK version
list_resources	✓	✓	List available resources
read_resource	✓	✓	Read specific resource by URI
execute_script	✓	✗	Disabled in SSE for security
list_available_streaming_stages	✓	✗	Requires authentication
list_all_available_stage_configurations_streaming	✓	✗	Requires authentication
list_available_batch_stages	✓	✗	Requires authentication
list_all_available_stage_configurations_batch	✓	✗	Requires authentication

Important Security Notes:

• When using SSE transport, tools requiring authentication or local execution are automatically disabled
• The SSE server allows remote connections, so ensure proper network security measures are in place
• WATSONX_API_KEY is optional for SSE transport (only needed if using authenticated tools)
• For stdio transport, WATSONX_API_KEY is required
• Environment variables must be set on the server side, not in the client configuration
• Consider using authentication/authorization mechanisms when exposing the server over HTTP

MCP Client Configuration for SSE:

For remote MCP clients, configure the server URL:

{
  "mcpServers": {
    "data-intg-mcp-remote": {
      "url": "http://127.0.0.1:8000/sse"
    }
  }
}

Note: Unlike stdio transport, environment variables cannot be passed from the client config when using SSE. They must be set in the server's environment before starting the server.

Command Line Options:

• --transport: Choose stdio (default) or sse
• --host: Host to bind to for SSE (default: 0.0.0.0, ignored for stdio)
• --port: Port to bind to for SSE (default: 8000, ignored for stdio)
• --disable-execution-tools: Disable execute_script and stage discovery tools (recommended for remote deployments)
• --clear-cache: Clear all cached documentation before starting
• --use-local-docs: Use bundled local HTML docs instead of retrieving documentation live