mcp-content-pipeline

A content analysis and digest pipeline for YouTube videos and X (Twitter) feeds, exposed as MCP tools. Extract transcripts, fetch posts from curated accounts, and generate key takeaways, TLDRs, social hooks, and comic-book infographics — all callable by any MCP-compatible AI client like Claude Desktop.

flowchart LR
    A[YouTube URL<br/>or X feed] --> B[Extract content<br/>Supadata / X API]
    B --> C[Claude analysis<br/>takeaways, TLDR, hook]
    C --> D[Gemini image<br/>comic infographic]
    D --> E[Sync to GitHub<br/>markdown + image]

Why?

Keeping up with YouTube channels and X accounts means scattered tabs, manual note-taking, and lost insights. This MCP server turns content consumption into structured, chainable tools. Analyse a Bloomberg video, digest your X feed, generate infographics, and sync everything to GitHub — all from a single conversation with Claude.

Role in ecosystem

• Uses: mcp-llm-eval for evaluation and CI quality gates
• Produces: benchmark JSON written to llm-benchmarks under text-generation/content-pipeline-*.json
• Visible at: LLMShot's Text Generation domain, Content Pipeline sub-benchmark

The eval dataset (eval/dataset.json) lives with this repo because the questions are specific to YouTube and X feed analysis — the dataset belongs with the use case, not the engine.

Quick Start

uvx mcp-content-pipeline

Or install explicitly:

uv tool install mcp-content-pipeline
mcp-content-pipeline

Claude Desktop Configuration

Add to your Claude Desktop MCP config (~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "content-pipeline": {
      "command": "/usr/local/bin/uvx",
      "args": ["mcp-content-pipeline"],
      "env": {
        "MCP_CP_ANTHROPIC_API_KEY": "sk-ant-...",
        "MCP_CP_SUPADATA_API_KEY": "sd_...",
        "MCP_CP_GITHUB_TOKEN": "ghp_...",
        "MCP_CP_GITHUB_REPO": "your-username/your-repo",
        "MCP_CP_GEMINI_API_KEY": "your-gemini-api-key",
        "MCP_CP_X_BEARER_TOKEN": "your-x-bearer-token",
        "MCP_CP_X_ACCOUNTS": "karpathy,bcherny,atmoio,steipete",
        "MCP_CP_X_TOPICS": "AI,tech,engineering"
      }
    }
  }
}

Usage

Once configured in Claude Desktop, use the tools in a single conversation.

Tip: Including "content-pipeline" for YouTube or "X feed" for Twitter helps Claude Desktop route to the right tool.

YouTube Analysis

"Use content-pipeline to analyse this video: https://www.youtube.com/watch?v=..." "Generate an image for this analysis" "Sync the analysis and image to GitHub"

Or all in one prompt:

"Use content-pipeline to analyse this video, generate the image, and sync to GitHub: https://www.youtube.com/watch?v=..."

X Feed Digest

"Analyse the X feed" "Analyse the X feed for karpathy, bcherny, atmoio, and steipete about AI today" "Analyse the X feed from the last 7 days"

Or with the full pipeline:

"Analyse the X feed, generate the image, and sync to GitHub"

Tools

Environment Variables

All prefixed with MCP_CP_:

Cost Projections

Estimated monthly costs for two usage patterns:

Claude API costs depend on your Anthropic billing plan and are not included in the totals above. If you already use Claude Pro ($20/mo), there is no additional Claude cost. The X API spending cap can be configured in the developer console.

What this replaces

Eval Gates

Prompt and model changes are automatically evaluated in CI using mcp-llm-eval. The eval dataset covers both YouTube analysis and X feed digest prompts, benchmarking 8 models (Claude Opus 4.7, Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.5, GPT-4o-mini, Gemini 3 Flash Preview, Gemini 2.5 Flash, Gemini 2.5 Flash-Lite) on the same test cases. PRs that touch system prompts or model config trigger an evaluation run that scores faithfulness and relevance against a reference dataset. The PR is blocked if quality regresses below configured thresholds.

See .eval-gate.yml for threshold configuration and eval/dataset.json for the test dataset.

Running benchmarks locally

The benchmark requires API keys for all providers. Create a .env file in the project root:

ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
GOOGLE_API_KEY=AIza...

Then run:

make benchmark        # Run eval against all 8 models
make benchmark-copy   # Copy results to llm-benchmarks repo

Results are written to eval/results/ (gitignored). The benchmark output feeds into LLMShot via the llm-benchmarks repo at text-generation/content-pipeline-summary.json and text-generation/content-pipeline-benchmark.json.

This project uses for benchmarking and CI quality gates. Production uses Claude Sonnet (claude-sonnet-4-6). The benchmark tracks all 8 models (3 Anthropic, 2 OpenAI, 3 Google) so we can re-evaluate provider choice as capabilities and pricing evolve.

Development

git clone https://github.com/your-username/mcp-content-pipeline.git
cd mcp-content-pipeline
uv sync
uv run pytest -v --cov=src/mcp_content_pipeline
uv run ruff check src/ tests/

Security

• All credentials are configured via local environment variables — never committed to the repo
• The tool is open source but your API keys, YouTube key, and GitHub token stay on your machine
• Never create a .env file in the repo — use shell exports or Claude Desktop config instead

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feat/my-feature)
3. Commit using Conventional Commits (feat: add new feature)
4. Push and open a Pull Request

License

MIT