Generate Text Contexts from Youtube Shorts
Turn a short-form Youtube video into structured, timestamped context in a text format. It includes both audio and visual context.
Most language models can't watch a video. This tool gives them a faithful text version of one: what was said, what was written on screen, and what was shown, with every item labelled by source and time.
[ON-SCREEN:LOGO 00:00–00:16] EXAMPLE CHANNEL
[AUDIO 00:07–00:11] Today we're testing three ways to brew coffee at home.
[ON-SCREEN 00:08–00:10] 3 WAYS TO BREW COFFEE
[VISUAL 00:07–00:16] A person in an apron pours hot water into a glass coffee maker in a bright kitchen.
Use the output as the input to summarisation, search, fact-checking, content moderation, RAG, or your own agents.
Contents
- How it works
- Quick start
- Choosing a vision model
- Usage
- Use it from AI tools (MCP)
- Output
- Troubleshooting
- Privacy and responsible use
- Development
- License
How it works
YouTube link or video file
│
download and check
│
├── audio ──► speech-to-text (Whisper, runs locally)
│
└── frames ─► vision model: on-screen text, scenes, settings, editing
│
record.json + record.md
Quick start
1. Install the requirements
You need uv, ffmpeg, and about 5 GB of free disk space. uv installs the right Python version for you.
| System | Install command |
|---|---|
| macOS | brew install uv ffmpeg |
| Ubuntu or Debian | sudo apt install ffmpeg && curl -LsSf https://astral.sh/uv/install.sh | sh |
2. Install video-to-context
uv tool install --python 3.12 video-to-context
Add uv's tools to your PATH, then open a new terminal:
uv tool update-shell
You can also use pip install video-to-context inside a Python 3.12 environment.
3. Choose a vision model (optional)
Download the settings template:
mkdir -p ~/.video-to-context
curl -o ~/.video-to-context/.env https://raw.githubusercontent.com/wani-ham/youtube-shorts-to-text-context/main/.env.example
Open ~/.video-to-context/.env and uncomment one option from the next section. Without one, the tool still produces the transcript and reads on-screen text locally, but it won't include visual analysis.
4. Run it
video-to-context "https://www.youtube.com/shorts/VIDEO_ID"
The first run is slower because it downloads the speech and OCR models. When it finishes, it prints the path to your result in a store/ folder inside your current directory.
Choosing a vision model
You don't need Claude. Any service with an OpenAI-compatible API and image support will work.
| Option | Cost | Your frames go to | Best for |
|---|---|---|---|
| Claude Haiku 5.5 | About 1 cent per video (estimate) | Anthropic | Tested default |
| Gemini | Free tier available | Trying it at no cost | |
| OpenAI | Paid | OpenAI | Existing OpenAI users |
| OpenRouter | Free and paid models | The selected provider | Comparing models |
| Ollama | Free | Nowhere, runs locally | Private text extraction, no account |
| None | Free | Nowhere | Transcript and text only |
Status: Claude has been tested on real videos. Ollama with Gemma 4 12B has been tested on one real video: it read on-screen text well, but its scene descriptions were unreliable. Gemini, OpenAI, and OpenRouter use the same connection as Ollama but haven't been tested with their real services.
Claude
Create a key in the Claude Console. API usage is billed separately from Claude Pro and Max plans.
ANTHROPIC_API_KEY=your-key
Gemini
Create a key in Google AI Studio. Free-tier content may be used to improve Google's products, so avoid private videos on that tier.
VISION_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/
VISION_API_KEY=your-key
VISION_MODEL=gemini-3.8-flash
OpenAI
VISION_BASE_URL=https://api.openai.com/v1
VISION_API_KEY=your-key
VISION_MODEL=gpt-5.4-mini
OpenRouter
Choose a model that accepts images from openrouter.ai/models.
VISION_BASE_URL=https://openrouter.ai/api/v1
VISION_API_KEY=your-key
VISION_MODEL=provider/model-name
Ollama (free and local)
Install Ollama, then download a vision model. Allow about 8 GB of disk space and at least 16 GB of memory, and close other heavy apps.
ollama pull gemma4:12b
Start Ollama with a larger context. Its default is too small for this prompt and the images:
OLLAMA_CONTEXT_LENGTH=16384 ollama serve
If Ollama is already running in the background, quit it first. Then add this to .env:
VISION_BASE_URL=http://localhost:11434/v1
VISION_MODEL=gemma4:12b
VISION_MAX_FRAMES=12
VISION_REASONING_EFFORT=none
VISION_TIMEOUT_S=600
VISION_MAX_FRAMESlimits how many images are sent per video. Fewer frames use less memory but can miss short captions.VISION_REASONING_EFFORT=noneturns off model thinking. On a 16 GB Apple Silicon Mac, one request took about 78 seconds this way and about 250 seconds with thinking, which gave somewhat better scene descriptions.VISION_TIMEOUT_Sgives slower local models enough time to answer.
Model names change often. If one is rejected, check your provider's current model list.
Usage
# A YouTube video or Short
video-to-context "https://www.youtube.com/shorts/VIDEO_ID"
# A video file on your computer
video-to-context --file path/to/video.mp4
# A file with details about where it came from
video-to-context --file ad.mp4 --sponsor "Brand" --date 2026-11-02 --caption "Post text"
# Process a video again, ignoring the saved result
video-to-context "https://youtu.be/VIDEO_ID" --force
Videos can be up to 5 minutes and 200 MB. To change these limits, edit config.py in a source install.
The command exits with 0 for a complete record, 1 for a partial record, and 2 for a rejected input.
Use it from Python
from dotenv import load_dotenv
from video_to_context.run import process
load_dotenv()
record = process(url="https://www.youtube.com/shorts/VIDEO_ID", out="store")
print(record.llm_document)
print(record.visual_context.summary if record.visual_context else "No visual analysis")
Use it from AI tools (MCP)
The included MCP server lets AI tools analyse a video when you ask, for example: "Summarise this Short and tell me what its on-screen text claims."
Finish the Quick start first. The server reads your settings from ~/.video-to-context/.env and stores results in ~/.video-to-context/store.
Claude Code
claude mcp add --scope user video-to-text -- "$(command -v video-to-context-mcp)"
Cursor, Claude Desktop, and other clients
Find the full command path with command -v video-to-context-mcp, then add it to the client's MCP configuration. For Cursor, that's ~/.cursor/mcp.json.
{
"mcpServers": {
"video-to-text": {
"command": "/absolute/path/to/video-to-context-mcp"
}
}
}
Tools
| Tool | What it does |
|---|---|
start_video_analysis |
Starts a background job for a YouTube link or allowed local file |
get_analysis_status |
Reports whether the job is queued, running, or finished |
get_video_text |
Returns the finished result as Markdown or JSON |
list_analyzed_videos |
Lists videos that already have results |
check_setup |
Checks ffmpeg, the vision model, and file access |
Videos run one at a time in the background, because processing takes longer than most tools wait for a single call.
Security
- Local files are off by default. To allow them, list specific folders in
VIDEO_TEXT_ALLOWED_DIRSin~/.video-to-context/.env. - URLs are limited to YouTube.
- The server runs locally over stdio and opens no network port.
- Video text is untrusted. A caption could try to manipulate your AI tool, so be careful when that tool can also run commands or edit files.
Output
Each video gets its own folder, named after a hash of the file, so the same file is never processed twice:
store/<sha256>/
├── record.md Document for LLMs and people
├── record.json Full structured data for code
├── video.mp4 The video
├── frames/ Keyframes referenced by the record
└── info.json YouTube metadata (for YouTube links)
record.md
Pass this file directly to a language model. It contains:
- YAML metadata: source, duration, language, footage type, status, and models used.
- Quality notes: anything downstream services should be careful about.
- Visual analysis: a summary, the distinct settings, editing techniques, and shots.
- Timeline: everything in time order, labelled by source.
| Label | Meaning | Safe to quote as the video's content? |
|---|---|---|
AUDIO |
Transcribed speech | Yes, but it may have recognition errors |
ON-SCREEN |
Captions and text in the frame | Yes |
ON-SCREEN:HEADLINE, :WATERMARK, :LOGO, :OTHER |
Text or marks that stay on screen | Yes |
VISUAL |
A model's description of a scene | No, it is an interpretation |
Video text is untrusted input. It's kept inside code blocks so that your prompt can treat it as data, but a caption could still contain text written to manipulate a model. Tell your downstream model not to follow instructions found in it.
record.json
The same information as structured fields, including transcript, on_screen_text, visual_context, visual_descriptions, keyframes, flags, and errors. Suspected invented speech is moved to transcript.removed_segments so you can review it.
If one stage fails, the others still finish and the record is marked partial.
Troubleshooting
| What you see | What to do |
|---|---|
fetch_failed |
The video may be private, removed, or age-restricted, or YouTube may be blocking the download. Download limits are stricter on cloud servers. Use --file instead. |
vision_not_configured in flags |
No vision model is set. Check that ~/.video-to-context/.env exists and an option is uncommented. |
vision_auth_failed |
The API key is wrong or expired. |
vision_invalid_output |
The model didn't return usable JSON. Try a more capable model. |
| Out of memory with Ollama | Lower VISION_MAX_FRAMES or use a smaller model. |
command not found: video-to-context |
Run uv tool update-shell, then open a new terminal. |
| MCP tools don't appear | Use the full path from command -v video-to-context-mcp, then restart the client. |
vision_timeout |
Raise VISION_TIMEOUT_S, or set VISION_REASONING_EFFORT=none. |
objc: Class AVF... is implemented in both |
A harmless macOS warning from two video libraries. |
ffmpeg not found |
Install ffmpeg (step 1), then open a new terminal. |
| It's slow | Speech-to-text runs on CPU unless you have an NVIDIA GPU. Expect a few minutes for a 2-minute video. |
Privacy and responsible use
- Unless you use Ollama or no vision model, video frames are sent to the provider you choose. Check its data retention and training terms.
- Videos contain people's faces and voices. The vision prompt tells the model not to identify people, and you shouldn't add face or voice identification without legal review.
- YouTube's terms restrict downloading. Make sure you have the right to process a video, especially for commercial or public use.
.env,store/, and media files are ignored by git. Never commit API keys.
Development
git clone https://github.com/wani-ham/youtube-shorts-to-text-context.git
cd youtube-shorts-to-text-context
python3.12 -m venv .venv
.venv/bin/pip install -e ".[dev]"
cp .env.example .env
.venv/bin/python -m pytest
.venv/bin/python -m video_to_context.evaluate
In a source checkout, settings come from the project's .env, and the MCP server stores results in the project's store/ folder.
To score accuracy on your own test set, put each video in tests/samples/<name>.mp4 and its hand-written labels in tests/expected/<name>.json:
{"transcript": "…", "on_screen": ["…"], "no_speech": false, "rejected": null}
License
Licensed under the Apache License 2.0. See NOTICE for copyright details.
Metadata
Release files for video-to-context 0.5.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| video_to_context-0.5.1.tar.gz | 46.1 kB | Without provenance | Details for video_to_context-0.5.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| video_to_context-0.5.1-py3-none-any.whl | Python 3 | none | any | Details for video_to_context-0.5.1-py3-none-any.whl |
Total release size: 89.8 kB
Release files / video_to_context-0.5.1.tar.gz
| Download URL | video_to_context-0.5.1.tar.gz |
|---|---|
| Size | 46.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cc6ca9bfcc53676a5beb965160e5d776a11a5718964f8ede0cd1365446d5857e
|
|
BLAKE2b-256 checksum How to use checksums |
beae1288bf5e8b103d3a0de981912b371a3021d809393e1e263ffe3107f016c0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.16
|
Release files / video_to_context-0.5.1-py3-none-any.whl
| Download URL | video_to_context-0.5.1-py3-none-any.whl |
|---|---|
| Size | 43.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1caf5cb5c09a1e940aa0549f1d559b15f89c43327a64ba80273f2eeadf210ab5
|
|
BLAKE2b-256 checksum How to use checksums |
4986aba2d0be88b664c3871062d2e233e203b216e3f434d8f6299aa78e161916
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.16
|
Security for video-to-context 0.5.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.