Skip to main content

clipsmith

Local Twitch → AI clip pipeline. Downloads a VOD, transcribes Spanish audio, ranks moments by chat activity, sends candidates to an LLM, and cuts 9:16 vertical MP4s — optionally with burned-in captions and a stacked webcam/gameplay layout for TikTok / YouTube Shorts.

Prerequisites

Tool Install
Python 3.11+ python.org or winget install Python.Python.3
ffmpeg winget install Gyan.FFmpeg — must be on PATH
twitch-dl bundled via pip install -e .
chat-downloader bundled via pip install -e .
faster-whisper bundled via pip install -e .

Optional — webcam auto-detection:

pip install -e ".[vision]"   # installs opencv-python-headless

Installation

git clone https://github.com/ricardogr07/clipsmith
cd clipsmith
pip install -e .

Configuration

Secrets — .env

Copy .env.example to .env and fill in your keys:

TWITCH_CLIENT_ID=...
TWITCH_CLIENT_SECRET=...
ANTHROPIC_API_KEY=...       # if using provider: anthropic
OPENAI_API_KEY=...          # if using provider: openai

Get Twitch credentials at https://dev.twitch.tv/console → Register Your Application → OAuth Redirect: http://localhost.

Behaviour — config.yaml

channels:
  - chuyelwuero             # Twitch logins to watch

llm:
  provider: anthropic       # anthropic | openai | ollama
  model_anthropic: claude-sonnet-4-6

clip:
  min_seconds: 15
  max_seconds: 30

caption:
  enabled: false            # set true to burn subtitles into the video
  font: Arial
  font_size: 72             # ASS pts at 1080×1920 PlayRes
  outline: 3
  position: bottom          # bottom | middle | top

reframe:
  # center | webcam | stacked | none
  mode: none
  # Source-pixel crop for the webcam/face panel [x, y, w, h]
  # Leave null to use center-crop fallback; set once per stream layout.
  webcam_rect: null
  # Source-pixel crop for the gameplay panel (stacked mode only)
  # null = center-crop fallback
  gameplay_rect: null
  # Fraction of 1920px height given to the top (webcam) panel in stacked mode
  # e.g. 0.4 → 768px webcam on top, 1152px gameplay on bottom
  split_ratio: 0.4

Usage

First-time setup

clipsmith setup

Saves your API key to .env and verifies ffmpeg is available.

Process a local MP4

clipsmith process path/to/recording.mp4

Runs the full pipeline — transcribe → score candidates → LLM selection → cut clips. Clips appear in out/<video_id>/.

Useful flags:

Flag Effect
--skip-transcribe Load cached transcript.json
--captions / --no-captions Override caption config
--reframe / --no-reframe Override reframe config
--provider anthropic|openai Override LLM provider

Daemon mode

clipsmith watch

Polls every poll_interval_s seconds (default 120). When a new archive VOD appears it runs the full pipeline automatically. State is persisted to state.json so already-processed VODs are skipped across restarts.

One-off Twitch VOD

clipsmith run-vod <video_id>

Useful flags:

Flag Effect
--skip-download Use the existing .mp4 in work/<id>/
--skip-transcribe Load cached transcript.json
--skip-chat Load cached chat.json
--skip-select Stop after candidates, skip LLM
--skip-clip Stop after picks, skip ffmpeg
--provider anthropic|openai Override config LLM
--max-candidates N Cap candidates sent to LLM (default 20)

Re-cut flat clips

Re-run only the ffmpeg step from an existing picks.json (e.g. after adjusting caption style):

clipsmith clip <video_id>

Stacked vertical layout (webcam + gameplay)

After reviewing the flat clips, pick the best ones and reframe them into a stacked 9:16 layout — webcam on top, gameplay on bottom:

clipsmith reframe <video_id> clip_01 clip_04 clip_09

Output goes to out/<video_id>/stacked/. The reframe rects come from config.yaml; see the reframe section above.

Auto-detecting the webcam rect:

If reframe.webcam_rect is not set in config.yaml, clipsmith will try to detect the face rectangle automatically using OpenCV (requires pip install -e ".[vision]"). The result is cached to work/<video_id>/webcam_rect.json and reused on subsequent runs.

To run detection manually and write the result directly into config.yaml:

clipsmith detect-webcam <video_id>

This samples frames, detects the webcam rectangle, and saves reframe.webcam_rect in config.yaml automatically — no manual copy-paste needed.

Sanity check

clipsmith whoami chuyelwuero

Output

work/
  <video_id>/
    <video_id>.mp4          downloaded (or copied) source VOD
    transcript.json         faster-whisper segments + word timestamps
    chat.json               chat replay
    candidates.json         ranked candidate moments
    picks.json              LLM-accepted clips with titles and reasons
    webcam_rect.json        auto-detected webcam rect (cached, vision only)

out/
  <video_id>/
    clip_01_momento_gracioso.mp4    flat 9:16 clip
    clip_01_momento_gracioso.ass    sidecar ASS subtitle file
    clip_02_reaccion_epica.mp4
    ...
    stacked/
      clip_01_momento_gracioso.mp4  stacked layout (webcam top, gameplay bottom)
      clip_04_reaccion_epica.mp4
      ...

All clips are 1080×1920 (9:16), 15–30 s, h264/aac, with optional burned-in Spanish karaoke captions.

Pipeline overview

watch / run-vod / process
  ↓
downloader      (twitch-dl)                → work/<id>/<id>.mp4
  ↓
detect-webcam   (opencv Haar cascade)      → work/<id>/webcam_rect.json  [optional]
  ↓
transcribe      (faster-whisper)           → transcript.json  (word timestamps, lang=es)
  ↓
chat            (chat-downloader)          → chat.json
  ↓
candidates      (density + clips + !clip)  → candidates.json
  ↓
selector        (LLM)                      → picks.json
  ↓
clipper         (ffmpeg + libass)          → out/<id>/clip_NN_<title>.mp4
  ↓  [manual: clipsmith reframe]
reframe         (ffmpeg filter_complex)    → out/<id>/stacked/clip_NN_<title>.mp4

REST API & Dashboard

Start the FastAPI server and the Next.js dashboard to review and approve clips in a browser:

pip install -e ".[server]"
clipsmith serve                   # http://localhost:8000/docs
cd web && pnpm install && pnpm dev  # http://localhost:3000

Key API endpoints:

Endpoint Description
POST /runs Start a pipeline run (requires X-Api-Key header)
GET /runs/{id}/progress SSE stream of live stage progress
PATCH /clips/{id} Approve or reject a clip
GET /metrics Prometheus scrape endpoint
GET /stats Run counts by status (JSON)

Observability

Sprint 7 adds distributed tracing (OpenTelemetry → Jaeger), Prometheus metrics, and a Grafana dashboard. Spin up the full local stack in one command:

pip install -e ".[observability]"
docker compose -f docker-compose.observability.yml up -d
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 clipsmith serve
UI URL
Grafana dashboard http://localhost:3001 (admin / $GRAFANA_ADMIN_PASSWORD, default: admin)
Jaeger traces http://localhost:16686
Prometheus http://localhost:9090

In production, set APPLICATIONINSIGHTS_CONNECTION_STRING to export traces to Azure Monitor Application Insights.

Development

pip install -e ".[dev,server,observability]"
pip install -e ".[vision]"   # optional, for detect-webcam tests
python -m pytest tests -q

Metadata

Release files for clipsmith-ai 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for clipsmith-ai 0.3.0
File Size Uploaded Provenance Actions
clipsmith_ai-0.3.0.tar.gz 1.0 MB With provenance Details for clipsmith_ai-0.3.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for clipsmith-ai 0.3.0
File Interpreter ABI Platform Actions
clipsmith_ai-0.3.0-py3-none-any.whl Python 3 none any Details for clipsmith_ai-0.3.0-py3-none-any.whl

Total release size: 1.1 MB

Release files / clipsmith_ai-0.3.0.tar.gz

Download URL clipsmith_ai-0.3.0.tar.gz
Size 1.0 MB
Tags Source
SHA-256 checksum
How to use checksums
6fbeffca73fc10872fa4994673fe37b3c00c3281065e7e182c102de5d55db49f
BLAKE2b-256 checksum
How to use checksums
842dd668c75ef53fecfd9fa99da5e84286d1cbbc2389e0567b65877d5064aec3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

Release files / clipsmith_ai-0.3.0-py3-none-any.whl

Download URL clipsmith_ai-0.3.0-py3-none-any.whl
Size 89.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
cd199c61b04cbe9c2194e0cca9a8ae13c36dd5fef2e212c2b0c9e092d85d105b
BLAKE2b-256 checksum
How to use checksums
74fe4c9a9815ee5fa158eba0c7a954e5ef15ea1f5b72bb475d154178870c84cc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

Security for clipsmith-ai 0.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for clipsmith_ai-0.3.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page