spectrumalyzer

Turn any song into animation-ready data. Give it music — get beats, BPM, onsets, frequency bands, song sections and loudness, delivered in the exact format your creative tool speaks: Remotion, HyperFrames, After Effects, Final Cut / DaVinci Resolve, spreadsheets, or plain JSON.

No subscriptions, no uploads, no cloud. Everything runs locally with two open-source workhorses: Python + ffmpeg.

Why spectrumalyzer?

Editors and animators don't think in seconds — they think in frames. Most audio tools stop at "here are some timestamps." spectrumalyzer goes the last mile:

• Frame-native output — beats pre-converted to your project's frame rate (24 / 25 / 30 / 60 fps), so there's zero runtime math in your compositions.
• Speaks your tool's dialect — a React hook for Remotion, a marker script for After Effects, an FCPXML for Final Cut and Resolve, a timeline JSON for HyperFrames.
• Musically grounded — the beat grid is phase-locked to the kick drum, not to generic loudness spikes, so your cuts land on the beat even when hi-hats are louder than the bass.
• Honest data — every signal is a real measurement (FFT-based), with confidence scores and documented limitations.

How it works — Architecture

spectrumalyzer is a pipeline: decode → analyze → frame-ize → export. Each stage is an independent, testable component.

flowchart LR
    subgraph DECODE["1 · Decode"]
        A["Audio file<br/>mp3 · wav · flac · ogg · m4a · video…"] --> B["ffmpeg → mono PCM<br/>44.1 kHz float"]
    end

    subgraph ANALYZE["2 · Analyze"]
        B --> C["STFT<br/>sliding Fourier transform"]
        C --> D["Spectral flux<br/>onset strength envelope"]
        D --> E["Onset picking<br/>→ raw hits + strengths"]
        D --> F["Tempo<br/>autocorrelation → BPM + confidence"]
        F --> G["Beat grid<br/>kick-anchored phase lock"]
        C --> H["6 frequency bands<br/>sub → treble per frame"]
        B --> I["Loudness contour<br/>+ waveform peaks"]
        I --> J["Sections<br/>quiet · mid · high · peak"]
    end

    subgraph FRAME["3 · Frame-ize"]
        K["time_to_frame<br/>+ band resampling at --fps"]
    end

    subgraph EXPORT["4 · Export"]
        L["6 exporters<br/>+ HTML report"]
    end

    E --> K
    G --> K
    H --> K
    J --> K
    K --> L

The stages in plain language

Component map

Quick start

# 1. Install (requires ffmpeg on your system)
pip install spectrumalyzer

# 2. Analyze a track for a 30 fps project, in three formats at once
spectrumalyzer analyze song.mp3 --fps 30 --format remotion,csv,ae

# 3. Open a visual report in your browser
spectrumalyzer report song.mp3 -o song.html

That's it. You now have beat numbers in frames, a marker script for After Effects, a CSV for Resolve, and a page that shows you the song's anatomy.

Installation

One install, everything included — DSP engine + semantic hearing:

# macOS
brew install ffmpeg && pip install spectrumalyzer

# Debian / Ubuntu / Mint
sudo apt install ffmpeg && pip install spectrumalyzer

# Windows (winget or choco)
winget install ffmpeg && pip install spectrumalyzer

The embedding model weights (~90 MB) download automatically the first time you run describe, find or analyze --embed — or pull them upfront with spectrumalyzer model pull. Everything runs locally; nothing leaves your machine.

Check your setup:

spectrumalyzer --version    # CLI is on PATH
ffmpeg -version             # decoder is reachable
spectrumalyzer model status # embedding model state

The three commands

1. analyze — the workhorse

spectrumalyzer analyze AUDIO [--fps N] [--format LIST] [options]

Example — everything, for a 24 fps film project:

spectrumalyzer analyze soundtrack.flac --fps 24 \
  --format json,remotion,hyperframes,ae,csv,fcpxml \
  --min-bpm 70 --max-bpm 180 -o deliverables/soundtrack

2. report — see the music

spectrumalyzer report AUDIO [-o report.html] [--fps N]

Generates one self-contained HTML file (no server, no external assets): waveform with the beat grid drawn over it, a band-by-time heatmap, the section strip, and all key stats. Perfect for client reviews or deciding where the edit's energy should go before you animate anything.

3. formats — the cheat sheet

spectrumalyzer formats

Lists every export format, its file suffix and what it's for.

Output formats — which one do I use?

Remotion in 30 seconds

spectrumalyzer analyze song.mp3 --fps 30 --format remotion
# → song.remotion.json + song.useBeats.ts — copy both next to your composition

import { useBeatPop } from "./song.useBeats";

export const MyScene = () => {
  const scale = useBeatPop(0.15);   // pops 15% on every beat, springs back
  return <h1 style={{ transform: `scale(${scale})` }}>ON BEAT</h1>;
};

The generated hook also includes useBeatIndex(), useBeatProgress(), useIsOnBeat(window), and the raw BEATS / BPM constants.

After Effects in 30 seconds

spectrumalyzer analyze song.mp3 --fps 30 --format ae
# → song.markers.jsx — then: File → Scripts → Run Script File…

Every beat, hit and section becomes a named comp marker (beat 1, hit, section: peak) — enable Layer → Time → marker snapping and keyframe straight to the music.

Raw JSON anatomy (abridged)

{
  "bpm": 128.32,
  "bpmConfidence": 0.91,          // 0..1 — how strong the tempo pulse is
  "fps": 30,
  "durationInFrames": 7690,
  "beats": [
    { "time": 0.5012, "frame": 15, "strength": 0.87 }
  ],
  "onsets": [                      // raw transients — hits, not grid beats
    { "time": 0.1203, "frame": 4, "strength": 1.0 }
  ],
  "bands": {
    "names": ["sub", "bass", "lowMid", "mid", "highMid", "treble"],
    "frameDuration": 0.0232,
    "matrix": [[0.0, 0.42, 0.11, 0.05, 0.02, 0.01], "…per analysis frame"]
  },
  "bandsByFrame": "…same 6 values, resampled to one row per VIDEO frame",
  "sections": [
    { "start": 0.0, "end": 24.8, "label": "quiet", "energy": 0.22,
      "startFrame": 0, "endFrame": 744 }
  ]
}

The signals — and what to animate with them

Python API

Prefer scripting? One import does everything:

from spectrumalyzer import analyze

result = analyze("song.mp3", fps=30)

result.bpm                    # 128.32
result.bpm_confidence         # 0.91
result.beats_as_frames()      # [15, 53, 91, ...]  at 30 fps
result.band_rows_at_fps()     # one [sub..treble] row per video frame
result.sections               # [Section(start, end, energy, label), ...]
result.to_dict()              # JSON-ready dict

Every option from the CLI is a keyword argument (bands=False, sections=False, min_bpm=70, max_bpm=180, …).

Troubleshooting & FAQ

ffmpeg not found — install it (see Installation) and make sure ffmpeg -version works in your terminal.

BPM is double or half what I expect. Heavily syncopated music can report the 8th-note pulse or the half-time feel. Constrain the search: --min-bpm 100 --max-bpm 140 for a house track you know is ~120.

bpmConfidence is low (< 0.5). The pulse is unclear — ambient, live drums, or rubato material. The beat grid may drift; prefer onsets[] (raw hits) for such tracks, or widen the BPM window.

Beats feel slightly off on hi-hat-heavy music. That was the v0.1 problem — v0.2 anchors the grid to the kick. If a track has no low-frequency pulse (acoustic, choir), the grid falls back to onset alignment (±12 ms).

Remotion export refuses to run. It needs --fps — frame numbers can't be computed without a frame rate.

Can it read video files? Yes — any file ffmpeg reads, including MP4/MOV; the audio track is extracted automatically.

Where is the old spectrum key? v0.1's "spectrum" was not a real FFT (it was loudness slices wearing frequency labels). v0.2 replaced it with the real bands matrix. bpm, beats, beatTimes are unchanged; see the CHANGELOG.

Development

git clone https://github.com/Dream-Pixels-Forge/spectrumalyzer
cd spectrumalyzer
pip install -e ".[dev]"      # pytest + ruff
pytest                       # 44 tests, synthetic fixtures, fully offline
ruff check src tests

Roadmap (downbeats, musical key detection, cut-list generation, DAW project stubs, Resolve scripting): see docs/ROADMAP.md.

License

MIT — see LICENSE. Built by Dream-Pixels-Forge.

Semantic hearing (v0.3) — describe, tag, and find music by meaning

Beyond beats, spectrumalyzer ships with a small local embedding model (EmbeddingGemma-300m, Google's current small Gemma — with a non-gated automatic fallback) to give its measurements language:

# describe a track in words + mood/energy/genre tags (weights auto-download on first run)
spectrumalyzer describe song.mp3

# semantic search across a library — plain language in, ranked tracks out
spectrumalyzer find "calm warm bed for a documentary intro" ./my-library
spectrumalyzer find "aggressive intense banger for a workout" ./my-library

# attach description + tags to any analysis
spectrumalyzer analyze song.mp3 --embed -o enriched.json

How it stays honest: tags are not magic. Every track is summarized in words derived from measurements (tempo class, dBFS loudness, brightness, dominant frequency band, transient density, section contrast), then tagged by blending embedding similarity with a numeric feature-profile match per descriptor. Pulseless ambient material is reported as "no clear pulse" — never an invented BPM.

The model lives in ~/.config/spectrumalyzer/embedding/ and everything runs locally — no audio or text ever leaves your machine. The core DSP analysis never depends on it.

MCP server — spectrumalyzer as tools for AI agents

Run it

spectrumalyzer mcp        # stdio MCP server (blocking)

Register with your MCP client

Claude Desktop / any JSON-config host — add to the MCP config:

{
  "mcpServers": {
    "spectrumalyzer": {"command": "spectrumalyzer", "args": ["mcp"]}
  }
}

pi / pi-coding-agent — add to ~/.pi/agent/mcp.json:

{"mcpServers": {"spectrumalyzer": {"command": "spectrumalyzer", "args": ["mcp"]}}}

Once registered, the agent gets these tools natively:

All tools return structured {"error": ...} payloads instead of crashing, so agents can react gracefully. Everything runs locally.

Agent skill — teach your agent to use it

Install the spectrumalyzer skill into any skill-aware agent (pi, Claude Code, OpenCode, …):

npx skills add github.com/Dream-Pixels-Forge/spectrumalyzer/tree/main/skills/

The skill lives in this repo at skills/spectrumalyzer/SKILL.md and covers when and how to use spectrumalyzer: analysis for scene timing, semantic music search, the signal→animation mapping (bass → glow, treble → sparkle), MCP tool selection, and export format choice per creative tool. Useful even before the MCP server is registered — and complementary to it.