ark-audio-agent

Autonomous text → music. Give it a sentence; it reasons about the feeling, composes a structured piece, and renders it to MIDI + audio — with no human in the loop.

The interesting part is the harness: composition is driven by a team of specialist agents running a ReAct loop (reason → act → observe) over a shared musical blackboard. It runs on the Claude API when a key is available, and on a fully-offline deterministic music-theory brain otherwise, so it always produces a result.

ark-audio "a rainy night in a neon city, lonely but hopeful"
# → renders/a-rainy-night-in-a-neon-city-lonely-but-hopeful/song.mp3  (+ .wav, .mid, report.md, trace.jsonl)

Why it's built this way

Rather than ask one model to emit raw notes (which drifts out of key and is hard to steer), the system separates musical intent from musical realization:

• Agents decide intent — key, mode, tempo, song form, chord progressions, a melody's contour/register/density, instrument roles, drum and bass styles.
• Deterministic engines realize it — pure music-theory functions turn that intent into in-key notes, which a mido writer turns into MIDI and a from-scratch numpy synthesizer turns into audio.

Because every decision an agent can make is a tool, both brains (Claude and the offline planner) drive the exact same action surface and produce the exact same trace format.

Architecture

            text prompt
                │
     ┌──────────▼───────────────────────────────────────────────┐
     │  Orchestrator (the harness)                                │
     │                                                            │
     │   Interpreter → Composer → Melodist → Arranger             │  each agent runs a
     │                               │                            │  ReAct loop:
     │                               ▼                            │   reason (thinking)
     │                            Critic ──reject──┐              │   → act  (tool call)
     │                               │approve      │ one targeted │   → observe (result)
     │                               │        revision (bounded)  │
     └───────────────────────────────┼────────────┘              │
                                      │                           │
          ┌───────────────────────────▼───────────────────────┐  │
          │  Tools  (the ReAct action space)                   │  │
          │  set_brief · get_theory · set_global ·             │  │
          │  set_structure · set_progression · design_melody · │  │
          │  set_instruments · set_bass · set_drums ·          │  │
          │  analyze_score · finish / approve / request_revision│ │
          └───────────────────────────┬───────────────────────┘  │
                                       ▼                          │
                            Score blackboard (shared state)       │
     └──────────────────────────────┬─────────────────────────────┘
                                     ▼
          performance.py  →  { melody, harmony, bass, drums } notes
                                     │
                 ┌───────────────────┴────────────────────┐
                 ▼                                         ▼
        midi_render.py (mido)                   synth.py (numpy software synth)
                 │                                         │
              song.mid                        song.wav ──(ffmpeg)──► song.mp3

The agents

Every agent also has analyze_score (a sensing action) and a terminal finish.

Install

Uses uv.

uv sync --extra llm      # installs the optional anthropic SDK for the Claude brain
# or, without the Claude brain (offline only):
uv sync

ffmpeg is optional — if present, an MP3 is exported alongside the WAV.

Usage

CLI

ark-audio "triumphant sunrise over the mountains"
ark-audio "a quiet dawn over still water" --offline --seed 7
ark-audio "lonely synthwave drive at 2am" --out renders/drive --no-mp3
echo "a storm breaking over the sea" | ark-audio      # reads stdin

Flags: --offline (force the deterministic brain), --seed, --out, --model, --effort low|medium|high|xhigh|max, --max-revisions, --reverb, --no-mp3.

Library

from ark_audio_agent import compose

result = compose("a melancholy rainy afternoon", out_dir="renders/rain", offline=True)
print(result.outputs["wav"], "-", result.score.title)

Outputs

Each run writes, into the output directory:

The two brains

Claude multi-agent (default when an API key is set). Each agent runs the canonical Messages-API tool-use loop with adaptive thinking; the model reasons, calls a tool, reads the observation, and continues until it finishes. Defaults to claude-opus-5-5. The API call degrades gracefully across SDK/account differences (server-side refusal fallback → plain streaming → drop effort → drop thinking), so the run never hard-fails on a capability mismatch.

export ANTHROPIC_API_KEY=sk-ant-...
ark-audio "a cathedral of light at the edge of the world" --effort high

Claude on Microsoft Foundry (Azure). Point the brain at a Foundry resource instead and it uses the SDK's AnthropicFoundry client with API-key auth. A Foundry endpoint takes precedence over ANTHROPIC_API_KEY:

export ANTHROPIC_FOUNDRY_API_KEY=...
export ANTHROPIC_FOUNDRY_RESOURCE=my-resource   # or ANTHROPIC_FOUNDRY_BASE_URL=https://my-resource.services.ai.azure.com/anthropic/
export ARK_MODEL=claude-opus-5-5                # your Foundry deployment name
ark-audio "a cathedral of light at the edge of the world"

Server-side refusal fallback is a first-party API feature, so it is skipped on Foundry. Adaptive thinking and effort still apply.

Deterministic (offline, always available). A rule-based planner maps the text to a feeling via a lexicon, then applies music-theory heuristics to make the same tool calls an LLM would. Reproducible given a --seed. This is what runs in CI and what powers --offline.

If no key is found (and --offline isn't set), the CLI automatically uses the deterministic brain and says so.

How the music is realized

• Theory (theory.py): scales/modes, diatonic chords, Roman-numeral resolution, scale-degree → MIDI, equal-temperament frequencies.
• Blackboard (blackboard.py): the declarative Score the agents edit.
• Performance (performance.py): realizes the Score into notes — a contour-driven melody that snaps to chord tones on strong beats, chord comping or sustained pads, several bass styles (roots, root-fifth, pulse, walking, arp), and synthesized drum patterns (four-on-floor, backbeat, breakbeat, …), all seeded and reproducible.
• Synth (synth.py): additive (band-limited) oscillator voices with ADSR, per-instrument patches, synthesized percussion, an FFT-convolution reverb, and a soft-clipping master bus.

Development

uv run pytest        # theory + pipeline tests (offline, no key needed)
uv run ark-audio "..." --offline

License

MIT.