ark-audio-agent

Autonomous text → music. Give it a sentence; it reasons about the feeling, composes a structured piece, and renders it to MIDI + audio — with no human in the loop.

The interesting part is the harness: composition is driven by a team of specialist agents running a ReAct loop (reason → act → observe) over a shared musical blackboard. It runs on the Claude API when a key is available, and on a fully-offline deterministic music-theory brain otherwise, so it always produces a result.

ark-audio "a rainy night in a neon city, lonely but hopeful"
# → renders/a-rainy-night-in-a-neon-city-lonely-but-hopeful/song.mp3  (+ .wav, .mid, report.md, trace.jsonl)

Add a direction to say how the piece must be built, in plain words. The text still sets the mood:

ark-audio "The Lord is my shepherd; I shall not want." \
  --direction "a few secs of intro and a bit of rising notes before the main melody, \
  structured, fewer long leaps, strongly leaning toward one chord and tone; after the \
  climax a falling melody, then a few sec of outro to end it"

Why it's built this way

Rather than ask one model to emit raw notes (which drifts out of key and is hard to steer), the system separates musical intent from musical realization:

• Agents decide intent — key, mode, tempo, song form, chord progressions, a melody's contour/register/density, instrument roles, drum and bass styles.
• Deterministic engines realize it — pure music-theory functions turn that intent into in-key notes, which a mido writer turns into MIDI and a from-scratch numpy synthesizer turns into audio.

Because every decision an agent can make is a tool, both brains (Claude and the offline planner) drive the exact same action surface and produce the exact same trace format.

Architecture

            text prompt
                │
     ┌──────────▼───────────────────────────────────────────────┐
     │  Orchestrator (the harness)                                │
     │                                                            │
     │   Interpreter → Composer → Melodist → Arranger             │  each agent runs a
     │                               │                            │  ReAct loop:
     │                               ▼                            │   reason (thinking)
     │                            Critic ──reject──┐              │   → act  (tool call)
     │                               │approve      │ one targeted │   → observe (result)
     │                               │        revision (bounded)  │
     └───────────────────────────────┼────────────┘              │
                                      │                           │
          ┌───────────────────────────▼───────────────────────┐  │
          │  Tools  (the ReAct action space)                   │  │
          │  set_brief · set_direction · get_theory ·          │  │
          │  set_global ·                                      │  │
          │  set_structure · set_progression · design_melody · │  │
          │  set_instruments · set_bass · set_drums ·          │  │
          │  analyze_score · finish / approve / request_revision│ │
          └───────────────────────────┬───────────────────────┘  │
                                       ▼                          │
                            Score blackboard (shared state)       │
     └──────────────────────────────┬─────────────────────────────┘
                                     ▼
          performance.py  →  { melody, harmony, bass, drums } notes
                                     │
                 ┌───────────────────┴────────────────────┐
                 ▼                                         ▼
        midi_render.py (mido)                   synth.py (numpy software synth)
                 │                                         │
              song.mid                        song.wav ──(ffmpeg)──► song.mp3

The agents

Every agent also has analyze_score (a sensing action) and a terminal finish.

Install

Uses uv. Runs on Windows, macOS and Linux, and CI tests all three.

uv sync --extra llm      # installs the optional anthropic SDK for the Claude brain
# or, without the Claude brain (offline only):
uv sync

ffmpeg is optional — if present, an MP3 is exported alongside the WAV (brew install ffmpeg, sudo apt install ffmpeg, or winget install Gyan.FFmpeg).

Usage

CLI

ark-audio "triumphant sunrise over the mountains"
ark-audio "a quiet dawn over still water" --offline --seed 7
ark-audio "lonely synthwave drive at 2am" --out renders/drive --no-mp3
echo "a storm breaking over the sea" | ark-audio      # reads stdin

Flags: --direction/-d (how to build it, see below), --direction-file, --text-file, --offline (force the deterministic brain), --seed, --out, --model, --effort low|medium|high|xhigh|max, --max-revisions, --reverb, --no-mp3.

Library

from ark_audio_agent import compose

result = compose("a melancholy rainy afternoon", out_dir="renders/rain", offline=True)
print(result.outputs["wav"], "-", result.score.title)

# text for the mood, direction for the form and constraints
result = compose("For God so loved the world", out_dir="renders/bgm",
                 direction="short intro, rising notes into the verse, a climax, "
                           "then a falling melody and a few seconds of outro")
for check in result.direction_checks:
    print("✓" if check["ok"] else "✗", check["requirement"], "-", check["detail"])

Musical direction

Two inputs, with separate jobs:

• The text (the positional prompt) sets the mood: key, mode, tempo, genre, instruments.
• The direction (--direction) sets how the piece is built. The engine treats it as requirements and lets the agents be creative inside them.

A rule-based reader turns the direction into a plan the agents can see, so it works offline as well. With the Claude brain, the Interpreter reviews that plan against your words and corrects it with set_direction. It understands:

Durations given in seconds are converted to whole bars at the chosen tempo.

How the engine enforces it. The melody engine guarantees the hard limits:

• No leap is wider than the limit, and that includes the step from one section into the next.
• Rises and falls move one way.
• The climax reaches above everything before it, and nothing after it goes higher.
• Cadences resolve to the tonic.

Then analyze_score measures the realized piece against each requirement. The Critic sends any unmet one back to the agent responsible, and the CLI and report.md show the result:

direction: 11/11 requirements met
  ✓ form follows the plan        sections: intro -> rise -> verse -> climax -> fall -> outro
  ✓ intro lasts ~4s              3.3s (1 bars at 73 BPM)
  ✓ rise rises                   G4 -> D5, trend +5.0 semitones
  ✓ climax holds the highest note
  ✓ fall falls                   C6 -> C5, trend -10.5 semitones
  ✓ no leap larger than 3 scale steps
  ✓ harmony leans on the tonic chord   tonic on 58% of bars (need 45%)
  ✓ melody ends on the tonic     last note C5
  ...

Outputs

Each run writes, into the output directory:

The two brains

Claude multi-agent (default when an API key is set). Each agent runs the canonical Messages-API tool-use loop with adaptive thinking; the model reasons, calls a tool, reads the observation, and continues until it finishes. Defaults to claude-opus-5-5. The API call degrades gracefully across SDK/account differences (server-side refusal fallback → plain streaming → drop effort → drop thinking), so the run never hard-fails on a capability mismatch.

export ANTHROPIC_API_KEY=sk-ant-...
ark-audio "a cathedral of light at the edge of the world" --effort high

Claude on Microsoft Foundry (Azure). Point the brain at a Foundry resource instead and it uses the SDK's AnthropicFoundry client with API-key auth. A Foundry endpoint takes precedence over ANTHROPIC_API_KEY:

export ANTHROPIC_FOUNDRY_API_KEY=...
export ANTHROPIC_FOUNDRY_RESOURCE=my-resource   # or ANTHROPIC_FOUNDRY_BASE_URL=https://my-resource.services.ai.azure.com/anthropic/
export ARK_MODEL=claude-opus-5-5                # your Foundry deployment name
ark-audio "a cathedral of light at the edge of the world"

Server-side refusal fallback is a first-party API feature, so it is skipped on Foundry. Adaptive thinking and effort still apply.

Deterministic (offline, always available). A rule-based planner maps the text to a feeling via a lexicon, then applies music-theory heuristics to make the same tool calls an LLM would. Reproducible given a --seed. This is what runs in CI and what powers --offline.

If no key is found (and --offline isn't set), the CLI automatically uses the deterministic brain and says so.

How the music is realized

• Theory (theory.py): scales/modes, diatonic chords, Roman-numeral resolution, scale-degree → MIDI, equal-temperament frequencies.

• Blackboard (blackboard.py): the declarative Score the agents edit.

• Direction (direction.py): reads a direction prompt into a section plan and constraints, and checks a realized piece against it.

• Performance (performance.py): realizes the Score into notes:

  • a contour-driven melody that snaps to chord tones on strong beats, or a phrase-structured melody when a direction (or the Melodist) asks for one. That melody is built from motifs, keeps leaps within a limit, leans on the tonic, flows across sections and ends on a cadence;
  • intensity ramps within a section;
  • chord comping or sustained pads;
  • several bass styles (roots, root-fifth, pulse, walking, arp, tonic pedal);
  • synthesized drum patterns (four-on-floor, backbeat, breakbeat, a riser, …).

  All of it is seeded and reproducible.

• Synth (synth.py): additive (band-limited) oscillator voices with ADSR, per-instrument patches, synthesized percussion, an FFT-convolution reverb, and a soft-clipping master bus.

Development

First-time setup (virtual environment)

You need Python 3.10+ (the repo pins 3.12 in .python-version). On macOS the system python3 is 3.9, which is too old, so use the Python that uv installs or one from Homebrew.

Option A: uv (recommended)

git clone https://github.com/ir-dev/ark-audio-agent.git
cd ark-audio-agent
uv sync --extra llm          # creates .venv: the package (editable), anthropic, pytest
source .venv/bin/activate    # Windows: .venv\Scripts\activate

If Python 3.12 is missing, uv downloads it. uv sync also installs the dev group (pytest) by default.

Option B: plain venv + pip

git clone https://github.com/ir-dev/ark-audio-agent.git
cd ark-audio-agent
python3.12 -m venv .venv     # any Python 3.10+
source .venv/bin/activate    # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip
pip install -e ".[llm]" pytest

-e is an editable install, so changes under src/ take effect without reinstalling.

Check it works

pytest -q                                                 # theory + pipeline tests (offline, no key needed)
ark-audio "a quiet dawn over still water" --offline --seed 7

With the venv activated, ark-audio and pytest run directly. Without activating it, prefix them with uv run.

API key (optional)

A key is only needed for the Claude brain. Copy the template and fill in one provider:

cp .env.example .env         # .env is git-ignored; never commit it

The CLI does not read .env by itself, so load it when you run:

uv run --env-file .env ark-audio "a storm breaking over the sea"     # any OS
# or export it into the current shell (bash/zsh; works with either option):
set -a; source .env; set +a
ark-audio "a storm breaking over the sea"

License

MIT.