ark-audio-agent

Autonomous text → music. Give it a sentence; it reasons about the feeling, composes a structured piece, and renders it to MIDI + audio — with no human in the loop.

The interesting part is the harness: composition is driven by a team of specialist agents running a ReAct loop (reason → act → observe) over a shared musical blackboard. It runs on the Claude API when a key is available, and on a fully-offline deterministic music-theory brain otherwise, so it always produces a result.

ark-audio "a rainy night in a neon city, lonely but hopeful"
# → renders/20261011-132450_a-rainy-night-in-a-neon-city-lonely-but/song.mp3  (+ .wav, .mid, report.md, trace.jsonl)

Add a direction to say how the piece must be built, in plain words. The text still sets the mood:

ark-audio "The Lord is my shepherd; I shall not want." \
  --direction "a few secs of intro and a bit of rising notes before the main melody, \
  structured, fewer long leaps, strongly leaning toward one chord and tone; after the \
  climax a falling melody, then a few sec of outro to end it"

Name reference music to draw on: a piece from the built-in library, a style, or your own MIDI file. The piece borrows its traits (chords, melodic shape, palette, groove), not its notes:

ark-audio "a rainy afternoon in an empty flat" -r satie --borrow harmony,melody,tempo
ark-audio "a summer night drive" -r synthwave-night-drive
ark-audio "develop my sketch" -r ./idea.mid --strength 0.8

Building an agent or app on top of this? docs/AI_SPEC.md is the full contract: every parameter, nuance and conflict. docs/PROMPTS.md has a ready-made system prompt and Claude, OpenAI-compatible, MCP and CLI wiring. ark_audio_agent.spec ships the tool definitions and a run_tool dispatcher.

Why it's built this way

Rather than ask one model to emit raw notes (which drifts out of key and is hard to steer), the system separates musical intent from musical realization:

• Agents decide intent — key, mode, tempo, song form, chord progressions, a melody's contour/register/density, instrument roles, drum and bass styles.
• Deterministic engines realize it — pure music-theory functions turn that intent into in-key notes, which a mido writer turns into MIDI and a from-scratch numpy synthesizer turns into audio.

Because every decision an agent can make is a tool, both brains (Claude and the offline planner) drive the exact same action surface and produce the exact same trace format.

Architecture

            text prompt
                │
     ┌──────────▼───────────────────────────────────────────────┐
     │  Orchestrator (the harness)                                │
     │                                                            │
     │   Interpreter → Composer → Melodist → Arranger             │  each agent runs a
     │                               │                            │  ReAct loop:
     │                               ▼                            │   reason (thinking)
     │                            Critic ──reject──┐              │   → act  (tool call)
     │                               │approve      │ one targeted │   → observe (result)
     │                               │        revision (bounded)  │
     └───────────────────────────────┼────────────┘              │
                                      │                           │
          ┌───────────────────────────▼───────────────────────┐  │
          │  Tools  (the ReAct action space)                   │  │
          │  set_brief · set_direction · get_theory ·          │  │
          │  find_references · choose_references · set_global ·│  │
          │  set_structure · set_progression · design_melody · │  │
          │  set_instruments · set_bass · set_drums ·          │  │
          │  analyze_score · finish / approve / request_revision│ │
          └───────────────────────────┬───────────────────────┘  │
                                       ▼                          │
                            Score blackboard (shared state)       │
     └──────────────────────────────┬─────────────────────────────┘
                                     ▼
          performance.py  →  { melody, harmony, bass, drums } notes
                                     │
                 ┌───────────────────┴────────────────────┐
                 ▼                                         ▼
        midi_render.py (mido)                   synth.py (numpy software synth)
                 │                                         │
              song.mid                        song.wav ──(ffmpeg)──► song.mp3

The agents

Every agent also has analyze_score (a sensing action) and a terminal finish.

Install

Uses uv. Runs on Windows, macOS and Linux, and CI tests all three.

pip install "ark-audio-agent[llm]"   # from PyPI, with the anthropic SDK for the Claude brain
pip install ark-audio-agent          # or offline only

From a checkout, uv sync installs everything, the anthropic SDK included (see Development).

ffmpeg is optional — if present, an MP3 is exported alongside the WAV (brew install ffmpeg, sudo apt install ffmpeg, or winget install Gyan.FFmpeg).

Usage

Run with Claude in two steps

cp .env.example .env     # then fill in ONE provider: ANTHROPIC_API_KEY, or the ANTHROPIC_FOUNDRY_* pair
uv run ark-audio "triumphant sunrise over the mountains"

The CLI loads ./.env on its own, so there is nothing to export or source. Check the start-up panel: it should read brain: claude multi-agent (...) and env: .env. If it says deterministic (offline — ...), the text after the dash names what's missing. Use --env-file path/to/file to load a different file; a variable already set in your shell overrides the file.

CLI

ark-audio "triumphant sunrise over the mountains"
ark-audio "a quiet dawn over still water" --offline --seed 7
ark-audio "lonely synthwave drive at 2am" --out renders/drive --no-mp3
echo "a storm breaking over the sea" | ark-audio      # reads stdin

Flags: --direction/-d (how to build it, see below), --direction-file, --reference/-r (music to draw on, repeatable), --borrow, --strength, --no-reference, --list-references, --text-file, --offline (force the deterministic brain), --env-file (default ./.env), --seed, --out, --model, --effort low|medium|high|xhigh|max, --max-revisions, --reverb, --no-mp3, --spec (print the JSON tool spec for AI agents). Every flag, its default and how it interacts with the others is in docs/AI_SPEC.md.

Library

from ark_audio_agent import compose

result = compose("a melancholy rainy afternoon", out_dir="renders/rain", offline=True)
print(result.outputs["wav"], "-", result.score.title)

# text for the mood, direction for the form and constraints
result = compose("For God so loved the world", out_dir="renders/bgm",
                 direction="short intro, rising notes into the verse, a climax, "
                           "then a falling melody and a few seconds of outro")
for check in result.direction_checks:
    print("✓" if check["ok"] else "✗", check["requirement"], "-", check["detail"])

# reference music: a library id/title or a .mid path; borrow picks what to take
result = compose("a rainy afternoon", references=["satie-gymnopedie-1"],
                 borrow=["harmony", "melody"], strength=0.6, offline=True)
print(result.inspiration_checks)   # how closely each borrowed aspect was followed

The library doesn't read .env (the CLI does). Load it with ark_audio_agent.envfile.load_env_file(".env"), and check result.brain_kind to see which brain ran.

Musical direction

Two inputs, with separate jobs:

• The text (the positional prompt) sets the mood: key, mode, tempo, genre, instruments.
• The direction (--direction) sets how the piece is built. The engine treats it as requirements and lets the agents be creative inside them.

A rule-based reader turns the direction into a plan the agents can see, so it works offline as well. With the Claude brain, the Interpreter reviews that plan against your words and corrects it with set_direction. It understands:

Durations given in seconds are converted to whole bars at the chosen tempo.

How the engine enforces it. The melody engine guarantees the hard limits:

• No leap is wider than the limit, and that includes the step from one section into the next.
• Rises and falls move one way.
• The climax reaches above everything before it, and nothing after it goes higher.
• Cadences resolve to the tonic.

Then analyze_score measures the realized piece against each requirement. The Critic sends any unmet one back to the agent responsible, and the CLI and report.md show the result:

direction: 11/11 requirements met
  ✓ form follows the plan        sections: intro -> rise -> verse -> climax -> fall -> outro
  ✓ intro lasts ~4s              3.3s (1 bars at 73 BPM)
  ✓ rise rises                   G4 -> D5, trend +5.0 semitones
  ✓ climax holds the highest note
  ✓ fall falls                   C6 -> C5, trend -10.5 semitones
  ✓ no leap larger than 3 scale steps
  ✓ harmony leans on the tonic chord   tonic on 58% of bars (need 45%)
  ✓ melody ends on the tonic     last note C5
  ...

Reference music

A third input, with its own job: music to draw on. A reference is a set of traits, not notes:

• a tonality, a tempo and meter, and a form;
• chord progressions as Roman numerals, so they move to any key;
• melodic traits plus the interval shape of a motif;
• an instrument palette and a groove.

The library ships 23 references:

• Public-domain works: Pachelbel's Canon, Satie's Gymnopédie No. 1, Bach's Prelude in C and Toccata in D minor, Beethoven's Moonlight Sonata, Ode to Joy and Für Elise, Debussy's Clair de lune, Grieg's Mountain King and Morning Mood, Amazing Grace, Greensleeves, Scarborough Fair and the Dies Irae.
• Genre archetypes, which describe a style rather than any one song: pop anthem, 12-bar blues, synthwave, lo-fi, epic trailer, ambient drone, flamenco, celtic reel and worship ballad.

ark-audio --list-references            # the whole library
ark-audio --list-references rainy piano   # ranked for those words

Choose what to take with --borrow, and how closely with --strength (0..1):

By default a reference you name lends form, harmony, melody, instruments and groove, but not tonality or tempo, because the text sets the mood. What you set is locked against the agents.

Without -r, the Interpreter may pick a reference itself when one clearly fits the text. Offline it pins the best match at strength 0.5; --no-reference turns this off.

Point -r at a .mid file and it is read into the same traits: key, tempo, meter, one chord per bar, melodic traits and a motif, palette, bass and drum styles. The reader only knows major and minor, so a modal piece comes back as the major or minor key with the same notes.

The precedence is direction > text > references. A direction's sections replace a reference's form, and a tonal anchor keeps the harmony tonic-heavy. analyze_score and report.md show an echo per borrowed aspect: how closely the piece follows its reference, from 0 to 1. It is a measure, not a target.

Outputs

Each run gets its own folder, renders/<YYYYMMDD-HHMMSS>_<slug of the text>, so runs sort by time and never overwrite each other (--out or out_dir= picks a folder yourself). It contains:

The two brains

Claude multi-agent (default when an API key is set). Each agent runs the canonical Messages-API tool-use loop with adaptive thinking; the model reasons, calls a tool, reads the observation, and continues until it finishes. Defaults to claude-opus-5-5. The API call degrades gracefully across SDK/account differences (server-side refusal fallback → plain streaming → drop effort → drop thinking), so the run never hard-fails on a capability mismatch.

# .env
ANTHROPIC_API_KEY=sk-ant-...

ark-audio "a cathedral of light at the edge of the world" --effort high

Claude on Microsoft Foundry (Azure). Point the brain at a Foundry resource instead and it uses the SDK's AnthropicFoundry client with API-key auth. A Foundry endpoint takes precedence over ANTHROPIC_API_KEY:

# .env
ANTHROPIC_FOUNDRY_API_KEY=...
ANTHROPIC_FOUNDRY_RESOURCE=my-resource   # or ANTHROPIC_FOUNDRY_BASE_URL=https://my-resource.services.ai.azure.com/anthropic/
ARK_MODEL=claude-opus-5-5                # your Foundry deployment name

If both ANTHROPIC_FOUNDRY_RESOURCE and ANTHROPIC_FOUNDRY_BASE_URL are set, the base URL is used.

Server-side refusal fallback is a first-party API feature, so it is skipped on Foundry. Adaptive thinking and effort still apply.

Deterministic (offline, always available). A rule-based planner maps the text to a feeling via a lexicon, then applies music-theory heuristics to make the same tool calls an LLM would. Reproducible given a --seed. This is what runs in CI and what powers --offline.

If no key is found (and --offline isn't set), the CLI automatically uses the deterministic brain and says so.

How the music is realized

• Theory (theory.py): scales/modes, diatonic chords, Roman-numeral resolution, scale-degree → MIDI, equal-temperament frequencies.

• Blackboard (blackboard.py): the declarative Score the agents edit.

• Direction (direction.py): reads a direction prompt into a section plan and constraints, and checks a realized piece against it.

• References (references.py, midi_analysis.py): the reference library, its search and name resolution, the measure of how closely a piece echoes its references, and the reader that turns a MIDI file into reference traits.

• Spec (spec.py): the library as tools for another model, with the definitions, a run_tool dispatcher, the recommended system prompt and the full machine-readable contract.

• Performance (performance.py): realizes the Score into notes:

  • a contour-driven melody that snaps to chord tones on strong beats, or a phrase-structured melody when a direction (or the Melodist) asks for one. That melody is built from motifs, keeps leaps within a limit, leans on the tonic, flows across sections and ends on a cadence;
  • intensity ramps within a section;
  • chord comping or sustained pads;
  • several bass styles (roots, root-fifth, pulse, walking, arp, tonic pedal);
  • synthesized drum patterns (four-on-floor, backbeat, breakbeat, a riser, …).

  All of it is seeded and reproducible.

• Synth (synth.py): additive (band-limited) oscillator voices with ADSR, per-instrument patches, synthesized percussion, an FFT-convolution reverb, and a soft-clipping master bus.

Development

First-time setup (virtual environment)

You need Python 3.10+ (the repo pins 3.12 in .python-version). On macOS the system python3 is 3.9, which is too old, so use the Python that uv installs or one from Homebrew.

Option A: uv (recommended)

git clone https://github.com/ir-dev/ark-audio-agent.git
cd ark-audio-agent
uv sync                      # creates .venv: the package (editable), anthropic, pytest
source .venv/bin/activate    # Windows: .venv\Scripts\activate

If Python 3.12 is missing, uv downloads it. uv sync also installs the dev group (pytest and the anthropic SDK) by default, so a later uv sync never drops the Claude brain.

Option B: plain venv + pip

git clone https://github.com/ir-dev/ark-audio-agent.git
cd ark-audio-agent
python3.12 -m venv .venv     # any Python 3.10+
source .venv/bin/activate    # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip
pip install -e ".[llm]" pytest

-e is an editable install, so changes under src/ take effect without reinstalling.

Check it works

pytest -q                                                 # theory + pipeline tests (offline, no key needed)
ark-audio "a quiet dawn over still water" --offline --seed 7

With the venv activated, ark-audio and pytest run directly. Without activating it, prefix them with uv run.

API key (optional)

A key is only needed for the Claude brain. Copy the template and fill in one provider:

cp .env.example .env         # .env is git-ignored; never commit it

The CLI reads ./.env itself, on every OS, so just run it from the repo root:

uv run ark-audio "a storm breaking over the sea"

License

MIT.