ark-audio-agent

Autonomous text → music. Give it a sentence; it reasons about the feeling, composes a structured piece, and renders it to MIDI + audio — with no human in the loop.

The interesting part is the harness: composition is driven by a team of specialist agents running a ReAct loop (reason → act → observe) over a shared musical blackboard. It runs on the Claude API when a key is available, and on a fully-offline deterministic music-theory brain otherwise, so it always produces a result.

ark-audio "a rainy night in a neon city, lonely but hopeful"
# → renders/20261011-132450_a-rainy-night-in-a-neon-city-lonely-but/song.mp3  (+ .wav, .mid, report.md, trace.jsonl)

Add a direction to say how the piece must be built, in plain words. The text still sets the mood:

ark-audio "The Lord is my shepherd; I shall not want." \
  --direction "a few secs of intro and a bit of rising notes before the main melody, \
  structured, fewer long leaps, strongly leaning toward one chord and tone; after the \
  climax a falling melody, then a few sec of outro to end it"

Name reference music to draw on: a piece from the built-in library, a style, or your own MIDI file. The piece borrows its traits (chords, melodic shape, palette, groove), not its notes:

ark-audio "a rainy afternoon in an empty flat" -r satie --borrow harmony,melody,tempo
ark-audio "lamps lit at dusk by the river" -r raga-yaman --borrow all
ark-audio "develop my sketch" -r ./idea.mid --strength 0.8

Building an agent or app on top of this? Start with the prompting guide below. docs/AI_SPEC.md is the full contract: every parameter, nuance and conflict. docs/PROMPTS.md has the system prompt and Claude, OpenAI-compatible, MCP and CLI wiring. ark_audio_agent.spec ships the tool definitions and a run_tool dispatcher.

Contents: What the AI does · Quick start · CLI cookbook · Prompting guide for developers · Python library · Musical direction · Reference music · Outputs · How it works · Development

What the AI does

With an Anthropic API key (or Claude on Microsoft Foundry), five Claude agents compose the piece. Each one reasons with adaptive thinking and acts only through tools on a shared score, so every musical decision is a tool call you can inspect afterwards.

Claude or offline: what changes

Quick start

Install from PyPI. The [llm] extra adds the anthropic SDK for the Claude brain:

pip install "ark-audio-agent[llm]"
ark-audio "a quiet dawn over still water" --offline     # no key needed: ~2 s, reproducible

Then turn on Claude. Put one provider in a .env file in the folder you run from; the CLI loads it by itself, so there is nothing to export:

# .env  (from a checkout: cp .env.example .env)
ANTHROPIC_API_KEY=sk-ant-...

ark-audio "triumphant sunrise over the mountains"

Check the start-up panel: it should read brain: claude multi-agent (...) and env: .env. If it says deterministic (offline — ...), the text after the dash names what's missing. A variable already set in your shell overrides the file.

• Offline only: pip install ark-audio-agent.
• MP3 export: install ffmpeg (brew install ffmpeg, sudo apt install ffmpeg, or winget install Gyan.FFmpeg). Without it you still get the WAV and MIDI.
• From a checkout: uv sync installs everything, the anthropic SDK included (see Development). Prefix commands with uv run unless the venv is active.

CLI cookbook

Every command below was checked against this release (offline). The ones that name a file (psalm23.txt, brief.txt, ./idea.mid, an env file) need that file to exist. Add --offline to any of them for a fast, reproducible run without a key.

First runs

ark-audio "a quiet dawn over still water"                     # Claude if a key is set, else offline
ark-audio "a quiet dawn over still water" --offline --seed 7  # reproducible: same seed, same piece
ark-audio "a quiet dawn over still water" --offline --seed 8  # another take of the same idea
echo "a storm breaking over the sea" | ark-audio --offline    # text from stdin
ark-audio --text-file psalm23.txt                             # text from a file

Shape it with a direction

# a song form with a build and a release
ark-audio "For God so loved the world" \
  -d "a 6 second intro, rising notes into the verse, a climax, then a falling melody and a short outro"

# background music to a length
ark-audio "a calm, still meditation; slow breathing, soft light" \
  -d "background music for a meditation video: a short intro, a main part, then a long outro. 60 seconds long overall, slow"

# a pop structure at a set tempo
ark-audio "a summer night, friends, windows down" -d "120 bpm, a verse, a big chorus, a bridge, then a short outro"

# a stated key, a singable line and phrase structure
ark-audio "a lullaby for a sleeping child" -d "in F major, stepwise, structured 4-bar phrases, a long outro"

# a dark piece with an arch
ark-audio "a lonely lighthouse in winter" \
  -d "in D minor, 80 bpm, a short intro, a verse, a climax, then a falling melody and a long outro"

# hypnotic: grounded on one chord, no intro
ark-audio "a hypnotic desert night" -d "no intro, a main part, strongly leaning toward one chord and tone, a short outro, 70 bpm"

# podcast bed
ark-audio "a podcast about the ocean" -d "bgm for a podcast, a 6 second intro, a verse, a short outro"

# a long direction kept in a file
ark-audio "For God so loved the world" --direction-file brief.txt

The CLI prints each requirement with ✓ or ✗ when the piece is done, and report.md keeps the same table.

Draw on reference music

ark-audio --list-references                   # the whole library: id, key, tempo, meter, character
ark-audio --list-references rainy piano       # ranked for those words

# the same chords, your own tune
ark-audio "a sad farewell at a train station in the rain" -r pachelbel-canon --borrow harmony

# a waltz: borrowing tempo brings the 3/4 meter with it
ark-audio "wistful, a memory of an old dance hall" -r greensleeves --borrow tempo,harmony,melody,instruments

# two references, followed loosely
ark-audio "a quiet evening walk" -r satie -r debussy-clair-de-lune --strength 0.4

# everything from one reference, key and tempo included
ark-audio "victory at dawn" -r epic-trailer-build --borrow all

# develop your own MIDI sketch and keep its key
ark-audio "develop my sketch" -r ./idea.mid --borrow tonality,harmony,melody,tempo --strength 0.8

# purely original: stop the agents picking a reference themselves
ark-audio "a quiet dawn over still water" --no-reference

Indian styles

A raga is its scale, so borrow tonality too (or all). Otherwise the text's mood picks the mode and only the phrases, palette and groove carry over.

ark-audio "lamps lit at dusk by the river" -r raga-yaman --borrow tonality,melody,instruments
ark-audio "a farewell at dawn" -r raga-bhairavi --borrow tonality,tempo,melody,instruments   # 6/8 dadra
ark-audio "a temple festival morning" -r carnatic-kriti-mohanam --borrow all
ark-audio "a monsoon love song" -r bollywood-romantic-ballad
ark-audio "a punjabi wedding baraat" -r bhangra-dhol --borrow all
ark-audio "a night of devotion at the dargah" -r sufi-qawwali -d "a short intro, build up, a climax, then a short outro"

Tune the Claude brain

ark-audio "a cathedral of light at the edge of the world" --effort high      # deeper reasoning
ark-audio "a cathedral of light at the edge of the world" --effort low --max-revisions 0   # fastest, cheapest
ark-audio "a cathedral of light at the edge of the world" --model claude-sonnet-5-5
ark-audio "a cathedral of light at the edge of the world" --env-file ~/.config/ark-audio/foundry.env

ARK_MODEL, ARK_EFFORT and ARK_MAX_TOKENS in .env set the same defaults. On Foundry, --model is your deployment name.

Output and sound

ark-audio "lonely synthwave drive at 2am" --out renders/drive --no-mp3   # pick the folder, skip ffmpeg
ark-audio "a storm breaking over the sea" --reverb 0.35                  # wetter (default 0.16)
ark-audio "a storm breaking over the sea" --reverb 0                     # dry

Scripts and other AI tools

ark-audio --spec > ark-audio.spec.json        # tools, enums, precedence, conflicts, system prompt
ark-audio --spec | jq '.tools[].name'         # "compose_music", "list_reference_music"

# three takes of one idea, side by side
for seed in 1 2 3; do
  ark-audio "a quiet dawn over still water" --offline --seed $seed --out renders/takes/$seed
done

Exit codes: 0 success · 2 bad input (no text, unknown reference, bad --borrow, strength out of range, more than 3 references, unreadable env file) · 130 interrupted.

Every flag

Defaults and interactions for every flag are in docs/AI_SPEC.md.

Prompting guide for developers

The same three inputs do the work whether a person types the command or your model builds the call. Putting each request into the right input matters more than the wording.

The three inputs

When they disagree: direction > text > references. A reference never changes the key, mode or tempo unless you borrow tonality or tempo, and a key or tempo stated in the direction beats even that.

Writing the text

Pass the user's words as they are. Concrete imagery ("rain on a tin roof at 2am") gives Claude more to work with than adjectives alone. Offline, the brain matches words against ten feelings, so it helps to know which words it reacts to:

A text with none of these words still composes. Offline it gets a neutral reading and no automatic reference.

Writing a direction

• Name the sections you want: "a short intro", "rising notes", "a verse", "a chorus", "a bridge", "a climax", "a falling melody", "a long outro". They always play in this order: intro, rise, main, chorus, bridge, climax, fall, outro.
• Put a length next to its section: "a 6 second intro", "an outro of 2 bars". Seconds are rounded to whole bars at the chosen tempo.
• State a key as in + a capital letter + a mode: "in D minor" works; "in d minor" does nothing.
• Shape the melody: "stepwise" (no leap over a third), "fewer long leaps" (a fourth), "structured, 4-bar phrases" (motifs repeated A A′ B A′).
• Ground it: "strongly leaning toward one chord and tone" gives tonic-heavy harmony, a pedal bass and a melody drawn home.
• Say what it's for: "background music for a podcast" gives a lighter arrangement.
• Name at least one section when you give a total length or a tonal anchor. Offline, both are fitted to the planned sections. With none named, the default form is used and that check can fail.
• Don't ask for a time signature (borrow tempo from a 3/4 or 6/8 reference instead), vocals or lyrics. Avoid the word "phrases" unless you want phrase structure.

The full vocabulary is under Musical direction, and docs/PROMPTS.md §4 has a phrasebook.

Choosing references and what to borrow

Search first (ark-audio --list-references WORDS or list_references("WORDS")), then pass an id. Map what the user says onto borrow and strength:

Pass borrow or strength only when the user says so: passing either locks it against the agents. For a modern copyrighted song, use the nearest archetype and tell the user it's modelled on the style, not the song.

Requests turned into calls

These are compose_music arguments. On the CLI they become the positional text, -d, -r, --borrow and --strength.

More worked examples, with the reasoning behind each, are in docs/PROMPTS.md §3.

Put it behind your own model

ark_audio_agent.spec gives you all three pieces: the system prompt (system_prompt(), reproduced in docs/PROMPTS.md), the tools (anthropic_tools(), openai_tools(), or ark-audio --spec for any other stack) and the dispatcher (run_tool(name, args): JSON in, JSON out, errors returned as {"ok": false, "error": ...}). A complete Claude loop:

import json

import anthropic

from ark_audio_agent.spec import anthropic_tools, run_tool, system_prompt

client = anthropic.Anthropic()          # ANTHROPIC_API_KEY, or an `ant auth login` profile
messages = [{"role": "user", "content": "Gentle background music for my morning-prayer "
                                        "podcast, about 45 seconds, a bit like Amazing Grace"}]
while True:
    response = client.beta.messages.create(
        model="claude-opus-5-5",
        max_tokens=16000,
        system=system_prompt(),
        tools=anthropic_tools(),
        output_config={"effort": "medium"},
        betas=["server-side-fallback-2026-07-01"],   # re-run a refused request on a
        fallbacks="default",                           # fallback model (Claude API only)
        messages=messages,
    )
    if response.stop_reason == "refusal":
        raise RuntimeError(f"declined: {response.stop_details}")
    messages.append({"role": "assistant", "content": response.content})   # unchanged
    if response.stop_reason != "tool_use":
        break
    results = []
    for block in response.content:
        if block.type == "tool_use":
            out = run_tool(block.name, {**block.input, "offline": True})   # fast and free
            results.append({"type": "tool_result", "tool_use_id": block.id,
                            "content": json.dumps(out), "is_error": not out.get("ok", False)})
    messages.append({"role": "user", "content": results})   # all results in one message

print(next(b.text for b in response.content if b.type == "text"))

Things that matter in production:

• Who composes. run_tool uses the brain of the machine it runs on. With a key present, each music request would start its own multi-agent Claude run (minutes, real tokens), so the loop above forces offline. Remove that to let Claude compose too.
• Don't force the tool. tool_choice any or tool returns a 400 on Claude Opus 5.5. The system prompt is what steers the model to call compose_music.
• Keep the history unchanged. Append response.content as it came back, so the thinking blocks stay valid.
• Other platforms. On Microsoft Foundry, Bedrock or Vertex, use that platform's client and client.messages.create, without betas and fallbacks.
• Async servers. run_tool blocks while it renders, so call await asyncio.to_thread(run_tool, name, args).
• Tell the user what happened. The result carries the title, key, tempo, form, file paths, direction.failed and the references used.
• More wiring. OpenAI-compatible function calling, an MCP server and the any-language CLI pattern are in docs/PROMPTS.md §2, with a production checklist in §5.

Python library

from ark_audio_agent import compose

result = compose("a melancholy rainy afternoon", out_dir="renders/rain", offline=True)
print(result.outputs["wav"], "-", result.score.title)

# text for the mood, direction for the form and constraints
result = compose("For God so loved the world", out_dir="renders/bgm",
                 direction="short intro, rising notes into the verse, a climax, "
                           "then a falling melody and a few seconds of outro")
for check in result.direction_checks:
    print("✓" if check["ok"] else "✗", check["requirement"], "-", check["detail"])

# reference music: a library id/title or a .mid path; borrow picks what to take
result = compose("a rainy afternoon", references=["satie-gymnopedie-1"],
                 borrow=["harmony", "melody"], strength=0.6, offline=True)
print(result.inspiration_checks)   # how closely each borrowed aspect was followed

The library doesn't read .env (the CLI does). Load it with ark_audio_agent.envfile.load_env_file(".env"), and check result.brain_kind to see which brain ran. list_references("rainy piano") searches the library from Python.

Musical direction

Two inputs, with separate jobs:

• The text (the positional prompt) sets the mood: key, mode, tempo, genre, instruments.
• The direction (--direction) sets how the piece is built. The engine treats it as requirements and lets the agents be creative inside them.

A rule-based reader turns the direction into a plan the agents can see, so it works offline as well. With the Claude brain, the Interpreter reviews that plan against your words and corrects it with set_direction. It understands:

Durations given in seconds are converted to whole bars at the chosen tempo.

How the engine enforces it. The melody engine guarantees the hard limits:

• No leap is wider than the limit, and that includes the step from one section into the next.
• Rises and falls move one way.
• The climax reaches above everything before it, and nothing after it goes higher.
• Cadences resolve to the tonic.

Then analyze_score measures the realized piece against each requirement. The Critic sends any unmet one back to the agent responsible, and the CLI and report.md show the result:

direction: 11/11 requirements met
  ✓ form follows the plan        sections: intro -> rise -> verse -> climax -> fall -> outro
  ✓ intro lasts ~4s              3.3s (1 bars at 73 BPM)
  ✓ rise rises                   G4 -> D5, trend +5.0 semitones
  ✓ climax holds the highest note
  ✓ fall falls                   C6 -> C5, trend -10.5 semitones
  ✓ no leap larger than 3 scale steps
  ✓ harmony leans on the tonic chord   tonic on 58% of bars (need 45%)
  ✓ melody ends on the tonic     last note C5
  ...

Reference music

A third input, with its own job: music to draw on. A reference is a set of traits, not notes:

• a tonality, a tempo and meter, and a form;
• chord progressions as Roman numerals, so they move to any key;
• melodic traits plus the interval shape of a motif;
• an instrument palette and a groove.

The library ships 29 references:

• Public-domain works: Pachelbel's Canon, Satie's Gymnopédie No. 1, Bach's Prelude in C and Toccata in D minor, Beethoven's Moonlight Sonata, Ode to Joy and Für Elise, Debussy's Clair de lune, Grieg's Mountain King and Morning Mood, Amazing Grace, Greensleeves, Scarborough Fair and the Dies Irae.
• Genre archetypes, which describe a style rather than any one song: pop anthem, 12-bar blues, synthwave, lo-fi, epic trailer, ambient drone, flamenco, celtic reel and worship ballad.
• Indian styles, also archetypes: Hindustani ragas Yaman (evening) and Bhairavi (morning), each with its pakad as the motif, a Carnatic kriti in raga Mohanam, a Bollywood romantic ballad, bhangra and Sufi qawwali. Their palettes use the sitar, shehnai and harmonium instruments.

ark-audio --list-references            # the whole library
ark-audio --list-references rainy piano   # ranked for those words

Choose what to take with --borrow, and how closely with --strength (0..1):

By default a reference you name lends form, harmony, melody, instruments and groove, but not tonality or tempo, because the text sets the mood. What you set is locked against the agents.

Without -r, the Interpreter may pick a reference itself when one clearly fits the text. Offline it pins the best match at strength 0.5; --no-reference turns this off.

Point -r at a .mid file and it is read into the same traits: key, tempo, meter, one chord per bar, melodic traits and a motif, palette, bass and drum styles. The reader only knows major and minor, so a modal piece comes back as the major or minor key with the same notes.

The precedence is direction > text > references. A direction's sections replace a reference's form, and a tonal anchor keeps the harmony tonic-heavy. analyze_score and report.md show an echo per borrowed aspect: how closely the piece follows its reference, from 0 to 1. It is a measure, not a target.

Outputs

Each run gets its own folder, renders/<YYYYMMDD-HHMMSS>_<slug of the text>, so runs sort by time and never overwrite each other (--out or out_dir= picks a folder yourself). It contains:

How it works

Why it's built this way

Rather than ask one model to emit raw notes (which drifts out of key and is hard to steer), the system separates musical intent from musical realization:

• Agents decide intent — key, mode, tempo, song form, chord progressions, a melody's contour/register/density, instrument roles, drum and bass styles.
• Deterministic engines realize it — pure music-theory functions turn that intent into in-key notes, which a mido writer turns into MIDI and a from-scratch numpy synthesizer turns into audio.

Because every decision an agent can make is a tool, both brains (Claude and the offline planner) drive the exact same action surface and produce the exact same trace format.

Architecture

            text prompt
                │
     ┌──────────▼───────────────────────────────────────────────┐
     │  Orchestrator (the harness)                                │
     │                                                            │
     │   Interpreter → Composer → Melodist → Arranger             │  each agent runs a
     │                               │                            │  ReAct loop:
     │                               ▼                            │   reason (thinking)
     │                            Critic ──reject──┐              │   → act  (tool call)
     │                               │approve      │ one targeted │   → observe (result)
     │                               │        revision (bounded)  │
     └───────────────────────────────┼────────────┘              │
                                      │                           │
          ┌───────────────────────────▼───────────────────────┐  │
          │  Tools  (the ReAct action space)                   │  │
          │  set_brief · set_direction · get_theory ·          │  │
          │  find_references · choose_references · set_global ·│  │
          │  set_structure · set_progression · design_melody · │  │
          │  set_instruments · set_bass · set_drums ·          │  │
          │  analyze_score · finish / approve / request_revision│ │
          └───────────────────────────┬───────────────────────┘  │
                                       ▼                          │
                            Score blackboard (shared state)       │
     └──────────────────────────────┬─────────────────────────────┘
                                     ▼
          performance.py  →  { melody, harmony, bass, drums } notes
                                     │
                 ┌───────────────────┴────────────────────┐
                 ▼                                         ▼
        midi_render.py (mido)                   synth.py (numpy software synth)
                 │                                         │
              song.mid                        song.wav ──(ffmpeg)──► song.mp3

The agents

Every agent also has analyze_score (a sensing action) and a terminal finish.

The two brains

Claude multi-agent (default when an API key is set). Each agent runs the canonical Messages-API tool-use loop with adaptive thinking; the model reasons, calls a tool, reads the observation, and continues until it finishes. Defaults to claude-opus-5-5. The API call degrades gracefully across SDK/account differences (server-side refusal fallback → plain streaming → drop effort → drop thinking), so the run never hard-fails on a capability mismatch.

# .env
ANTHROPIC_API_KEY=sk-ant-...

ark-audio "a cathedral of light at the edge of the world" --effort high

Claude on Microsoft Foundry (Azure). Point the brain at a Foundry resource instead and it uses the SDK's AnthropicFoundry client with API-key auth. A Foundry endpoint takes precedence over ANTHROPIC_API_KEY:

# .env
ANTHROPIC_FOUNDRY_API_KEY=...
ANTHROPIC_FOUNDRY_RESOURCE=my-resource   # or ANTHROPIC_FOUNDRY_BASE_URL=https://my-resource.services.ai.azure.com/anthropic/
ARK_MODEL=claude-opus-5-5                # your Foundry deployment name

If both ANTHROPIC_FOUNDRY_RESOURCE and ANTHROPIC_FOUNDRY_BASE_URL are set, the base URL is used.

Server-side refusal fallback is a first-party API feature, so it is skipped on Foundry. Adaptive thinking and effort still apply.

Deterministic (offline, always available). A rule-based planner maps the text to a feeling via a lexicon, then applies music-theory heuristics to make the same tool calls an LLM would. Reproducible given a --seed. This is what runs in CI and what powers --offline.

If no key is found (and --offline isn't set), the CLI automatically uses the deterministic brain and says so.

How the music is realized

• Theory (theory.py): scales/modes, diatonic chords, Roman-numeral resolution, scale-degree → MIDI, equal-temperament frequencies.

• Blackboard (blackboard.py): the declarative Score the agents edit.

• Direction (direction.py): reads a direction prompt into a section plan and constraints, and checks a realized piece against it.

• References (references.py, midi_analysis.py): the reference library, its search and name resolution, the measure of how closely a piece echoes its references, and the reader that turns a MIDI file into reference traits.

• Spec (spec.py): the library as tools for another model, with the definitions, a run_tool dispatcher, the recommended system prompt and the full machine-readable contract.

• Performance (performance.py): realizes the Score into notes:

  • a contour-driven melody that snaps to chord tones on strong beats, or a phrase-structured melody when a direction (or the Melodist) asks for one. That melody is built from motifs, keeps leaps within a limit, leans on the tonic, flows across sections and ends on a cadence;
  • intensity ramps within a section;
  • chord comping or sustained pads;
  • several bass styles (roots, root-fifth, pulse, walking, arp, tonic pedal);
  • synthesized drum patterns (four-on-floor, backbeat, breakbeat, a riser, …).

  All of it is seeded and reproducible.

• Synth (synth.py): additive (band-limited) oscillator voices with ADSR, per-instrument patches, synthesized percussion, an FFT-convolution reverb, and a soft-clipping master bus.

Development

First-time setup (virtual environment)

You need Python 3.10+ (the repo pins 3.12 in .python-version). On macOS the system python3 is 3.9, which is too old, so use the Python that uv installs or one from Homebrew.

Option A: uv (recommended)

git clone https://github.com/ir-dev/ark-audio-agent.git
cd ark-audio-agent
uv sync                      # creates .venv: the package (editable), anthropic, pytest
source .venv/bin/activate    # Windows: .venv\Scripts\activate

If Python 3.12 is missing, uv downloads it. uv sync also installs the dev group (pytest and the anthropic SDK) by default, so a later uv sync never drops the Claude brain.

Option B: plain venv + pip

git clone https://github.com/ir-dev/ark-audio-agent.git
cd ark-audio-agent
python3.12 -m venv .venv     # any Python 3.10+
source .venv/bin/activate    # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip
pip install -e ".[llm]" pytest

-e is an editable install, so changes under src/ take effect without reinstalling.

Check it works

pytest -q                                                 # theory + pipeline tests (offline, no key needed)
ark-audio "a quiet dawn over still water" --offline --seed 7
./examples/run_examples.sh                                # a spread of moods, offline

With the venv activated, ark-audio and pytest run directly. Without activating it, prefix them with uv run.

API key (optional)

A key is only needed for the Claude brain. Copy the template and fill in one provider:

cp .env.example .env         # .env is git-ignored; never commit it

The CLI reads ./.env itself, on every OS, so just run it from the repo root:

uv run ark-audio "a storm breaking over the sea"

License

MIT.