margin-cost

Your invoice shows the bill. It rarely shows which vendor earned it.

pip install margin-cost
margin-cost --example | margin-cost -

That is the whole thing. No file to author, no config, no account, no backend — --example emits a sample estate and the second call reports on it, so you get real output on the first command you run. Point it at your own file when you have one.

Then make it a gate, which is the part nothing else in this category does:

margin-cost estate.json --fail-over 0.14        # exit 1 when cost per outcome is over budget
margin-cost estate.json --fail-if-unmeasured    # exit 1 when an outcome was guessed, not measured

Exit codes are the contract: 0 pass, 1 the gate says no, 2 could not run — so a CI author can tell a refusal from a failure. Drop it in a workflow and a cost regression fails the build instead of showing up on next month's invoice.

Your invoice shows the bill. It rarely shows which vendor earned it.

margin-cost divides spend by outcomes, per vendor, from a file on your disk. No account, no signup, no backend. Nothing leaves the machine.

Why cost per outcome

Cost per token tells you what you spent. It says nothing about what the spend bought. Two agents can burn the same tokens and resolve very different numbers of tickets. The question a budget owner actually has is "which of my vendors is worth its invoice?", and the honest denominator for that is the outcome, not the token.

That number works even on vendors that expose nothing. Glean bills per seat and you will never see its tokens, but you know what it produced, so you can still divide its invoice by the outcomes it delivered.

What this tool does not do

margin-cost measures. It does not act.

It will tell you that one vendor costs $4.03 per outcome and another costs $0.018. It will not switch anything, and it has no opinion about whether the cheaper one would have held up.

That second question is the hard one, and it is deliberately not in here:

in this tool, free	not in this tool
cost per outcome, per vendor	routing a call to a cheaper model
retry tax, cache efficiency, token yield	proving the cheaper model held quality first
the three coverage tiers	promoting a route, and reverting it when parity slips
runs on your machine, no account	any of it running continuously

Why the line sits there. Measuring your spend is becoming table stakes — several vendors give it away now, and gatekeeping it would be both futile and annoying. Deciding what to change and proving the change did not cost you quality is the part that is actually hard, and that is the commercial product.

If you run this and dislike the number, that is the tool working. What to do about the number is at trymargin.io.

⚠️ No account, no signup, and no telemetry — this tool never phones home, including to us. We have no idea you ran it. (margin-meter, published separately, is the opposite by design: it is the client that does send call economics to a Margin ingest API. Different package, different job, on purpose.)

Install

pip install margin-cost

Or run it straight from a clone, since it has no dependencies:

python -m margin_cost estate.json

Use

Write a corpus file describing your vendors, then point the tool at it:

margin-cost estate.json
margin-cost estate.json --json     # machine-readable
margin-cost --example              # print a sample corpus to edit

# gate your build on it — exit codes, ready for CI
margin-cost estate.json --fail-over 0.14       # exit 1 when cost/outcome is over budget
margin-cost estate.json --fail-if-unmeasured   # exit 1 on an outcome that was guessed

Output:

COST PER OUTCOME  ·  measured locally, nothing sent anywhere

  vendor                       tier        cost/outcome         spend   passed   retry   cache
  --------------------------------------------------------------------------------------------
  support-agent (self-hosted)  measured         $0.0182         $0.04        2   32.7%   14.0%
  Glean                        allocated        $4.0323     $5,000.00     1240       —       —
  Zapier                       uncovered   NOT MEASURED       $300.00        0       —       —

COVERAGE  ·  a vendor is only 'measured' if you gave it the calls
  measured         $0.04   real tokens, real per-call cost
  allocated    $5,000.00   invoice ÷ outcomes
  uncovered      $300.00   spend seen, no outcomes — NOT priced
  ------------------------
  total        $5,300.04   94.3% of spend has a cost per outcome

The three tiers, and why the tool will not overstate them

Spend cannot always be measured to the same depth. The tool names the depth it had rather than pretending to more:

• measured — you gave it the per-call rows, so it sees real tokens and real per-call cost. Full metrics: cost per outcome, retry tax, cache efficiency, token yield.
• allocated — you gave it an invoice total and an outcome count, so it divides. A real cost per outcome, coarser than metered, and the only per-token rates it can honestly show are none.
• uncovered — you gave it spend and no outcomes. It reports the money and says the per-outcome cost is not measured.

The tier is derived from the evidence in your file. You cannot mark a vendor measured by writing it in the JSON; only per-call rows earn that. A cost per outcome with no data behind it prints NOT MEASURED, never a flattering zero.

Gate your builds on it

A report is a number you read. A gate is a decision your pipeline can act on, and the exit code is the whole contract:

exit	meaning
0	pass — the gate ran and nothing failed it
1	the gate says no — a measured number is over budget, or an outcome was guessed
2	could not run — unreadable input, or no outcome in the estate to price

--fail-over 0.14 compares the estate's blended cost per outcome (the same measured-plus-allocated number the report prices) against your budget and exits 1 when it strictly exceeds it. --fail-if-unmeasured exits 1 the moment any vendor's outcome is not positively determined — spend with no outcome count, or a metered vendor whose calls all failed. That second flag is the honesty rule made executable: a cost-per-outcome whose denominator was assumed is a fabricated number, so the gate refuses rather than passing a guess.

Wire it up like any other CI check:

# .github/workflows/ai-spend.yml
name: ai-spend gate
on:
  schedule:
    - cron: "17 9 * * 1"   # every Monday; run it whenever your corpus updates
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install margin-cost
      - run: margin-cost estate.json --fail-over 0.14 --fail-if-unmeasured

exit 2 exists so a CI author can tell a refusal from a failure — a corpus that stops producing outcomes stops the pipeline with "could not run", not with a pass.

What it does not do

It measures. It does not act. There is no routing, no model swapping, no "we will cut this for you." That line is deliberate: measuring your spend is something you should be able to do yourself, for free, without trusting anyone.

Corpus format

A JSON object with a vendors list. Each vendor is one of three shapes:

{
  "vendors": [
    {
      "name": "support-agent (self-hosted)",
      "calls": [
        {"cost_usd": 0.0121, "input_tokens": 820, "output_tokens": 190,
         "cache_read_tokens": 400, "is_retry": false, "outcome": "pass"}
      ]
    },
    {
      "name": "Glean",
      "invoice_usd": 5000.0,
      "outcomes": {"passed": 1240, "total": 1500}
    },
    {
      "name": "Zapier",
      "invoice_usd": 300.0
    }
  ]
}

• A vendor with calls is measured. Each call needs cost_usd; tokens, the retry flag, and the pass/fail outcome are optional. An outcome that is neither pass nor fail is refused, not guessed.
• A vendor with invoice_usd and a positive outcomes.passed is allocated.
• A vendor with invoice_usd and no outcome count is uncovered.

Invoices as CSV

Invoices usually arrive as a spreadsheet export, so a CSV works directly:

margin-cost invoices.csv

vendor,invoice_usd,passed,total
Glean,5000.00,1240,1500
Siena,2200.00,860,900
Zapier,300.00,,

Headers are matched loosely (vendor/name/service, invoice_usd/spend/cost, passed/resolved, total). A $ and thousands commas in the spend column are fine. A CSV is the invoice path, so every row is allocated or uncovered; the measured tier needs per-call rows, which are JSON.

License

MIT.