ai-taxman 🧾
knock knock
Open up. It's the taxman — you're being audited.
A command-line tool for auditing AI providers: send a file of messages to a model and record every raw response. Also usable as a Python package.
Install
uv tool install "ai-taxman[all]" # every provider
#uv tool install "ai-taxman[openai]" # one provider
#uv tool install "ai-taxman[openai,anthropic]" # several
Currently available: openai, gemini, and anthropic.
Quick start
taxman init # set up a project in this directory
taxman audits new openai <audit-name> # writes taxman/audits/<audit-name>.yaml
Edit the audit file: set api_key_env: to the name of the variable holding your key, and change any other setting.
Then write your messages, one per line:
export OPENAI_API_KEY="sk-..."
echo "When is the next US federal election?" > taxman/messages/<audit-name>.txt
taxman audits validate <audit-name> # check everything; sends nothing
taxman collect <audit-name> # sends the messages; bills your account
Responses are written to taxman/data/<audit-name>/<run_id>/responses.jsonl.
How it works
- Set up a project.
taxman initcreates ataxmanproject, which contains a handful of directories. - Create an audit.
taxman audits new <provider> <audit-name>creates an "audit file" (<audit-name>.yaml) for the user-specified provider (<provider>). All audit settings and parameters are specified here with human-readable text. - Write messages. A
.txt"messages file" contains all of the messages that will be sent (separately) to the AI provider. One message per line. - Add a system prompt (optional). A system prompt, sent with every message, can be specified in a separate
.txtfile. - Collect.
taxman collect <audit-name>collects the audit as specified in the audit file and creates a manifest of what was run.
Projects
A project is a directory with a taxman.yaml at its root.
taxman init creates a project in the current working directory and prompts the user to set paths manually or accept the defaults.
If taxman init --yes is called, default paths are automatically accepted.
taxman.yaml # marks the root and records the folders below
taxman/
├── audits/ # one YAML file per audit
├── messages/ # .txt messages files
├── prompts/ # system prompt files
├── data/ # collected responses
└── logs/ # run logs
Commands work from anywhere inside the project, like git.
Audits
taxman audits new <provider> <audit-name> takes no other arguments.
It writes every setting at its default, with a comment above each giving its type, default, options, and a link to the provider's docs.
You configure an audit by editing that file.
Without the comments:
audit: <audit-name>
provider: openai
api_key_env: <insert_api_key_env_var_here>
messages: taxman/messages/<audit-name>.txt
system_prompt:
output:
dir: taxman/data/{audit}/{run_id}
filename: responses.jsonl
compress: false
log_dir: taxman/logs/{audit}
execution:
repeats: 1 # times each message is sent
max_concurrency: 8
batch: false
timeout_s: 120
max_retries: 5
on_error: continue
shuffle: false
model: # provider-specific; this is OpenAI's
name: gpt-5
temperature:
# ...
Relative paths resolve against the project root.
API keys
api_key_env: names one environment variable, and taxman reads only that one. It never
stores a key. Export the variable however you like (shell profile, direnv, CI secret).
Because each audit names its own variable, one machine can hold several keys for the same
provider.
If the field is missing, still the placeholder, or names an unset variable, collect stops
before sending anything.
Messages
One message per line in a .txt file. Blank lines and lines starting with # are skipped;
start a line with \# to send a message that begins with #.
System prompt
Put the text in a file and name it in the audit:
system_prompt: taxman/prompts/neutral.txt
Each provider sends it through its own API field. The manifest records the file's path, text, and hash; every response row records the hash.
Collecting
taxman collect <audit-name>
Each message is sent repeats times, concurrently. Each run writes to its own directory:
responses.jsonl: one row per response, flushed as it arrives.manifest.json: resolved config, tool version, messages-file hash, counts, and timings. It is written before the first request and updated when the run ends.
The manifest's status is running until the run ends, then complete, stopped_early,
interrupted, or failed. n_messages × repeats is the number of responses expected. A
killed run keeps every response already received.
| Option | Effect |
|---|---|
-r, --repeats N |
Override execution.repeats |
-c, --concurrency N |
Override execution.max_concurrency |
-q, --quiet |
Print only the summary |
--log-file PATH |
Write the log to this file only |
--log-level LEVEL |
debug, info (default), warning, or error |
-b, --background |
Run detached; print the pid |
--run-id ID |
Use this run id; an existing id appends to that run |
A run id may contain letters, digits, ., -, and _, and must start with a letter or
digit. If you remove {run_id} from output.dir, a second run into the same directory is
refused.
Logs
Each run logs one line per response to stderr and to <output.log_dir>/<run_id>.log. The
summary goes to stdout. Message text and API keys are never logged.
Background runs
-b detaches the run. stdout is the pid and nothing else; the run id, log path, and output
path go to stderr. The pid is also saved to collect.pid in the run directory. The audit is
validated before the run detaches.
openai=$(taxman collect openai-probe -b)
anthropic=$(taxman collect anthropic-probe -b)
while kill -0 "$openai" 2>/dev/null || kill -0 "$anthropic" 2>/dev/null; do sleep 5; done
kill <pid> stops gracefully: in-flight requests finish and the manifest is set to
stopped_early. After kill -9 the responses on disk survive, but the manifest still says
running.
Output
One JSON object per line, the same shape for every provider:
{
"schema_version": 1,
"audit": "<audit-name>",
"run_id": "20260830T142201Z-a1b2c3",
"message_id": "m0007",
"message_hash": "sha256:…",
"message": "When is the next US federal election?",
"repeat": 2,
"system_prompt_hash": "sha256:…",
"provider": "openai",
"model": "gpt-5",
"requested_at": "…", "received_at": "…", "latency_ms": 812,
"status": "ok",
"error": null,
"attempts": 1,
"raw": {}
}
raw is the provider's response, verbatim. taxman does not parse it; extracting text,
token counts, or citations is a separate step run on the data afterward.
Schema versions
Changes are additive unless listed here.
| Version | Change |
|---|---|
| 1 | Initial schema. |
Commands
| Command | What it does |
|---|---|
taxman init |
Set up a project in this directory |
taxman audits new <provider> <audit-name> |
Write a new audit file |
taxman audits list |
List the project's audits |
taxman audits show <audit-name> |
Print an audit's resolved settings |
taxman audits validate <audit-name> |
Check an audit without sending anything |
taxman collect <audit-name> |
Run an audit |
taxman providers |
List the installed providers |
taxman doctor |
Check PATH and tab completion, and offer to fix them |
Python
from ai_taxman import run_audit
result = run_audit("<audit-name>")
print(result.n_ok, result.n_error, result.output_path)
A name resolves as on the command line; a path ending in .yaml is read directly.
run_audit also accepts an AuditConfig from load_audit() and an on_record callback.
Tab completion
taxman --install-completion # or accept the offer from `taxman doctor`
Restart your shell. Completion covers commands, providers, models, and audit names.
zsh: check startup time. The installer appends
fpath+=~/.zfunc; autoload -Uz compinit; compinit to ~/.zshrc. If your setup already runs
compinit (oh-my-zsh, prezto, most prompts), zsh then rebuilds its completion cache on every
start. Run time zsh -i -c exit twice; if the second run takes much more than 0.2s:
- Shorten the installer's line to
fpath+=~/.zfunc. - Move it to the top of
~/.zshrc, above anysourceline.compinitreadsfpathonce, so a directory added after it is ignored. rm ~/.zcompdumpand open a new terminal.
Check with print -r -- ${_comps[taxman]:-NOT REGISTERED}. It should print _taxman; if it
prints NOT REGISTERED, the fpath line is still below compinit.
The installer also adds zstyle ':completion:*' menu select, which enables arrow-key menus
for every command. Delete it if you don't want that.
To uninstall: rm ~/.zfunc/_taxman and delete the block in ~/.zshrc marked
# taxman additions. bash and fish need no extra steps.
Supported providers
| Provider | Extra | Status |
|---|---|---|
| OpenAI | ai-taxman[openai] |
Responses API; no batch mode yet |
| Anthropic | ai-taxman[anthropic] |
Messages API; no batch mode yet |
| Google Gemini | ai-taxman[gemini] |
Interactions API; no batch mode or safety settings |
Each provider is isolated from the others. To add one, see CLAUDE.md and
docs/provider-apis/.
Development
uv sync --all-extras
uv run pytest
License
MIT — see LICENSE. No tax is owed on this one.
Metadata
Release files for ai-taxman 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ai_taxman-0.1.1.tar.gz | 64.7 kB | Without provenance | Details for ai_taxman-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ai_taxman-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for ai_taxman-0.1.1-py3-none-any.whl |
Total release size: 155.7 kB
Release files / ai_taxman-0.1.1.tar.gz
| Download URL | ai_taxman-0.1.1.tar.gz |
|---|---|
| Size | 64.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c0843d1e6592319ae1933daa7b67bf7878ce556c3f0f1454b82ad92d4f7eaca0
|
|
BLAKE2b-256 checksum How to use checksums |
5c429a8649adfe31f409701d4ffaa272ec292c88e792c45584510a3e79a67ac7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.10.2 {"installer":{"name":"uv","version":"0.10.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / ai_taxman-0.1.1-py3-none-any.whl
| Download URL | ai_taxman-0.1.1-py3-none-any.whl |
|---|---|
| Size | 90.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ce7f086a71f3750688ac6b095ee00cc81fea178ef63b671f4a82d1ed53426c37
|
|
BLAKE2b-256 checksum How to use checksums |
98f1087eb6e0f6cf9f1d703e552113aef4a8b48063c28a14fc566b61e0ff2121
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.10.2 {"installer":{"name":"uv","version":"0.10.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for ai-taxman 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.