bulk-post

A near-stdlib Python CLI that fires templated HTTP requests driven by CSV data. You define the request — URL, method, body, headers — with {{placeholder}} slots, and each CSV row supplies the values that fill them: one request per row, or a multi-step request workflow per row in --workflow mode. Supports bearer or basic auth (default: no auth) with automatic 401 re-prompt, a live terminal UI with pause/resume, parallel execution, and a redrive file for failed rows. Third-party dependencies: PyYAML and jsonpath-ng — both always installed, but lazily imported (PyYAML only on the --workflow code path, jsonpath-ng only on the workflow-variables code path).

Requirements

• Python 3.12+
• PyYAML — runtime dependency; lazily imported, only exercised in --workflow mode
• jsonpath-ng — runtime dependency; lazily imported, only exercised when a workflow declares variables

Installation

From PyPI

Install from PyPI:

uv tool install bulk-post   # or: pipx install bulk-post  /  pip install bulk-post
bulk-post --help

With Homebrew

Install from the tap:

brew tap true-monte-kristo/tap
brew trust true-monte-kristo/tap   # Homebrew 6.x requires trusting third-party taps once
brew install bulk-post
bulk-post --help

From source

Install globally from a checkout with uv:

uv tool install .
uv tool install . --reinstall   # re-install after changing the code

Or run directly without installing (from the repo root):

python -m bulk_post --help          # ensure pyyaml and jsonpath-ng are installed (uv run handles this)

Usage

bulk-post -u <url-template> -c <csv-file> [options]

Examples

Cancel every invoice in a CSV using DELETE:

bulk-post \
  -u "https://api.example.com/invoices/{{id}}/cancel" \
  -c invoices.csv \
  -m DELETE

PATCH with a JSON body, 200 ms between requests, verbose output:

bulk-post \
  -u "https://api.example.com/invoices/{{id}}/status" \
  -c invoices.csv \
  -m PATCH \
  -b '{"status": "cancelled", "reason": "{{reason}}"}' \
  -d 200 \
  -v

POST form-encoded data:

bulk-post \
  -u "https://api.example.com/items" \
  -c items.csv \
  -m POST \
  -b "id={{id}}&status={{status}}" \
  -C "application/x-www-form-urlencoded"

Resume after a failure at row 47:

bulk-post -u "https://api.example.com/items/{{id}}" -c items.csv -o 47

CLI flags

CSV format

The CSV must have a header row. Column names are used as placeholder names in --url, --body, and --header values. Every {{placeholder}} in the URL, body, or header values must match a column name; the script exits with an error if any are missing.

The input delimiter is detected automatically (comma, semicolon, tab, or pipe), falling back to comma when it can't be determined. The failed-rows redrive CSV is written with the same delimiter as the input. There is no delimiter flag.

id,reason
1001,duplicate
1002,customer_request

Auth

Select the auth method with --auth-type / -a (default: none):

Bearer token

Pass --auth-type bearer (or -a bearer). Token resolution order: --token / -t flag → BULK_TOKEN env var → interactive prompt at startup.

If the server returns 401 mid-run, the script pauses, prompts for a fresh token, and retries the failed row automatically. This is handy when tokens are short-lived SSO bearer tokens that must be copied manually (e.g. from browser DevTools) and cannot be fetched programmatically.

Basic auth

Pass --auth-type basic (or -a basic). Credentials (user:pass) are resolved in the same order: --user / -U flag → BULK_USER env var → interactive prompt. On 401, the script prompts for new credentials and retries.

No auth (default)

The default when --auth-type is omitted (or pass --auth-type none / -a none explicitly). No Authorization header is sent.

Retries

Retries are disabled by default. Pass --retry-on with a comma-separated list of HTTP status codes to enable them:

bulk-post -u "https://api.example.com/{{id}}" -c rows.csv -R 503,429 -B exponential

Only the exact listed status codes trigger a retry — network errors and timeouts do not. If the server returns a Retry-After or X-Retry-After header, the wait is extended to at least that value before the next attempt. All waits are capped by --max-retry-delay. Each retry prints a short [RETRY] notice; --verbose adds the elapsed time and body snippet of the failed attempt.

Flags:

Notes:

• 401 cannot be listed in --retry-on when --auth-type is bearer or basic — the 401 auth-refresh flow owns that status code.
• Retry sleeps are interruptible: /pause freezes the countdown, /exit abandons the wait and routes the row to the redrive file.
• In workflow mode, retry flags are not available; use retry_policy: in the workflow YAML instead (see Workflow mode).

Terminal UI

When running in an interactive terminal, a live bottom bar shows:

• Progress bar — current / total rows with a visual fill bar
• Command input — type a command and press Enter; Tab autocompletes

Available commands:

In non-TTY mode (piped input, CI, test environments) the bottom bar is skipped and no interactive commands are available.

Redrive file

Rows that fail (network error, non-2xx response, or substitution error) are written to the redrive file. By default this is <csv-stem>_failed.csv next to the input file. Re-run with -c <stem>_failed.csv to redrive only those rows.

If no rows fail, the redrive file is deleted automatically.

Workflow mode

Instead of a single --url template, you can define a multi-step workflow in a YAML file and run it with --workflow / -w.

Each CSV row fires all steps in document order. Steps within a row are always sequential; --parallel controls per-row concurrency across rows.

Workflow YAML format

workflow:
  persist_context: false           # optional; default false — see "Redrive and resume" below
  description: Optional human-readable description  # skipped at runtime

  groupA:                          # logical grouping for shared auth
    auth:
      type: bearer                 # bearer | basic | none
      token: some_token            # optional — prompted if omitted
    endpoints:
      - step-name:                 # user-chosen name; unique within the group
          url: https://api.example.com/{{id}}
          method: POST             # default POST
          headers:
            Content-Type: application/json
            X-Custom: value
          body: '{"key": "{{col}}"}'
          on_error: stop           # stop (default) | continue
          auth:                    # step-level auth overrides group auth
            type: bearer
            token: override_token

  groupB:
    auth:
      type: basic
      user: alice
      password: secret
    endpoints:
      - another-step:
          url: https://other.example.com/{{id}}
          method: DELETE

  groupC:                          # no auth
    endpoints:
      - no-auth-step:
          url: https://public.example.com/{{id}}
          method: GET

Key rules:

• Execution order — steps fire in the order they appear in the document (top to bottom across all groups).
• Group auth — all steps in a group inherit the group's auth unless they declare their own.
• on_error — stop (default) halts remaining steps for that row and writes it to the redrive file; continue logs the failure, writes the row, and proceeds to the next step.
• Placeholders — {{col}} in url, body, and header values is replaced with the matching CSV column value, same as in single-URL mode.

Workflow retries

Each group and each endpoint may carry an optional retry_policy: block. An endpoint-level retry_policy replaces the group's entirely — there is no merging. retry_policy: none on an endpoint disables any inherited group policy. on_error evaluates only the final post-retries result.

workflow:
  groupA:
    auth:
      type: bearer
    retry_policy:           # group-level policy, inherited by all endpoints in the group
      retry_on: 503,429     # required: comma-separated string or YAML list
      backoff: exponential  # optional: fixed (default) or exponential
      multiplier: 1.5       # optional: exponential only; defaults to 1.5
      delay: 200            # optional: fixed delay / initial delay in ms; defaults to 200
      max_retries: 5        # optional: retries after the initial attempt; defaults to 5
      max_delay: 30000      # optional: hard cap on any single retry wait in ms; defaults to 30000
    endpoints:
      - call-api:
          url: https://api.example.com/{{id}}
          method: POST

  groupB:
    endpoints:
      - call-other:
          url: https://other.example.com/{{id}}
          method: DELETE
          retry_policy:     # endpoint-level policy replaces any group policy entirely
            retry_on: 503,429
            backoff: fixed
            delay: 200
            max_retries: 5

  groupC:
    retry_policy:           # group has a policy …
      retry_on: 503
    endpoints:
      - no-retry-step:
          retry_policy: none  # … but this endpoint opts out
          url: https://public.example.com/{{id}}
          method: GET

Redrive and resume

Mid-workflow resume is opt-in via a persist_context: true key at the top of the workflow: mapping (default false).

• persist_context: false (default): failed rows are written to the redrive file with only the original input columns. Re-running always starts each row from step 1. If the input CSV contains _bulk_post_step or _bulk_post_var/… context columns (e.g. from a previous run with persist_context: true), they are ignored, with a startup warning printed to stderr.

• persist_context: true: when a step fails, the row is written to the redrive file with an extra column _bulk_post_step set to the path of the first failed step (e.g. groupA/step-name) plus any persisted variable columns. Re-running with that redrive CSV skips all steps before the failed one, resuming mid-workflow automatically.

Security note: persist_context: true writes response-derived data (potentially sensitive) to disk in plaintext. Do not share or commit redrive CSVs produced with this option enabled.

Workflow variables

A step can capture a value from an earlier step's JSON response and pass it as a {{$name}} placeholder into that step's url, headers, or body. This lets you chain steps — for example, create a resource in step 1 and use the returned id in the URL of step 2.

Variables are declared under a variables: key at group level (inherited by all endpoints in the group) and/or at endpoint level (overrides the group on name conflict):

workflow:
  groupA:
    auth:
      type: bearer
    endpoints:
      - create-item:
          url: https://api.example.com/items
          method: POST
          body: '{"name": "{{name}}"}'

  groupB:
    endpoints:
      - delete-item:
          # Capture the id from create-item's response at endpoint level
          variables:
            $id:
              source: .workflow.groupA.create-item  # leading dot and "workflow." prefix are optional
              jsonPath: $.id                        # JSONPath; first match is used
              nullable: false                       # fail this step if id is missing
          url: https://api.example.com/items/{{$id}}
          method: DELETE

Variable rules:

• Names must start with $ (e.g. $id) and are referenced as {{$id}} in URL, headers, and body.
• source is written as .workflow.<group>.<endpoint> (or just <group>/<endpoint>). It must refer to an endpoint that runs before the current step — forward and self references are rejected at startup.
• jsonPath uses full JSONPath syntax (powered by jsonpath-ng). Only the first match is used. A match that is an object or array (non-scalar) fails the step.
• nullable defaults to true. When false, a null value or no-match fails the step (row written to redrive file); when true, it resolves to an empty string.
• Variable values are scoped to a single CSV row and are never shared across rows.
• All variable declarations are validated at startup — undefined references, bad names, unreachable sources, and invalid JSONPath expressions all cause an immediate exit with a clear error.

Resume/redrive with variables:

Requires persist_context: true at the top of the workflow: mapping. When enabled, on row failure any resolved variable values are persisted into reserved redrive-CSV columns named _bulk_post_var/<source_path>/<name>. Re-running the redrive CSV skips completed steps and reads these persisted values for variables whose source step was skipped. When persist_context is false (the default), variable values are not persisted and re-runs always start from step 1.

Security note: redrive CSVs may contain response-derived data (potentially sensitive) in plaintext. Do not share or commit redrive CSVs that were produced from workflows using variables.

Example

bulk-post -w workflow.yaml -c rows.csv

Running tests

uv run python -m unittest discover tests/

Tests use stdlib unittest. PyYAML and jsonpath-ng are runtime dependencies (always installed); the workflow-parsing cases use PyYAML and the workflow-variable cases use jsonpath-ng. uv run provides both from uv.lock automatically; if you run python -m unittest directly, do so inside a virtualenv that has both packages.