Skip to main content

Speculative Programmatic Tool Calling

Speculative programmatic tool-calling (sPTC) is a technique for harnesses that use tools like sub-agents / sub-calls in code. While the LLM is streaming tokens to generate a REPL call, sPTC speculates and queues up tool calls in the partially-generated code that act as Futures when the actual code is executed.

Learn more in the blogpost here.

sPTC vs serial comparison

Many harness designs like Recursive Language Models (RLMs) and CodeAct rely on programmatic tool-calling (PTC), where all tools are embedded as functions inside a single code REPL tool that is generated per turn. For RLMs in particular, sub-LLM and sub-RLM calls are expensive, often blocking tools in code that take up a majority of the runtime. sPTC is the general technique of speculating tool and sub-LLM calls that will happen as the root LLM is generating the codeblock, allowing the RLM to batch and asynchronously compute these expensive calls while the full codeblock is still being generated to overlap these calls with the logic of the code REPL.

baseline   tokens──────────────────▶ exec: call₁──▶call₂──▶…──▶callₙ──▶ answer
spec-ptc   tokens──────────────────▶ exec: claim·claim·claim ──▶ answer
                 ╲ call₁ ▶▶▶ done ╱
                  ╲ call₂ ▶▶▶ done╱     (calls run inside generation time)

This repository is a simple library and demo for this technique.

Getting Started

You can either clone this repository (uses uv), or install with:

pip install spec-ptc

The Speculator object is used to track and store tools to be speculated, as well as the shadow REPL that is used to speculate. You can add tools with the spec.tool decorator and control whether you want them to be speculated or not.

The simplest example is to install tool hooks into the REPL you already have, feed tokens as they stream and feed them to the speculator, then exec as usual when finished:

from spec_ptc import Speculator

spec = Speculator()


# tools can also be async
@spec.tool(speculatable=True, pure=True)  # add as tool to be speculated
def llm_query(prompt: str) -> str:
    return sub_lm.complete(prompt)


@spec.tool()  # side effects: never speculated
def send_report(text: str) -> str:
    return mail.send(text)


ns.update(spec.hooks())  # same names, claim-or-run

code = ""
with spec.turn(repl_locals=ns) as t:  # snapshot → discarded shadow fork
    for delta in model_stream:
        code += delta
        t.feed(delta)  # closed stmts launch calls now
exec(code, ns)  # hits return immediately

For the RLM this is one line: from demo.rlm import patch_rlm; patch_rlm().

example.py is an example you can start with for looking how this is done for the RLM.

For arbitrary harness, we provide a simple daemon spec-ptc-daemon that runs the same shadow + store out of process (default socket /tmp/spec-ptc.sock) with four JSON-lines messages:

turn_begin {vars}      snapshot REPL variables into the shadow
feed {delta}           stream tokens; the daemon launches calls
resolve {tool, args}   → hit{result} | miss   (miss: run the tool yourself)
turn_end               evict leftovers, return hit/miss counts
from plugins.client import SpecClient  # ~60 lines, stdlib only — copy it

c = SpecClient()
c.turn_begin({"context": doc})
c.feed(delta)  # per streamed token
hit = c.resolve("llm_query", [prompt])  # result, or None → call it yourself
c.turn_end()

Wrappers in plugins/: Claude Code (PreToolUse), OpenCode, Pi-mono.

Metadata

Release files for spec-ptc 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for spec-ptc 0.1.1
File Size Uploaded Provenance Actions
spec_ptc-0.1.1.tar.gz 52.4 kB Without provenance Details for spec_ptc-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for spec-ptc 0.1.1
File Interpreter ABI Platform Actions
spec_ptc-0.1.1-py3-none-any.whl Python 3 none any Details for spec_ptc-0.1.1-py3-none-any.whl

Total release size: 90.7 kB

Release files / spec_ptc-0.1.1.tar.gz

Download URL spec_ptc-0.1.1.tar.gz
Size 52.4 kB
Tags Source
SHA-256 checksum
How to use checksums
d6b7f1bb0a5cbae56924a8bce23acf5dedd31b6c29b1471b7cbe0e6cf44e73b2
BLAKE2b-256 checksum
How to use checksums
1514992df2852404087f065e5370951aa9fa740b5f9a084249ab1367e13a7288
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / spec_ptc-0.1.1-py3-none-any.whl

Download URL spec_ptc-0.1.1-py3-none-any.whl
Size 38.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
75936f6c8804057e8506614eea9a1cd62eb904e4b7896f3a2289ccbd73fa653c
BLAKE2b-256 checksum
How to use checksums
b383ee5086ec05e8df25ec9519193ed0c9db7c1e83b92ee09f70cad9be6546de
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for spec-ptc 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page