python-plugin 0.1.0

Wire-compatible Python port of go-plugin (gRPC subprocess plugins with AutoMTLS)

pyplugin

A Python port of go-plugin, byte-for-byte wire-compatible with the original — including AutoMTLS with ECDSA P-521 ephemeral certs.

A Python host can launch a Go plugin built with go-plugin, and a Go host built against go-plugin can launch a Python plugin built with pyplugin. Both directions, with or without AutoMTLS. Verified against the upstream examples/grpc/plugin-go-grpc binary in the test suite.

Install

pip install python-plugin

The PyPI distribution name is python-plugin; the Python import name is pyplugin (from pyplugin import Client, serve, ...).

Why grpclib (and async)

go-plugin generates ECDSA P-521 ephemeral certificates for AutoMTLS. grpcio is built on BoringSSL, which deliberately omits P-521 from its TLS signature algorithm list — there's no way to configure it back in, and we want exact wire-compat. grpclib is a pure-Python gRPC library on top of Python's ssl module (OpenSSL), which supports P-521 freely. That makes interop with stock go-plugin work out of the box.

The cost: the public API is async. Plugin servicers are async def; host code uses async with Client(...) as c: await c.start() etc.

Quick start

A complete runnable example lives in examples/greeter/ — clone the repo, pip install -e '.[dev]', then:

python examples/greeter/host.py "ada"            # insecure
AUTO_MTLS=1 python examples/greeter/host.py "ada" # P-521 mTLS

Plugin (async)

# my_plugin.py
from pyplugin import HandshakeConfig, Plugin, ServeConfig, serve
from grpclib.client import Channel

# stubs generated by grpclib's protoc plugin (see scripts/gen_protos.py):
import myservice_grpc, myservice_pb2

class MyServicer(myservice_grpc.MyServiceBase):
    async def Greet(self, stream):
        request = await stream.recv_message()
        await stream.send_message(myservice_pb2.GreetResponse(message=f"hello {request.name}"))

class MyPlugin(Plugin):
    def servicers(self, broker):
        return [MyServicer()]
    def stub(self, broker, channel: Channel):
        return myservice_grpc.MyServiceStub(channel)

if __name__ == "__main__":
    serve(ServeConfig(
        handshake_config=HandshakeConfig(
            protocol_version=1,
            magic_cookie_key="MYPLUGIN_COOKIE",
            magic_cookie_value="hello",
        ),
        plugins={"my": MyPlugin()},
    ))

Host (async)

import asyncio, sys
from pyplugin import Client, ClientConfig, HandshakeConfig

async def main():
    async with Client(ClientConfig(
        handshake_config=HandshakeConfig(1, "MYPLUGIN_COOKIE", "hello"),
        plugins={"my": MyPlugin()},
        cmd=[sys.executable, "my_plugin.py"],
        auto_mtls=True,           # P-521 mTLS, fully wire-compatible with go-plugin
    )) as client:
        stub = client.dispense("my")
        resp = await stub.Greet(myservice_pb2.GreetRequest(name="world"))
        print(resp.message)

asyncio.run(main())

What's implemented

Feature	Status
stdout handshake protocol (6/7 segments, base64.RawStdEncoding cert)	✅
magic cookie validation	✅
gRPC transport: unix sockets (POSIX) and TCP loopback	✅
AutoMTLS with ECDSA P-521 / SHA-512 (matches go-plugin)	✅
GRPCController.Shutdown graceful exit	✅
Kill ladder: Shutdown → SIGTERM → SIGKILL	✅
stderr forwarding with hclog parser (JSON + pretty)	✅
GRPCBroker bidirectional sub-channels (Accept/Dial)	✅
GRPCStdio post-handshake stdout/stderr stream	✅
ReattachConfig (host re-connects to running plugin)	✅
VersionedPlugins negotiation	✅
gRPC reflection + health (service name plugin)	✅
PLUGIN_MULTIPLEX_GRPC (broker over single socket)	❌ deferred (advertised as not supported)

Verified Python ↔ Go interop

The test suite includes 4 real interop tests against upstream go-plugin:

tests/interop/test_python_host_drives_go_plugin.py
    test_python_host_drives_go_plugin_no_mtls          ✓
    test_python_host_drives_go_plugin_with_p521_automtls ✓

tests/interop/test_go_host_drives_python_plugin.py
    test_go_host_drives_python_plugin_no_mtls            ✓
    test_go_host_drives_python_plugin_with_p521_automtls ✓

These run only when the binaries are present; the README of tests/interop/ describes how to build them. Out of the box you can reproduce the matrix locally:

# Build go-plugin's example KV plugin (Go)
git clone --depth=1 https://github.com/hashicorp/go-plugin /tmp/gp
(cd /tmp/gp/examples/grpc && go build -o /tmp/plugin-go-grpc ./plugin-go-grpc)
PYPLUGIN_GO_PLUGIN_KV=/tmp/plugin-go-grpc pytest tests/interop/test_python_host_drives_go_plugin.py

For the Go-host-drives-Python-plugin direction, see the small Go host template at the end of this README — drop it into tests/interop/go-host/ with a replace directive in go.mod pointing at the local go-plugin clone, go build, then point PYPLUGIN_GO_HOST_BIN at the binary.

Layout

src/pyplugin/
  handshake.py      # stdout protocol line format/parse
  cookie.py         # magic-cookie validation
  mtls.py           # ephemeral P-521 cert generation + ssl.SSLContext builders
  transport.py      # unix / tcp listener helpers
  server.py         # serve(ServeConfig) — sync entry, internal asyncio loop
  client.py         # Client / ClientConfig — async host launcher
  process.py        # cross-platform subprocess termination
  reattach.py       # ReattachConfig
  controller.py     # GRPCController.Shutdown servicer (grpclib async)
  broker.py         # GRPCBroker bidirectional multiplexer (grpclib async)
  stdio.py          # GRPCStdio post-handshake stream (grpclib async)
  health.py         # static grpc.health.v1 servicer (returns SERVING for "plugin")
  plugin.py         # Plugin ABC, PluginSet, VersionedPlugins
  logging_bridge.py # hclog (JSON + pretty) line parser
  errors.py         # exception hierarchy
  proto/            # vendored .proto files from go-plugin (verbatim)
  _generated/       # checked-in grpclib stubs
fixtures/example_kv/  # example KV plugin used by smoke tests
tests/                # 40 unit + Python↔Python tests
tests/interop/        # 4 real-go-plugin interop tests

Development

python3 -m venv .venv
.venv/bin/pip install -e '.[dev]'
.venv/bin/python scripts/gen_protos.py     # regenerate stubs
.venv/bin/python -m pytest                  # run tests

Go host template

Use this with go.mod's replace github.com/hashicorp/go-plugin => /path/to/clone:

package main

import (
    "fmt"; "io"; "log"; "os"; "os/exec"
    hclog "github.com/hashicorp/go-hclog"
    "github.com/hashicorp/go-plugin"
    "github.com/hashicorp/go-plugin/examples/grpc/shared"
)

func main() {
    log.SetOutput(io.Discard)
    client := plugin.NewClient(&plugin.ClientConfig{
        HandshakeConfig:  shared.Handshake,
        Plugins:          map[string]plugin.Plugin{shared.PluginGRPC: &shared.KVGRPCPlugin{}},
        Cmd:              exec.Command(os.Args[1], os.Args[2]),
        AllowedProtocols: []plugin.Protocol{plugin.ProtocolGRPC},
        AutoMTLS:         os.Getenv("AUTO_MTLS") == "1",
        Logger:           hclog.New(&hclog.LoggerOptions{Output: io.Discard, Level: hclog.Off}),
    })
    defer client.Kill()
    rpc, err := client.Client(); if err != nil { panic(err) }
    raw, err := rpc.Dispense(shared.PluginGRPC); if err != nil { panic(err) }
    kv := raw.(shared.KV)
    if err := kv.Put(os.Args[4], []byte(os.Args[5])); err != nil { panic(err) }
    v, err := kv.Get(os.Args[4]); if err != nil { panic(err) }
    fmt.Print(string(v))
}

License

MIT. The vendored .proto files in src/pyplugin/proto/ retain their upstream MPL-2.0 headers from hashicorp/go-plugin; MPL-2.0 is file-level and is compatible with MIT for the rest of the project.