s2s-forwarder

A Python library for sending log events to Splunk indexers using the Splunk-to-Splunk (S2S) protocol version 4 — the native protocol used by Splunk Universal and Heavy Forwarders. No official Python library exists for S2S v4 sending, so this community project was started to fill that gap for testing and edge-case purposes.

Install

Not yet published to PyPI. Install from a checkout:

pip install .                      # from the repo root
pip install -e .                   # editable, for development
pip install /path/to/s2s           # from elsewhere

That gives you the s2s-forwarder package and the s2s-fwd command. Python 3.9+ - the only runtime dependency is click.

Verify:

python -c "import s2s_forwarder; print(s2s_forwarder.__version__)"
s2s-fwd --help

Usage

Within a python script as a simple event sender

from s2s_forwarder import S2SClient, Event

with S2SClient("splunk.example.com", port=9997) as client:
    client.send(Event(
        raw="2026-08-19 10:30:00 something happened",
        index="main",
        source="s2s:appname",
        sourcetype="myapp",
    ))

Use the context manager. close() is required for correctness in this protocol — see "Closing the connection" — and with guarantees it even if your code raises. The explicit form (connect() … close()) also works.

Event fields: raw (required), time, index, host, source, sourcetype. Unset fields fall back to sensible defaults.

send() also accepts a list of Events, coalescing consecutive events that share one channel into fewer wire sends. Call send_single(event) directly if you want a guaranteed one-event-per-send with no coalescing at all.

client.send([Event(raw=line, source="app.log", sourcetype="myapp")
             for line in lines])

Event.time is a fallback, not an override. Splunk prefers a timestamp it can parse out of raw, and only uses Event.time when raw has none — verified live. So setting time on an event whose text already carries a timestamp does nothing. To control event time explicitly, either put the timestamp in raw (what a real forwarder does) or leave raw without one and set time.

Command line

# a single event
s2s-fwd --host splunk.example.com --index main --sourcetype syslog "something happened"

# one event per line from stdin - a bounded queue keeps memory flat even
# on a large file, and consecutive lines are coalesced into fewer sends
cat /var/log/app.log | s2s-fwd --host splunk.example.com --index main

# follow a live log
tail -f /var/log/app.log | s2s-fwd --host splunk.example.com --index main

--source, --sourcetype, --index and --event-host set event metadata; --sender-hostname sets the name this sender identifies as during the handshake. --batch-size/--batch-linger control how stdin lines are coalesced (defaults: up to 100 lines, waiting up to 0.1s for more before sending a partial batch — keeps a live tail -f responsive instead of stalling until a full batch arrives). s2s-fwd --help lists everything.

asyncio

AsyncS2SClient mirrors S2SClient — same arguments, same defaults, same semantics, same encoder. Both share one session implementation, and a test asserts the two constructors stay identical, so neither the protocol rules nor the policy built on them can drift between the clients.

import asyncio
from s2s_forwarder import Event
from s2s_forwarder.aio import AsyncS2SClient

async def main():
    async with AsyncS2SClient("splunk.example.com") as client:
        await client.send(Event(raw="2026-08-19 10:30:00 something happened",
                                index="main", sourcetype="myapp"))

asyncio.run(main())

One client owns one connection, and S2S channel state is per-connection, so send() serialises via an internal lock — concurrent callers are safe but share the wire. For real parallelism use several clients, i.e. several connections, which is what a forwarder does too:

async def worker(n):
    async with AsyncS2SClient("splunk.example.com") as c:
        for line in lines[n]:
            await c.send(Event(raw=line, source=f"/var/log/app{n}.log"))

await asyncio.gather(*(worker(n) for n in range(4)))

Python logging handler

Drop-in handler for the stdlib logging:

import logging
from s2s_forwarder.logging_handler import SplunkS2SHandler

handler = SplunkS2SHandler("splunk.example.com", index="main",
                           sourcetype="myapp", tls=True, ca_cert="ca.pem")
logging.getLogger().addHandler(handler)

logging.info("service started")     # returns immediately
logging.shutdown()                  # flushes and closes cleanly

Records are written by a background thread, so logging.info() never waits on the network. If Splunk is unreachable and the queue fills, the handler drops records and counts them (handler.stats) rather than blocking the application — stalling the program producing the telemetry is worse than losing some of it. Send failures never propagate to the caller and never kill the worker.

close() matters here: it triggers the per-channel padding described above. logging.shutdown() calls it at normal interpreter exit, so the protocol quirk stays invisible — but a hard kill -9 still costs the last record per channel.

Extra keyword arguments are passed to S2SClient, so TLS, mTLS and retry work exactly as they do there.

Containers: shipping a file-based log

The common pattern — a sidecar or entrypoint tailing an application's log file and forwarding it:

FROM python:3.12-slim

COPY . /src
RUN pip install --no-cache-dir /src && rm -rf /src

# certs are mounted at runtime, never baked into the image
ENV SPLUNK_HOST=inputs.example.splunkcloud.com \
    SPLUNK_INDEX=main \
    SPLUNK_SOURCETYPE=myapp

# tail -F (capital F) survives log rotation; -f does not
CMD tail -F /var/log/app/app.log | s2s-fwd \
      --host "$SPLUNK_HOST" \
      --index "$SPLUNK_INDEX" \
      --sourcetype "$SPLUNK_SOURCETYPE" \
      --source /var/log/app/app.log \
      --tls --ca-cert /certs/ca.pem --client-cert /certs/client.pem \
      --key-password "$(cat /certs/key.pw)" \
      --retry 20 --quiet

docker run -v /var/log/app:/var/log/app:ro -v ./certs:/certs:ro myimage

Points that matter in a container specifically:

• tail -F, not -f. Capital F reopens the file after rotation; lowercase -f keeps holding the old inode and silently stops producing lines.
• --retry — a long-running shipper will outlive indexer restarts and network blips. Without it the first failure ends the process.
• SIGTERM is handled. docker stop, podman stop and Kubernetes all terminate with SIGTERM, and Python's default handler kills the process without running cleanup — which would drop the last event on every channel, every restart. s2s-fwd and SplunkS2SHandler both install a handler so shutdown stays graceful. Give the container enough grace period to finish (docker stop -t 15).
• Mount certificates at runtime; don't bake credentials into an image.
• --source is worth setting explicitly, since the log's path inside the container is rarely the name you want in Splunk.
• PID 1: if you wrap the command in a shell, signals may not reach the process. Use an init (docker run --init) or exec the pipeline.

Note: for a containerised Python application, prefer the logging handler over tailing a file — it keeps tracebacks intact as single events and avoids the file entirely.

No fishbucket — position tracking is on you

A Universal Forwarder checkpoints its read position per file (its "fishbucket"), so a restart resumes from exactly where it left off instead of either re-reading old lines or silently skipping whatever was written during the outage. This library is a sender, not a UF reimplementation, and deliberately does not do this. tail -F | s2s-fwd has neither property: tail -F starts wherever tail starts — the current end of the file by default — so a restart during downtime silently skips every line written in the gap. Easy to miss if you're assuming UF-equivalent behaviour.

If gap-free delivery across restarts matters for your use case, examples/resumable_tail.py shows the minimal pattern: persist a byte offset, seek to it on start, advance it after each send() returns. It's not production-grade fishbucket parity (a real UF also tracks inode+size to catch rotation more robustly), but it's a starting point.

TLS

TLS is a transport wrap — the S2S bytes inside the tunnel are identical to the plaintext case. Parameters mirror Splunk's outputs.conf names so an existing forwarder config translates directly.

client = S2SClient(
    "splunk.example.com",
    tls=True,
    ca_cert="/path/ca.pem",          # Splunk's sslRootCAPath
    client_cert="/path/client.pem",  # Splunk's clientCert (mutual TLS)
    key_password="…",                # Splunk's sslPassword
)

s2s-fwd --host splunk.example.com --tls --ca-cert /path/ca.pem --index main "event"

Certificate verification is on by default. Splunk's own forwarder default (sslVerifyServerCert = false) is the opposite, but silently trusting any certificate is a poor default for a library, so disabling it is explicit: verify=False / --no-verify. For the common lab case of a self-signed cert whose CN doesn't match the IP, keep verification on and turn off just the hostname check (check_hostname=False / --no-check-hostname).

Error handling

from s2s_forwarder import S2SError, S2SHandshakeError, S2SConnectionError

S2SError is the base S2SHandshakeError means the handshake didn't complete S2SConnectionError means the connection failed or was closed

Library logging

s2s_forwarder logs its own operational events — connect, reconnect, rotate, retry, and (at DEBUG) wire-level detail like channel registration and closing padding — through the standard logging module, under the "s2s_forwarder" logger. Following the same convention as requests and urllib3, it attaches a NullHandler and stays completely silent until an application configures that logger itself:

import logging
logging.getLogger("s2s_forwarder").addHandler(logging.StreamHandler())
logging.getLogger("s2s_forwarder").setLevel(logging.INFO)  # or DEBUG

s2s-fwd wires this up for you via -v/-vv:

s2s-fwd --host splunk.example.com -v  "something happened"   # INFO
s2s-fwd --host splunk.example.com -vv "something happened"   # + DEBUG

Two recipes worth knowing about:

• Local troubleshooting — a rotating file handler on "s2s_forwarder" is the one diagnostic path guaranteed to work even when the Splunk connection itself is broken (examples/local_troubleshooting_handler.py).
• Self-diagnostics in Splunk — attach a second SplunkS2SHandler instance to "s2s_forwarder" itself with forward_internal_logs=True, so the library ships its own connect/reconnect/rotate activity into Splunk alongside a first handler shipping the application's normal logs (examples/self_diagnostic_logging.py). Close the two handlers explicitly, application handler first, rather than relying on logging.shutdown(): it closes handlers in reverse creation order, so the diagnostics handler would otherwise close before the handler it's observing has necessarily done anything worth observing yet.

SplunkS2SHandler does not ship the library's own log records by default (forward_internal_logs=False). This matters because the handler is normally attached to the root logger, which s2s_forwarder's records propagate to — so without the filter the handler is fed by the thing it is feeding. Left unchecked that is not merely untidy: a failing indexer makes each failed batch log a warning that becomes the next batch, which also fails, indefinitely. The filter is the first line of defence and a re-entrancy guard is the second; handler.stats["internal_filtered"] and ["reentrant_drops"] show each firing.

When forwarding is enabled, the library's own records do not inherit your application's metadata. They are routed to:

This mirrors a real Universal Forwarder, which ships its own splunkd.log and metrics.log to _internal rather than into whichever index the data it forwards belongs to. It keeps sender diagnostics out of your application searches, off your application index's retention, and in the place a Splunk admin already looks for forwarder health. Set internal_index=None to send them to the same index as everything else.

What works so far

• S2S v4 handshake
• Compact-format event encoding
• Multiple channels — each distinct (source, host, sourcetype) — on one connection
• Single-event and high-volume multi-event sessions
• Batching (send() with a list, the logging handler, and CLI stdin all coalesce)
• Explicit error types for handshake and connection failures
• s2s-fwd CLI, including streamed and batched stdin
• TLS, including mutual TLS and certificate verification
• Opt-in reconnection and retry (see "Reconnection and retry")
• Auto load balancing across several indexers (see "Load balancing")
• asyncio client (s2s_forwarder.aio.AsyncS2SClient)
• logging.Handler for Python applications (see "Python logging")
• Library-internal operational logging, silent by default (see "Library logging")

Not implemented yet

• Acknowledgements — see "Delivery semantics"
• Compression (compression=1)

Delivery semantics

The library currently negotiates ack=0, matching a default-configured forwarder, so there is no confirmation that any given event was durably received. A raised S2SConnectionError tells you something broke, but events already "sent" before it may or may not have landed.

Do not use this where losing any events is unacceptable.

Reconnection and retry

Opt in with retry=N (or --retry N) to reconnect and re-send after a connection failure, with exponential backoff:

client = S2SClient("splunk.example.com", retry=10, retry_backoff=1.0)
...
print(client.stats)   # {'sent': 812, 'reconnects': 2, 'resends': 2}

Retry is off by default because, with no acknowledgement, a failure gives no indication whether the bytes already arrived — so a re-sent event may be duplicated. Enable it when keeping a long-running stream up matters more than avoiding duplicates.

Load balancing across indexers

Pass more than one indexer and the client spreads across them, the way a forwarder does. hosts is polymorphic — you never have to normalise it:

S2SClient("idx1")                          # one indexer
S2SClient("idx1:9998")                     # with a port
S2SClient("idx1,idx2")                     # comma-separated
S2SClient(["idx1", "idx2:9998"])           # a list

s2s-fwd --host idx1 --host idx2:9998 --index main "event"
s2s-fwd --host idx1,idx2 --index main "event"      # equivalent

Every auto_lb_frequency seconds (default 30, matching Splunk's own autoLBFrequency) the client closes its connection gracefully and opens a new one. Selection is random but never re-picks the indexer already in use, so each rotation actually moves; lb_strategy="round_robin" gives deterministic cycling instead.

Event breaking: what Splunk does with your data

This trips people up and is not a bug in this library — a real Universal Forwarder behaves identically.

Splunk decides event boundaries itself, at the indexer, using its documented props.conf defaults: it splits on newlines, then re-merges those lines into events (SHOULD_LINEMERGE=true), starting a new event only before a line carrying a recognisable date (BREAK_ONLY_BEFORE_DATE=true), up to MAX_EVENTS=256 lines.

So if you stream content with no recognisable dates, everything you send is delivered completely and intact — but it may be indexed as a few large merged events rather than one event per line.

If you want one event per line, configure it receiver-side in props.conf for your sourcetype:

[my_sourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)

That is what Splunk itself recommends for well-structured data.

Requirements

Python 3.9+ (ships with RHEL 9). No runtime dependencies beyond click for the CLI.

Disclaimer

This library implements the Splunk-to-Splunk (S2S) protocol v4 based entirely on packet capture analysis and publicly available third-party documentation. No Splunk source code was used or referenced. Splunk (Cisco) does not publish the S2S specification and does not support third-party use of this protocol.

This library is intended for development, testing, and research use. It does not implement the S2S acknowledgement mechanism, consistent with the default configuration of the majority of enterprise Splunk deployments.

Tested against Splunk Enterprise 9.0.5 (plaintext and TLS on TCP 9997) and against Splunk Cloud over mutual TLS with full certificate and hostname verification, both negotiating v4=true / pl=6. Protocol behaviour may change in future Splunk releases without notice.

Development

pip install -e ".[dev]"
pytest

License

Apache 2.0 — see LICENSE.