zelos-packet

Rust-first packet capture and decode for the Zelos ecosystem: live capture through a bundled helper process (AF_PACKET on Linux, /dev/bpf on macOS), plus pcap/pcapng offline decode, emitting packet and capture-statistics events into a Zelos trace.

The helper links libpcap statically, so the wheel has no runtime dependency on a system libpcap; the extension module links none at all. Live capture needs a one-time grant (below); offline decode needs none.

pip install zelos-packet

Offline

import zelos_packet, zelos_sdk

with zelos_sdk.TraceWriter("capture.trz"):
    d = zelos_packet.PacketDecoder("dump")
    d.convert_file("dump.pcap")     # or d.decode_stream(raw_bytes)
    # rows land at Packet.dump/packets.<field>

convert_file(path, realtime=True) replays at the capture's own pace, shifted so the first packet lands at now. Both read gzip or zstd compressed captures directly, detected by magic bytes, decompressed as a stream. A compressed file that stops decoding part-way (cut short, corrupt) keeps every packet before the damage, then raises CaptureTruncatedError.

Live

# The helper streams to this agent; rows are read back from there, never from
# a local TraceWriter. Defaults to $ZELOS_AGENT_URL, else localhost:2300.
cap = zelos_packet.PacketCapture("en0", snaplen=128, agent_url="http://localhost:2300")
cap.start()                        # rows land at Packet.en0/packets.<field>
print(cap.stats().kernel_drops, cap.metrics().emit_stall_ms)
cap.stop()

sudo "$(command -v python3)" -m zelos_packet install-helper   # once per machine
python3 -m zelos_packet status                                # would capture work now?

Then pick the group up. Linux stamps a session's group set at login (initgroups), so log out and back in. macOS resolves membership per process, so a fresh terminal is enough. Either way, anything already running (this shell, a running Zelos app) keeps the group set it started with.

Platform	What install-helper does	What the group grants
Linux	Installs the helper to a root-owned path with cap_net_raw=ep (never CAP_NET_ADMIN) and creates/joins zelos-packet	The ability to run a capture. The helper opens the socket, drops every capability, verifies the drop from /proc/self/status, and only then reads. It never hands out the socket, so not frame injection.
macOS	Installs a ChmodBPF-style boot daemon putting /dev/bpf* in access_bpf	Capture and sending arbitrary frames — a bpf device is opened read-write and capture needs the write side, exactly as for Wireshark.

zelos_packet.permission_remediation() returns the same text a failed open raises inside CapturePermissionError.

One capture path: the helper

zelos-packet-helper runs every live capture on both platforms. It opens the handle, drops every capability it holds, and streams decoded rows straight to an agent; PacketCapture supervises that process and nothing else. No packet enters the calling process, so capture privilege never lands on the interpreter and no capture-capable descriptor is handed out.

• source= is a hard error. Rows are produced behind the helper's own SDK connection, so nothing would reach a source passed here. Use agent_url=.
• An agent must be reachable for rows to land anywhere. A local TraceWriter sees nothing.
• Constructing is not starting. PacketCapture(...) validates the interface name, resolves the agent exclusion, and locates a runnable helper, so a typo, an unresolvable agent host, or a missing helper raises there. The permission verdict comes from start(), where the helper opens the handle. (zelos-can starts from its constructor; this does not.)

Catalog layout

Every capture writes into ONE trace source, Packet, and is told apart by its own event-name prefix, so a field is addressed Packet.en0/packets.src_ip:

Packet
└── en0                      # the capture's `name`; defaults to the interface
    ├── packets              # zelos.packet.v1 — one row per frame
    ├── stats                # zelos.packet.stats.v1 — one row per interval
    ├── someip               # one row per SOME/IP message (SD included)
    ├── someip_sd            # one row per SOME/IP-SD entry
    ├── doip                 # one row per DoIP message
    └── pdu                  # one row per SoAd PDU (ARXML PDU sockets)
SOMEIP                       # with arxml=: ARXML-decoded SOME/IP payloads
└── <Service>/<Element>      # one event per event, method, field
PDU                          # with arxml=: ARXML-decoded SoAd PDUs
└── <PDU>                    # one event per SoAd signal PDU, by short-name

The automotive rows appear only for traffic on the ports given, all off by default: SOME/IP-SD on someip_sd_port (customarily 30490), DoIP on doip_port (customarily 13400), SOME/IP on the ports SD offers, the ARXML names, or someip_ports lists. The pkt row of such a packet reads SOME/IP, SOME/IP-SD or DoIP in proto; with nothing given, pkt rows are as before.

Topic	Behavior
Messages	several per datagram or TCP segment, walked by Length; a TCP direction's bytes carry over between segments, a sequence gap resyncs
SOME/IP-TP	segments are rows; the reassembled original is one more row (TP flag cleared, tp_offset null); ascending order only, a gap drops the message, 1 MiB cap
instance_id	from SD offers (endpoint) and acks (multicast), and ARXML provided instances; null when unknown
arxml=	R19-11 / R4.2 SOME/IP service interface deployments; payloads per PRS SOME/IP serialization; one event per method and field with message_type, session_id, client_id, return_code, instance_id (fields add method_id)
Leaves	nested members join with _; fixed arrays up to 16 flatten to name_<i>; longer and dynamic arrays are a bytes column plus name_count; unions add name_selector
SoAd PDUs	PDUs behind a SoAd header on an ARXML PDU socket decode through the CAN PDU pipeline into PDU/<PDU>, e.g. PDU/DemoPdu: header_id, src_ip, src_port, dst_ip, dst_port, the signals, then e2e_status (E2E profiles 1, 2, 4, 5, 6, 7, 11, 22) and secoc (unverified); contained and dynamic PDUs as <PDU>/<PDU>; two SoAd PDUs with one short-name are a load error; pdu_signal_rows, pdu_decode_errors
Not decoded	a payload that does not fit its type (counted in someip_malformed); ids the ARXML lacks (someip_unknown, warned once); labels on a float-typed PDU signal

Pass name= to override the prefix; it goes through sanitize_name(), which collapses catalog separators (., :, @, /) to _, so a VLAN device eth0.100 lands at Packet.eth0_100/packets.

Two captures may share one source as long as their names differ. Event registration is strict-create: a second capture registering a name already on that source raises rather than merging, which keeps two captures from silently interleaving into one table.

API

Symbol	Purpose
PacketDecoder(...)	Bring-your-own-bytes: decode_frame, decode_stream, convert_file, push_stats, register_schemas, metrics, flush
PacketCapture(...)	Live handle: start, stop, stats, metrics, interface, link_type, is_active, error; also a context manager
list_interfaces()	InterfaceInfo per NIC — name, index, up/running/loopback, addresses, MAC
permission_remediation()	Copy-pasteable privilege fix for this platform
capture_supported()	False outside Linux/macOS; offline decode still works
sanitize_name(name)	The event-prefix rule, shared with the packet extension
CapturePermissionError	PermissionError subclass, message embeds the remediation
InterfaceNotFoundError	ValueError subclass for an unknown interface name

The frame column

frame is the source of truth every decoded column is checked against, so both entry points populate it by default (log_frames=True), clipped to stored_frame_bytes=256 bytes. stored_frame_bytes=None stores every captured byte; log_frames=False drops the column.

Knob	Applies to	Effect
snaplen	live capture only	Bytes the kernel copies out. Uncaptured bytes are dissected by nobody; the shortfall shows up as orig_len > cap_len with truncated set.
stored_frame_bytes	live + offline	Bytes stored in frame. Dissection always reads the full captured bytes, so this changes storage and nothing else.

orig_len / cap_len / truncated describe capture truncation only. Storage truncation is len(frame) < cap_len; the frozen schema has no column for it.

Drop policy

While the sink is consuming, userspace blocks rather than dropping. Every emit is the SDK's normal send path, never try_send, so a full router channel stops the read loop, the kernel ring fills, and the kernel drops where it can be counted.

kernel_drops	decode_stall_ms	Meaning
0	high	Trace store is backpressuring
climbing	~0	Line-rate overload; the read loop is the bottleneck

Metrics.emit_stall_ms is the sink-only slice of decode_stall_ms; the difference is dissection cost.

Where that guarantee stops. Backpressure only exists while something is pulling. If the helper's gRPC publish stream to the agent breaks, the router's publisher subscription goes away while decode keeps running: rows are produced and discarded, and kernel_drops stays flat because the kernel handed them over successfully. So kernel_drops == 0 means "the kernel lost nothing", not "the trace is complete". What does move is Metrics.emit_errors / Metrics.flush_errors, and at stop time CaptureStats.tail_abandoned (non-zero = rows the publisher held never reached the agent) and CaptureStats.kernel_stats_errors (non-zero = kernel_packets / kernel_drops are a stale reading, not totals). Check those before treating a capture as complete. There is no counter for rows dropped into a dead subscription mid-run.

Self-traffic exclusion

When the agent streams over the interface being captured, every emitted row makes more captured bytes. At a large snaplen that loop does not converge.

On by default for PacketCapture, derived from agent_url. With neither exclude_agent_addrs nor exclude_agent_port passed, both come from the URL the capture streams to (agent_url, else $ZELOS_AGENT_URL, else $ZELOS_TRACE_FORWARD_URL, else http://localhost:2300), resolving a host name to every address it has — so the localhost default excludes 127.0.0.1 and ::1 both. PacketDecoder takes no default at all: it is handed bytes that already exist, so there is no loop to close and dropping rows out of a user-supplied file would be silent data loss.

You want	Pass
A different endpoint	exclude_agent_addrs=[...] and exclude_agent_port=... — no resolution happens
The agent's traffic captured	exclude_agent_addrs=[]
Nothing else	(nothing)

Either argument alone is an error; the empty list is the one address list a port may be absent from. A host that does not resolve is a construction error, not a fallback to capturing unfiltered.

A live capture compiles the exclusion with pcap_compile and installs it with pcap_setfilter on the helper's handle, so the kernel rejects excluded traffic before a packet is copied out. If pcap_compile cannot express it for the link type, the capture fails to open rather than running unfiltered. Offline decode has no kernel in the path; its userspace AgentFilter matches on ports only, but it is still applied — a pcap being converted loses its agent-endpoint rows unless you pass exclude_agent_addrs=[].

The expression is not recorded in the trace. It is logged at capture start (RUST_LOG=zelos_packet=debug, "compiling the agent filter"), which is the only place to read back what was installed — worth capturing alongside a trace you intend to analyze, because of the three gaps below.

It drops more than the agent port. Closing the IPv4 fragment hole statelessly costs precision, on any port, for the agent addresses only:

Traffic to/from an agent address	Kept?
IPv4 first fragment (offset 0)	kept, unless on the agent port
IPv4 continuation fragment (offset > 0)	dropped, any port
IPv6 with any extension header — Fragment, Hop-by-Hop, Routing, Destination Options, ESP (50) or AH (51)	dropped, any port

So fragmented UDP telemetry to an agent address on port 9999 keeps each datagram's first fragment and loses every continuation, and an agent address reached over IPsec transport mode contributes no packets at all. The idiomatic not (host X and port Y) accepts continuation fragments, and those are what the feedback loop is made of; statelessly, rejecting on the address alone is the only correct answer. There is no counter for it either — pcap_stats's ps_recv counts packets that already passed the kernel filter, so excluded traffic appears in no column. To see what was excluded, capture with exclude_agent_addrs=[] on an interface the agent does not stream over.

Tunnels defeat it. Both the kernel program and the userspace backstop read the outer IP header, so agent traffic inside WireGuard, any other VPN, VXLAN or GRE carries its addresses in the inner header and is re-admitted. Capture the tunnel interface (wg0, utun0, …) instead, where the agent's addresses are the outer ones.

Three or more stacked VLAN tags defeat it. libpcap's host / port primitives do not look past a VLAN tag, so the expression is repeated under vlan and vlan and vlan — untagged, single-tagged and QinQ. Under a third tag the address test reads the wrong offset, does not match, and the enclosing not (…) therefore accepts the frame. The arms stop there because each one multiplies the compiled program, and the worst case already sits close to BSD's 512-instruction BPF_MAXINSNS ceiling.

Development

just develop     # uv sync + maturin develop
just test        # cargo test (all three crates) + pytest + check-static
just lint        # cargo clippy -D warnings
just stub_gen    # regenerate py/zelos_packet/_native.pyi

Live-capture tests are opt-in (ZELOS_PACKET_TEST_LIVE=1, optionally ZELOS_PACKET_TEST_IFACE=<name>) and need a zelos-agent binary, since rows are read back out of a real agent. Supervision and the permission verdict need no privilege on any platform: rs/supervisor.rs and test/test_helper_backend.py both drive a fake helper.

The Rust toolchain is pinned by the nix dev shell; there is deliberately no rust-toolchain file. Release wheels come from the api-py-zelos-packet workflow, or locally from just build-linux-wheels; both build the static libpcap with docker/manylinux/build-libpcap.sh (the one place its version, hash and configure flags live) and gate the artifact with docker/manylinux/check-wheel-static.sh. A bare maturin build elsewhere does neither. RELEASE.md has the release order.