pcapy-ng 2.1.0

What is Pcapy-NG?

Pcapy-NG is a Python extension module that lets Python programs use the libpcap packet-capture library. It is a maintained replacement for Pcapy, which is no longer maintained and stopped working on Python 3.10 (issue). The classic Pcapy API — open_live, open_offline, loop, next, setfilter, dump_open and the rest — is unchanged, so existing Pcapy code keeps working; it builds on Linux, macOS and Windows (libpcap / Npcap), on CPython 3.9 and newer.

On top of that it provides an optional set of capture primitives for programs where per-packet Python work is the bottleneck: loop_filtered classifies packets in C and invokes the callback only for the classes you admit, loop_to_buffer writes admitted packets into a buffer you own with the GIL released, next_batch returns many packets per call, filtered_stats exposes live counters, and set_fanout joins a live handle to a Linux PACKET_FANOUT group so several handles can share one interface's traffic. The classifier can track flow heads (the first N packets of each flow), match against arbitrarily large IPv4 address sets, and run a built-in security profile that tags DNS, HTTP, TLS/QUIC handshakes and TCP SYNs. All of it is opt-in — ignore it and pcapy-ng behaves exactly like Pcapy.

Contents

• Install
• Quick start
• loop_filtered: filtering in C
  • Why not just a BPF filter?
  • Classes and the admit mask
  • Flow heads: the start of every connection
  • Address-set matching
  • loop_to_buffer: filling a shared buffer
  • Live counters and non-Ethernet links
• Built-in security profile
• next_batch: many packets per call
• set_fanout: several handles on one interface
• API reference
• Examples
• Building & compatibility
• License & credits

Install

pip install pcapy-ng

Releases from 2.0.1 on ship an sdist plus Linux (x86_64/aarch64, glibc and musl) and macOS (x86_64/arm64) wheels for CPython 3.9–3.14; those wheels vendor libpcap, so nothing is compiled at install time. Everywhere else — Windows, BSD, and on 2.0.0 and older — pip falls back to the sdist, which needs a C++ compiler and libpcap headers (libpcap-dev / libpcap-devel; on Windows the Npcap SDK, with WPDPACK_BASE pointing at it). To build a checkout:

pip install .

Python 2.7 is not a supported target: python_requires is >=3.9, nothing in CI tests it and no wheel is built for it. The C sources do still carry their Python 2 code paths, so a 2.7 build can be driven the old way — python setup.py install, since the PEP 517 build needs setuptools >= 61, which is Python 3 only — but that is unsupported and untested.

Quick start

import pcapy

print(pcapy.findalldevs())                       # list devices

cap = pcapy.open_live("eth0", 65535, True, 100)  # device, snaplen, promiscuous, timeout(ms)
cap.setfilter("tcp port 80")                     # optional BPF filter (kernel-side)

while True:
    hdr, data = cap.next()
    if hdr is None:
        break
    print(hdr.getts(), hdr.getlen(), len(data))

cap.loop(-1, lambda hdr, data: print(len(data))) # ...or use a callback

cap = pcapy.open_offline("capture.pcap")         # read a saved capture
dumper = cap.dump_open("out.pcap")               # write packets out

hdr is a Pkthdr with getts() → (seconds, microseconds), getlen() (wire length) and getcaplen() (captured length). This is the original Pcapy surface, unchanged.

loop_filtered: filtering in C

cap.loop_filtered(cnt, callback, admit_mask=7, addr_set=b"", flow_cutoff=0)

Everything after callback is optional and can be passed positionally or by keyword.

loop_filtered classifies each packet in C, hands your callback(hdr, data, cls) only the packets you asked for, and drops the rest without building a Python object for them. On a busy link where most traffic is bulk you don't care about, that per-packet object + callback cost is the thing that makes a pure-Python sniffer fall behind; this avoids paying it.

Why not just a BPF filter?

Fair question — and for a lot of cases the answer is "use BPF." setfilter("udp port 53") runs in the kernel and is the right tool for stateless port/protocol matching. You can (and should) keep using it; loop_filtered runs after it.

loop_filtered exists for the two things a BPF program can't do, because BPF is stateless and size-limited:

1. Flow state — "give me the first N packets of each connection, drop the rest." BPF can't track flows. This is how you grab the start of every conversation (where the useful bytes are) while ignoring the bulk that follows.
2. Large set membership — "deliver packets whose source or destination is in this set of addresses," where the set has thousands or millions of entries. That won't fit in a BPF program; here it's an average O(1) hash lookup.

If you only need stateless port/proto matching, use BPF. Reach for loop_filtered when you need flow state or set membership (and you can combine both with a BPF prefilter).

Classes and the admit mask

Each delivered packet comes with a small integer cls saying why it was kept:

Classes are mutually exclusive — each packet gets exactly one cls and increments exactly one counter. When a packet could be more than one, precedence is SET_MATCH > FLOW_HEAD > OTHER (a flow's first packet that is also a set hit is reported as SET_MATCH).

admit_mask is a bitmask over the class number — bit cls admits that class; the rest are dropped in C. Default 7 (= 1<<0 | 1<<1 | 1<<2) admits all three, so a bare loop_filtered behaves like loop but tagged + counted. To keep only flow heads and set matches, drop OTHER:

OTHER, FLOW_HEAD, SET_MATCH = 0, 1, 2
admit = (1 << FLOW_HEAD) | (1 << SET_MATCH)
stats = cap.loop_filtered(-1, my_cb, admit, addr_set, flow_cutoff=3)
# stats -> (admitted, dropped, n_other, n_flow_head, n_set_match)

The return value is (admitted, dropped, <count per class>). The per-class counts are independent of admit_mask, so admit_mask=0 profiles your traffic with no Python work at all.

Flow heads: the start of every connection

flow_cutoff=N delivers the first N packets of each flow (TCP and UDP) as FLOW_HEAD and lets you ignore the rest. The opening packets of a connection are where the interesting metadata lives — the requested hostname inside a TLS handshake, the first request line, the greeting of a text protocol — so this captures that and sheds the bulk payload after it.

def on_head(hdr, data, cls):
    # data is the start of a connection; parse out whatever you need (e.g. the TLS server name)
    print("connection start:", len(data), "bytes")
cap.loop_filtered(-1, on_head, (1 << 1), flow_cutoff=3)   # FLOW_HEAD only

What "flow" means here. A flow is keyed by the directional 5-tuple (src, dst, src_port, dst_port, protocol) — so the two directions of a connection are two flows, and flow_cutoff=N yields up to N packets per direction. There is no TCP-state tracking (it counts packets per key) and no time-based expiry; the flow table is a fixed ~12 MB hash (about 1M entries, allocated only when flow_cutoff > 0) that evicts the least-recently-seen key on collision — so memory is bounded even under high-cardinality UDP, at the cost of an occasional re-count when a long-idle flow is evicted. Eviction can only make a flow look new again (an extra head), never hide one: the slot stores the full 64-bit key of the 5-tuple, and that key is an avalanching hash, so two different flows sharing one counter — the only way a real flow head could be missed — is a ~2⁻⁶⁴ event rather than a property of the fold. IP fragments are not reassembled (non-first fragments aren't matched as flow heads).

See examples/04_flow_heads.py.

Address-set matching

addr_set is a packed list of IPv4 addresses (4 bytes each, network order). Any packet whose source or destination is in it is delivered as SET_MATCH, regardless of port or protocol — a fast way to watch for traffic to/from a set of hosts (an allowlist, a blocklist, an asset inventory…). The set can be huge.

IPv4 only. The classifier operates on IPv4; IPv6 packets are always OTHER and never match addr_set. (setfilter("ip6 ...") still works for plain IPv6 capture.)

import socket
watch = ["198.51.100.10", "203.0.113.7"]
addr_set = b"".join(socket.inet_aton(ip) for ip in watch)
cap.loop_filtered(-1, on_match, (1 << 2), addr_set)       # SET_MATCH only

loop_to_buffer: filling a shared buffer

Same classification, but instead of calling Python per packet it writes the admitted packets into a writable buffer you provide, with the GIL released for the whole loop. Pair it with multiprocessing.shared_memory and a pool of workers when one thread isn't enough.

from multiprocessing import shared_memory
shm = shared_memory.SharedMemory(create=True, size=64 * 1024 * 1024)
res = cap.loop_to_buffer(-1, shm.buf, addr_set=addr_set, flow_cutoff=3)
# res -> (written, dropped, overflow, bytes_used, <count per class>)

Each record is [u32 caplen little-endian][u8 class][caplen bytes], written back-to-back with no padding/alignment. The only metadata stored is the caplen and the class — no timestamp and no wire length; if you need those, use loop_filtered (its callback gets the full Pkthdr). When the buffer fills, extra packets are counted in overflow instead of overrunning it. See examples/05_shared_memory_ring.py.

Live counters and non-Ethernet links

filtered_stats() returns the current run's (admitted, dropped, <per class>) counters and is safe to read from another thread while the loop runs.

loop_filtered needs to know where the IPv4 header starts, via l2_offset: 14 for Ethernet (default), 16 for Linux "cooked" capture (DLT_LINUX_SLL, the any device), 0 for raw IP (DLT_RAW). VLAN tags are skipped automatically. Derive it from cap.datalink():

OFFSETS = {pcapy.DLT_EN10MB: 14, pcapy.DLT_LINUX_SLL: 16, pcapy.DLT_RAW: 0}
cap.loop_filtered(-1, cb, l2_offset=OFFSETS.get(cap.datalink(), 14))

Built-in security profile

loop_filtered(..., profile=1) swaps the generic classifier for a built-in one aimed at security/traffic monitoring — it tags DNS, HTTP, TLS/QUIC handshakes, TCP SYNs and address-set hits in a single pass, configurable via dns_port, dpi_ports and tls_ports. It exists so tools like Maltrail can do their triage in C; if you're not building that kind of tool, ignore it and stay with the generic classes above.

Two numbers, two jobs — this is the part to get right:

• The cls id delivered to your callback is 100 + index (so it never collides with the generic 0/1/2).
• admit_mask is a bitmask over the index, not the id. To admit DNS + ADDR you write (1 << 0) | (1 << 3) — not 1 << 100. The default mask for this profile is 59 (= 1<<0 | 1<<1 | 1<<3 | 1<<4 | 1<<5), i.e. everything except OTHER (index 2).

The per-class counts in the return tuple are in index order (admitted, dropped, dns, http, other, addr, handshake, syn).

next_batch: many packets per call

next_batch(max_n) reads up to max_n packets in a single call and returns them as (packet_bytes, packed_meta): packet_bytes is every packet concatenated, and packed_meta is an array of fixed 16-byte records — one per packet, four native-endian uint32s (sec, usec, offset, caplen) — where offset/caplen slice a packet out of packet_bytes. An empty packed_meta means EOF (offline) or timeout (live). It crosses the C↔Python boundary once per batch instead of once per packet, and unlike loop_filtered it does no classification — you get everything, with timestamps.

import struct
pkts, meta = cap.next_batch(1024)
for i in range(0, len(meta), 16):
    sec, usec, off, caplen = struct.unpack_from("=IIII", meta, i)
    data = pkts[off:off + caplen]

set_fanout: several handles on one interface

One capture socket is one thread's worth of work. On Linux, set_fanout(group_id, fanout_type) joins a live, activated handle to a kernel PACKET_FANOUT group: open N handles on the same interface, put them all in the same group, read each in its own thread or process, and the kernel spreads the interface's packets across them without duplicating any. With the default PACKET_FANOUT_HASH every flow stays on one handle, so per-flow state stays consistent.

cap = pcapy.open_live(dev, 65535, True, 100)
cap.set_fanout(0x4711, pcapy.PACKET_FANOUT_HASH)

Constants: PACKET_FANOUT_HASH, PACKET_FANOUT_LB, PACKET_FANOUT_CPU, PACKET_FANOUT_ROLLOVER, PACKET_FANOUT_RND. Linux only, live handles only; anything else raises. See examples/07_fanout_scale.py.

API reference

Module: open_live(dev, snaplen, promisc, timeout_ms), open_offline(path), create(dev), findalldevs(), lookupdev(), compile(...), the DLT_*, PCAP_D_* and PACKET_FANOUT_* constants, and the PcapError / BPFError exceptions.

Reader (classic): next() → (hdr, data), loop(cnt, cb), dispatch(cnt, cb), setfilter(bpf), datalink(), getnet(), getmask(), getnonblock()/setnonblock(), setdirection(...), stats() → (recv, drop, ifdrop), dump_open(path), sendpacket(data), getfd(), close(). Usable as a context manager.

Reader (unactivated, from create()): set_snaplen(n), set_promisc(bool), set_timeout(ms), set_buffer_size(n), set_rfmon(bool) (monitor mode, where supported), then activate().

Reader (filtering):

cnt = -1/0 means "until EOF (offline) or forever (live)".

Code that must also run against stock Pcapy can probe with hasattr(reader, "loop_filtered") and fall back to loop() / next().

Examples

Runnable, self-contained scripts in examples/ — each takes a device (live) or a .pcap (offline) as its first argument:

Building & compatibility

Builds from source on CPython 3.9–3.14, on Linux, macOS and Windows (libpcap / Npcap); all it needs is a C++ compiler and the libpcap headers. CI builds and tests every commit on CPython 3.9–3.14 (Linux and macOS). Wheels are produced for those two platforms only, with cibuildwheel, and vendor libpcap, so installing from a wheel needs no compiler or libpcap-dev. There are no Windows wheels — on Windows pip builds from the sdist against the Npcap SDK.

set_fanout is Linux-only; the classifier used by loop_filtered / loop_to_buffer is IPv4-only (IPv6 packets are always OTHER). Everything else is portable.

License & credits

Apache Software License — see LICENSE. Pcapy-NG is maintained by Miroslav Stampar (contact@sekuripy.hr) and builds on the original Pcapy by CORE Security. Bug reports, patches and suggestions welcome.