network_core

A generic Go + Python library that extracts flow-level features from network captures. Designed to survive high-speed captures with real packet loss (including capture-side truncation and non-IP tail noise).

Author: rushi. Contributors: peter, levi.

What v0.8.0 changes

Schema bump (SchemaVersion=3). mergedFlows.csv gains five columns. Existing columns keep their positions; readers of the pre-v0.8.0 5-column schema continue to work if they use csv.DictReader or row.get(); strict-positional readers of columns [0..4] see the same values as before.

Full v0.8.0 header:

FlowId,FiveTuple,StartTime,EndTime,SNI,ALPN,DNSName,DNSDelta,DNSCandidates,SNIReason,ImplausibleTsSkipped

Column semantics

SNI

The hostname parsed from the flow's ClientHello (TLS on TCP, or QUIC v1 Initial). When the extractor could not observe a ClientHello but a prior flow with a live carry-map entry did, the inherited SNI lands here (see SNIReason=carry).

Value is nan when the extractor could not attribute a hostname — see SNIReason for the specific reason.

ALPN — CLIENT-OFFERED from THIS flow's ClientHello

The value is the FIRST-OFFERED ALPN entry from the ClientHello THIS flow parsed, or carried alongside the SNI from an earlier flow via §2 continuity carry / §3 QUIC CID linkage. nan if no CH was parsed and no carry succeeded.

• TCP: tls_sni.ALPN[0] from the parsed CH.
• QUIC: quic_sni.ALPN[0] from the parsed QUIC-CRYPTO-carried CH.
• Carry: inherited from the prior flow's SNIInfo.

nan explicitly means "this flow's CH didn't parse and no carry supplied a value". It does NOT mean the client didn't offer an ALPN extension — some clients omit it entirely — but from the wire we cannot tell those cases apart.

The column is a client-side observation, not a negotiated protocol. For TLS 1.2 the server's negotiated choice is in the (cleartext) ServerHello; for TLS 1.3 it moves into EncryptedExtensions and is NOT recoverable. If you need the definitely-negotiated protocol you must decrypt the handshake — this extractor doesn't.

Behavior change vs 0.7.0 (rc6): removed the ServerHello cross-flow inference, the plaintext HTTP-version fallback, and the QUIC port-443 → h3 heuristic. All three could put a value in the column that the flow's own CH never offered. See CHANGELOG §Behavior changes for details.

DNSName / DNSDelta / DNSCandidates

Populated by the finalize step, not the Go extractor. The Go pass writes mergedFlows.csv with these columns empty and concurrently appends valid DNS observations to a temp dns_obs.csv. The Python finalize step (network_core.finalize) runs a polars join_asof at EOF, streams the enriched Flows.csv to a temp path, atomically renames it onto mergedFlows.csv, and deletes dns_obs.csv (--keep-dns-obs preserves it for debug).

Consumer contract: if you read mergedFlows.csv AFTER the finalize step (the normal case), these columns are populated. Raw Go-only output (e.g. during a live capture before finalize has run) has them EMPTY strings.

• DNSName — the LATEST canonical A/AAAA name for this flow's server IP, answered BEFORE StartTime, from the full pcap DNS scan. If no answer exists before start, the EARLIEST answer after start (with a positive DNSDelta).
• DNSDelta — signed seconds (answer_ts − StartTime). Negative when the answer preceded flow start (fresh). Positive when the answer arrived after (post-hoc; consumers decide whether to trust).
• DNSCandidates — all distinct canonicals mapped to the server IP across the WHOLE pcap, |-joined, chronological.

SNIReason — one of 8 machine-friendly codes

ImplausibleTsSkipped

Per-flow count of packets dropped by the timestamp-sanity gate (implausible ts: pre-2020 or > now+24h). Non-zero flows are flagged in the diagnostics sidecar.

The diagnostics sidecar

mergedFlows.diagnostics.json sits next to mergedFlows.csv. Every key listed below is REQUIRED — consumers may reject a diagnostics blob with UN-listed keys.

• session_start_utc, session_end_utc
• capture_span_from_plausible_packets_s — capture length as measured by plausible-ts packets only. The yt_7 truncated tail never widens this.
• n_pcap_records_read, n_skipped_non_ip, n_skipped_implausible_ts, n_dropped_all_implausible_flows
• n_dns_answers_accepted, n_dns_answers_rejected_qr, n_dns_answers_rejected_source, n_dns_answers_rejected_txnid
• n_dns_stale_seconds_hist — [0-60, 60-300, 300-3600, 3600-inf] age of the DNS answer that produced each flow's DNSName. Written by the Python finalize.
• n_dns_post_hoc — count of flows whose ONLY DNS answer arrived after StartTime (positive DNSDelta).
• schema_version — matches MergedFlowsSchemaVersion.
• producer_version — the network_core binary's version string (network_core --version). Brad's oracle reads this to identify which extractor produced the artifact.
• flow_pathology — array of per-flow finer-grained failure details. Each entry: {flow_id, reason, specific_detail, extra}. reason matches the coarse SNIReason column; specific_detail names WHICH concrete pathology hit (gap_in_clienthello, short_header_no_length_context, etc.).

CLI

network_core <pcap> <flows.csv> <packets.csv> [dns_obs.csv] [diagnostics.json]
network_core --version

Default output paths (when the last two args are omitted): <flows>.dns_obs.csv and <flows>.diagnostics.json.

Pass - as <pcap> to read a pcap or pcapng stream from stdin. Useful for piping a decompressor or a remote fetch without staging the file to disk:

zstdcat capture.pcap.zst | network_core - flows.csv packets.csv

Stdin mode is bit-for-bit equivalent to file input; the reader detects pcap vs pcapng from the first four magic bytes.

Python API

import network_core
print(network_core.__version__)  # "0.8.0"

# read a finalized mergedFlows.csv
from network_core import get_connections_from_csv
conns = get_connections_from_csv("mergedPackets.csv", "mergedFlows.csv")

CHANGELOG

See CHANGELOG.md.