fast-ccsds-parser

Project page · 日本語のドキュメントはこちら

Fast, dependency-free parsing of CCSDS Space Packets (CCSDS 133.0-B-2) for the modern Python data stack. Turn raw telemetry dumps into dicts and structured objects in one pass: primary headers, packet boundaries, and payloads, with a pure standard-library core that installs anywhere (pip install and go).

Built for small-satellite teams, university labs, and data engineers who just want their packets parsed, without pulling a decoding framework.

Install

pip install fast-ccsds-parser

The core has no dependencies. For columnar export, add an extra (arrow, polars, pandas, or all):

pip install "fast-ccsds-parser[pandas]"

or from source:

pip install git+https://github.com/daikiito-dk/fast-ccsds-parser.git

Quickstart

from fast_ccsds_parser import parse_primary_header, iter_file

# One header, as a plain dict:
header = parse_primary_header(raw_bytes)
# {'version': 0, 'type': 0, 'secondary_header_flag': 1, 'apid': 42,
#  'sequence_flags': 3, 'sequence_count': 1337, 'packet_length': 255}

# A whole ground-station dump, streamed packet by packet (constant memory):
for pkt in iter_file("dump_20261010.pkts"):
    print(pkt.header.apid, pkt.header.sequence_count, len(pkt.payload))

# In-memory buffers work the same way:
packets = [p.to_dict() for p in iter_packets(buffer)]

Need a whole dump as a DataFrame? Scan it column by column instead of building one dict per packet (needs fast-ccsds-parser[pandas]):

from fast_ccsds_parser import scan_file, write_parquet

df = scan_file("dump_20261010.pkts").to_pandas()
write_parquet("dump_20261010.pkts", "dump_20261010.parquet")

What it parses

The 6-byte CCSDS primary header (all fields big-endian):

bit:   0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
0-1    | version  | T | S |                APID                       |
       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
2-3    | seq flags |        sequence count / packet name               |
       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
4-5    |                     packet length                            |
       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+

API

SpacePacket carries header (a PrimaryHeader), payload (the data field bytes), raw (the complete packet bytes), and a to_dict() that flattens everything into a single dict.

Columnar output

PacketTable stores header fields in typed array.array columns and all payloads in one contiguous buffer, so memory stays close to the input size. Columns: offset, version, type, secondary_header_flag, apid, sequence_flags, sequence_count, packet_length, and payload.

table = scan_file("dump_20261010.pkts")   # or scan_bytes(buffer)
table.apid                                # array.array, one entry per packet
table.payload(0)                          # payload bytes of packet 0

arrow = table.to_arrow()                  # pyarrow.Table (header columns are zero-copy)
df = table.to_polars()                    # polars.DataFrame
df = table.to_pandas()                    # pandas.DataFrame (payload stays Arrow-backed)

Export needs the matching extra: to_arrow() and write_parquet() need pyarrow (fast-ccsds-parser[arrow]), to_polars() and to_pandas() add polars / pandas (the polars and pandas extras). Scanning itself works with the standard library alone; pass payload=False to keep only the header columns and use much less memory.

Semantics to know

• packet_length is zero-based. The data field is packet_length + 1 bytes, so a total packet is 6 + packet_length + 1 bytes. This is the number-one source of off-by-one bugs when handling CCSDS data, and the parser handles it for you.
• payload = the full data field. Per CCSDS 133.0-B-2 the data field is an optional secondary header followed by user data; the payload bytes are returned as-is. Check header.has_secondary_header to know whether a mission-specific secondary header is in front. Decoding those layouts is out of scope for the structural parser (see the roadmap).
• Idle packets (APID = 0x7FF) are yielded like any other packet by default; pass skip_idle=True to filter them out.
• Strict mode (default) raises InvalidPacketError when the packet version is not 0, which usually means your byte stream is misaligned. Pass strict=False to parse anyway.
• All errors derive from FastCCSDSError (a ValueError).

Related projects

Several mature Python libraries already handle CCSDS space packets, and all of them can split packets and read primary headers without a mission packet definition. Pick the one that fits your job:

If you need to decode packet contents against a mission packet database, use ccsdspy or space-packet-parser; both are mature and used on flight missions. If you build or parse ECSS PUS / CFDP traffic, use spacepackets. fast-ccsds-parser focuses on getting large packet dumps and live streams into data-analysis tools quickly and with little memory.

Performance

benchmarks/bench.py measures packet throughput and peak memory, running each case in a fresh process. Results for 1,000,000 synthetic 32-byte packets (32 MB), Python 3.12, Apple M4, pandas 3.0.6, polars 2.0.0, pyarrow 26.0.0:

(ccsdspy 2.0.1 and space_packet_parser 6.2.0 rows were measured in the same setup.)

What this shows:

• The columnar scan is the fastest way to a DataFrame: to_pandas() is about 5x faster than building from per-packet dicts and uses about a quarter of the memory (188 MB vs 677 MB for 32 MB of input).
• Streaming with iter_file uses the least memory of all cases.
• Iterating raw packet bytes is about 2x faster in ccsdspy and space_packet_parser, because iter_file also decodes every header and copies each payload. This is a known optimization target.

Run it yourself:

pip install -e ".[all]" ccsdspy space_packet_parser
python benchmarks/bench.py                  # synthetic data
python benchmarks/bench.py path/to/dump.bin # your own dump

Roadmap

• Phase 2a (done in 0.2.0): columnar output (scan_file(), to_arrow(), to_polars(), to_pandas(), write_parquet()) that keeps memory close to the input size, with Arrow / Polars / pandas as optional extras.
• Phase 2b: decoding CCSDS time codes (CUC / CDS) from secondary headers into timestamps.
• Phase 2c: asyncio parsing of live TCP socket / WebSocket streams (WebSocket as an optional extra).
• Phase 3: move the hot loops that benchmarks show are slow to Rust (PyO3), keeping the Python API unchanged.

Development

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest            # offline test suite
ruff check .      # lint

Tests marked network additionally parse public, real-world packet dumps downloaded on the fly; they skip automatically when offline (CI runs them).

Releasing (PyPI)

Publishing uses trusted publishing (no API tokens). One-time setup is described in .github/workflows/publish-testpypi.yml (the file name is historical; it publishes to PyPI). Bump __version__ in src/fast_ccsds_parser/__init__.py, add a CHANGELOG entry, then:

git tag vX.Y.Z
git push origin vX.Y.Z

The workflow builds sdist + wheel, runs twine check, and uploads to PyPI.

License

MIT