Custom Binary Protocol API

Requirements

• Python 3.8 or higher
• asyncio
• pytest
• matplotlib & seaborn
• psutil

Overview

This project implements a lightweight, configurable binary protocol for client-server communication in Python. Unlike traditional REST APIs, this implementation uses a custom binary protocol for data transfer, resulting in lower latency and reduced network overhead.

Key Features

• Structured Message Format: Define message structures with typed fields
• Efficient Data Transfer: Custom binary protocol minimizes overhead compared to HTTP/REST
• Type-safe Field Handling: Automatic serialization of primitive types
• Asynchronous Communication: Built on asyncio for high-performance I/O operations
• Zero Dependencies: Pure Python implementation with no external requirements
• Bi-directional Communication: Supports full-duplex communication between client and server
• Fixed and Variable Length Fields: Support for both fixed and variable-length data
• Optimized Performance: Fast-path for fixed-length messages

Quick Examples

Message Type Definition

from enum import IntEnum
from binary_protocol.protocol.message_structure import FieldDefinition, MessageStructure

# Define message types
class MessageType(IntEnum):
    SENSOR_DATA = 1
    STATUS = 2
    BENCHMARK = 3
    FIXED_BENCHMARK = 4

# Define fixed-length structure for high-speed sensor data
sensor_data_structure = MessageStructure([
    FieldDefinition("sensor1", float, format_string=">f"),  # 4-byte float
    FieldDefinition("sensor2", float, format_string=">f"),
    FieldDefinition("sensor3", float, format_string=">f"),
    FieldDefinition("sensor4", float, format_string=">f")
])

# Variable-length structure example
status_structure = MessageStructure([
    FieldDefinition("status_code", int, format_string=">I"),
    FieldDefinition("message", str)  # Variable length
])

# Fixed-length benchmark structure
fixed_benchmark_structure = MessageStructure([
    FieldDefinition("value", int, format_string=">I")  # 4-byte integer
])

Protocol Configuration

from binary_protocol import ProtocolConfig

config = ProtocolConfig(
    message_type_enum=MessageType,
    message_structures={
        MessageType.SENSOR_DATA: sensor_data_structure,
        MessageType.STATUS: status_structure,
        MessageType.BENCHMARK: benchmark_structure,
        MessageType.FIXED_BENCHMARK: fixed_benchmark_structure
    },
    header_format=">H",  # 2-byte header for message type
    max_payload_size=10 * 1024 * 1024  # 10MB
)

Benchmarking Features

Available Benchmark Types

1. Fixed Size Benchmark

   • Optimized for consistent message sizes
   • Validates message integrity
   • Uses predefined message templates

2. Variable Size Benchmark

   • Tests multiple message sizes
   • Measures scaling characteristics
   • Configurable size ranges

3. Realistic Benchmark

   • Multi-client simulation
   • Random message sizes
   • Configurable client behavior

Running Benchmarks

python examples/basic_client.py --benchmark [options]

Options:

• --all: Run all benchmark types
• --fixed: Run fixed-size benchmarks
• --variable: Run variable-size benchmarks
• --realistic: Run realistic benchmarks

Output Formats

Console Output

• Detailed benchmark summary tables
• Resource utilization metrics
• Statistical analysis
• Performance comparisons

Generated Files

• JSON result files
• Performance plots (PNG)
• Comparison reports
• Resource usage graphs

Protocol Configuration Options

Field Types and Performance

Security Considerations

1. Encryption: Consider adding TLS for secure communication
2. Authentication: Implement a secure authentication mechanism
3. Input Validation: Validate field contents
4. Rate Limiting: Protect against DoS attacks
5. Message Size Limits: Configure appropriate max_payload_size

Performance Optimizations

When enable_performance_improvements is True (default):

• Uses message pooling to reduce GC pressure
• Implements zero-copy operations where possible
• Optimizes fixed-length message handling
• Reduces memory allocations

Example configuration with performance improvements:

config = ProtocolConfig(
    message_type_enum=MessageType,
    message_structures={
        MessageType.SENSOR_DATA: sensor_data_structure,
        MessageType.STATUS: status_structure
    },
    enable_performance_improvements=True  # Enable optimizations
)

License

This project is licensed under the MIT License. See the LICENSE file for details.