Snowflake Streaming Ingest Python SDK

A high-performance Python SDK for streaming data ingestion into Snowflake, built with Rust for optimal performance and reliability.

Overview

The Snowflake Streaming Ingest Python SDK provides a Python interface for real-time data streaming into Snowflake tables. It leverages a Rust core for high performance while providing a familiar Python API for easy integration.

Supported Platforms

Operating Systems

• Linux: x86_64, aarch64
• macOS: arm64 (Apple Silicon)
• Windows: x86_64

Python Versions

• Python 3.9 (minimum required)
• Python 3.10
• Python 3.11
• Python 3.12
• Python 3.13

Minimum System Requirements

Linux

• glibc: 2.26 or higher
• Architecture: x86_64, aarch64

macOS

• Version: macOS 11.0 (Big Sur) or higher
• Architecture: Apple Silicon (arm64)

Windows

• Version: Windows 10 or higher
• Architecture: x86_64

Installation

From PyPI (Recommended)

pip install snowpipe-streaming

Quick Start

from snowflake.ingest.streaming import StreamingIngestClient

# Create a client
client = StreamingIngestClient(
    client_name="my_client",
    db_name="my_database",
    schema_name="my_schema",
    pipe_name="my_pipe",
    properties={
        "account": "your_account",
        "user": "your_user",
        "private_key": "your_private_key",
        "url": "https://your_account.snowflakecomputing.com"
    }
)

# Open a channel
channel, status = client.open_channel("my_channel")

# Insert data
row_data = {
    "id": 1,
    "name": "John Doe",
    "timestamp": "2024-01-01T00:00:00Z"
}
channel.append_row(row_data)

# Close resources
channel.close()
client.close()

Features

• High Performance: Rust-based core for optimal throughput and low latency
• Memory Efficient: Minimal memory footprint with efficient data handling
• Type Safety: Strong typing support with Python type hints
• Automatic Retries: Built-in retry logic for transient failures
• Backpressure Handling: Intelligent backpressure management
• Comprehensive Logging: Detailed logging for debugging and monitoring
• Cross-Platform: Native support for Linux, macOS, and Windows

Architecture

The SDK uses a hybrid Rust-Python architecture:

• Rust Core: High-performance data processing, networking, and Snowflake communication
• Python Bindings: Pythonic API using PyO3 for seamless integration
• Stable ABI: Uses Python's stable ABI (abi3) for forward compatibility

Dependencies

Core Dependencies

• msgspec: Fast JSON serialization/deserialization
• Native Extensions: Rust-based binary modules (automatically included)

Build Dependencies (for development)

• maturin: Rust-Python build tool
• pytest: Testing framework
• Rust toolchain: Required for building from source

Support

• Documentation: Snowflake Documentation
• Issues: GitHub Issues
• Community: Snowflake Community

License

This project is licensed under the Apache License 2.0. See LICENSE-2.0.txt for details.

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests: python -m pytest tests/
5. Submit a pull request

For more details, see the main repository's contribution guidelines.