🔗 Flyte Connectors Plugin

🚀 Seamlessly connect Flyte workflows to external data sources and services

Build powerful data pipelines with native integrations to popular cloud services

🚀 Quick Start

Installation

pip install --pre flyteplugins-connectors[bigquery]  # Install BigQuery connector

BigQuery Integration

Execute SQL queries on BigQuery and seamlessly integrate results into your Flyte workflows:

from flyteplugins.connectors.bigquery.task import BigQueryConfig, BigQueryTask
import flyte
from flyte.io import DataFrame

# Configure your BigQuery connection
config = BigQueryConfig(
    ProjectID="your-gcp-project",
    Location="US"  # Optional: specify region
)

# Create a task environment
env = flyte.TaskEnvironment(name="analytics_env")

# Define your BigQuery task
analytics_task = BigQueryTask(
    name="user_analytics",
    inputs={
        "user_id": int,
        "start_date": str,
        "end_date": str
    },
    output_dataframe_type=DataFrame,
    plugin_config=config,
    query_template="""
        SELECT
            user_id,
            COUNT(*) as event_count,
            MAX(timestamp) as last_activity
        FROM events
        WHERE user_id = {{ .user_id }}
          AND DATE(timestamp) BETWEEN '{{ .start_date }}' AND '{{ .end_date }}'
        GROUP BY user_id
    """
)

env.from_task(analytics_task)

# Run your workflow
if __name__ == "__main__":
    flyte.init_from_config()
    result = flyte.with_runcontext(mode="remote").run(
        analytics_task,
        user_id=12345,
        start_date="2024-01-01",
        end_date="2024-01-31"
    )
    print(f"Workflow URL: {result.url}")

📚 Available Connectors

Connector	Status	Description	Use Cases
🔷 BigQuery	✅ Stable	Google Cloud data warehouse	Analytics, ML training, reporting
🔗 More Coming Soon	🚧	Additional connectors in development	-

🧪 Testing

Run the test suite to ensure everything works correctly:

# Run all connector tests
pytest plugins/connectors/tests/ -v

# Run specific connector tests
pytest plugins/connectors/tests/test_bigquery.py -v

# Run with coverage
pytest plugins/connectors/tests/ --cov=flyteplugins.connectors --cov-report=html

🤝 Contributing

We welcome contributions! Here's how to get started:

Fork the repository
Create a feature branch: git checkout -b feature/amazing-connector
Write tests for your changes
Ensure tests pass: pytest plugins/connectors/tests/
Submit a pull request

Adding a New Connector

Create your connector module in src/flyteplugins/connectors/
Implement the TaskTemplate interface
Add comprehensive tests in tests/
Update this README with examples
Add example usage in examples/connectors/

🔧 Requirements

Python: 3.10+
Flyte: Latest version
Dependencies: See pyproject.toml for full requirements

🆘 Support

📬 Community: Flyte Slack
🐛 Issues: GitHub Issues
💬 Discussions: GitHub Discussions
📚 Documentation: docs.flyte.org

Made with ❤️ by the Flyte Community

⭐ Star us on GitHub • 🐦 Follow us on Twitter • 💼 LinkedIn

Release files for flyteplugins-connectors 2.0.0b53

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for flyteplugins-connectors 2.0.0b53
File	Interpreter	ABI	Platform	Reset
flyteplugins_connectors-2.0.0b53-py3-none-any.whl 15.4 kB Jan 29, 2026	Python 3	none	any	Details

Release files / flyteplugins_connectors-2.0.0b53-py3-none-any.whl

Download URL	flyteplugins_connectors-2.0.0b53-py3-none-any.whl
Size	15.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`b810ce58b40e66e58d5a95a179a3eaa648c565fdbe4d2dd88a1b8c9197cc0778`
BLAKE2b-256 checksum How to use checksums	`d81ff179985662e75375f1623fb7cac44ced9034c979d2a339f3690f9b8e229d`
Upload date	Jan 29, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.13.11`