sayou-connector

Overview

The Universal Data Ingestion Engine for Sayou Fabric.

sayou-connector provides a unified interface to fetch data from diverse sources—Files, Cloud Drives, Databases, and SaaS APIs—normalizing everything into a standard format called SayouPacket.

It decouples the logic of Navigation (Generator) from Retrieval (Fetcher), enabling complex recursive crawling, pagination, and API traversal strategies out of the box.

1. Architecture & Role

The Connector Pipeline manages the Feedback Loop between discovery and retrieval. It yields a stream of SayouPacket objects ready for the next stage (Refinery).

graph LR
    Source[Source String] --> Pipeline[Connector Pipeline]
    
    subgraph Generators [Navigation]
        Dir[File Walker]
        Crawler[Web Frontier]
        APIPag[API Paginator]
    end
    
    subgraph Fetchers [Retrieval]
        Local[File Read]
        HTTP[Requests]
        SQL[DB Query]
    end
    
    Pipeline --> Generators
    Generators -->|Task| Fetchers
    Fetchers -->|Packet| Pipeline
    Pipeline -->|Feedback| Generators

1.1. Core Features

Generator/Fetcher Pattern: Separates "Where to go next" (Generator) from "How to get it" (Fetcher).
Unified Packet: Whether the source is a Notion Page or a PostgreSQL Row, the output is always a uniform SayouPacket.
Resilience: Built-in rate limiting, retries, and error handling for unstable network sources.

2. Supported Sources

sayou-connector supports a vast array of plugins, continuously expanding to cover Enterprise SaaS and Databases.

Category	Key Sources	Description
Local / File	`file`, `obsidian`	Local file systems, Markdown vaults.
Web / Media	`web`, `youtube`, `wikipedia`, `rss`	Web crawling (Trafilatura), YouTube transcripts, Wiki articles.
SaaS / Cloud	`github`, `notion`, `google_drive`, `gmail`	Repository code, Notion workspaces, G-Suite documents.
Database	`postgres`, `mysql`, `mongodb`, `oracle`	SQL/NoSQL databases with pagination support.

3. Installation

pip install sayou-connector

4. Usage

The ConnectorPipeline acts as the entry point. It automatically detects the source type or accepts a specific strategy.

Case A: Local & Web (Simple)

Fetching simple files or web pages.

from sayou.connector import ConnectorPipeline

packets = ConnectorPipeline.process(
    source="./my_docs",
    strategy="file"
)

web_packets = ConnectorPipeline.process(
    source="https://news.daum.net/tech",
    strategy="web"
)

for packet in web_packets:
    print(f"[Fetched] {packet.uri} ({len(packet.data)} bytes)")

Case B: SaaS Integration (GitHub / Notion)

Fetching structured data from external APIs.

from sayou.connector import ConnectorPipeline

repo_packets = ConnectorPipeline.process(
    source="https://github.com/sayouzone/sayou-fabric",
    strategy="github"
)

print(f"Collected {len(list(repo_packets))} files from repo.")

Case C: Database Ingestion

Fetching rows from a database table.

from sayou.connector import ConnectorPipeline

db_config = {
    "host": "localhost",
    "user": "admin",
    "password": "password",
    "db": "sales_db"
}

# Fetch rows from 'orders' table
db_packets = ConnectorPipeline.process(
    source="orders", 
    strategy="postgres",
    config=db_config
)

# Each packet contains a batch of rows
for packet in db_packets:
    print(f"Batch rows: {len(packet.data)}")

5. Configuration Keys

The config dictionary is crucial for authentication and connection settings.

auth: API Keys (e.g., github_token, notion_token, google_creds).
db: Database credentials (host, port, user, password).
crawl: Web crawling settings (user_agent, depth_limit, domain_lock).
filter: File extensions to include/exclude (e.g., include=[".py", ".md"]).

6. License

7. Plugin List

Plugin	Example	Description
`GitHub Connector`	▶
`Gmail Connector`	▶
`Google Calendar Connector`	▶
`Google Drive Connector`	▶
`Google Docs Connector`	▶
`Google Sheets Connector`	▶
`Google Slides Connector`	▶
`Youtube Connector`	▶
`Youtube Public Connector`	▶
`Email Connector`	▶
`MongoDB Connector`	▶
`MSSQL Connector`	▶
`MySQL Connector`	▶
`Oracle Connector`	▶
`PostgreSQL Connector`	▶
`SQLite Connector`	▶
`Local File Connector`	▶
`Requests Connector`	▶
`Notion Connector`	▶
`Obsidian Connector`	▶
`RSS Connector`	▶
`Trafilatura Connector`	▶
`wikipedia Connector`	▶

Metadata

Release files for sayou-connector 0.5.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sayou-connector 0.5.0
File	Size	Uploaded	Provenance
sayou_connector-0.5.0.tar.gz	96.5 kB	Apr 1, 2026	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for sayou-connector 0.5.0
File	Interpreter	ABI	Platform	Reset
sayou_connector-0.5.0-py3-none-any.whl 82.5 kB Apr 1, 2026 Without provenance	Python 3	none	any	Details

Total release size: 179.0 kB

Release files / sayou_connector-0.5.0.tar.gz

Download URL	sayou_connector-0.5.0.tar.gz
Size	96.5 kB
Tags	Source
SHA-256 checksum How to use checksums	`c913da637b40acf9528f91216d4995559ce838e8a55c8e7cbb15230ed4a92382`
BLAKE2b-256 checksum How to use checksums	`3f5026d0b0ebb84d054fdad4c6dd8e408858d2f10e2ea947da1b07d4e9d2e619`
Upload date	Apr 1, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.7`

Release files / sayou_connector-0.5.0-py3-none-any.whl

Download URL	sayou_connector-0.5.0-py3-none-any.whl
Size	82.5 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`bb3b7ee8692811c85150a712093c822b1ad58bc76291b4cdbae96807120f329c`
BLAKE2b-256 checksum How to use checksums	`73ac3f6c4c2d38a3a7ec3296a739e1284f2600e9d2b071497d6a6a21406d419d`
Upload date	Apr 1, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.7`

Security for sayou-connector 0.5.0

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.