dlt-community-sources

Community-maintained dlt sources for various APIs. Install only what you need via extras.

Available Sources

Installation

pip install dlt-community-sources[app-store-connect]

Or with uv:

uv add dlt-community-sources[app-store-connect]

Usage

import dlt
from dlt_community_sources.twilio import twilio_source

pipeline = dlt.pipeline(
    pipeline_name="twilio",
    destination="duckdb",
    dataset_name="twilio_data",
)

source = twilio_source(
    account_sid="your_account_sid",
    api_key_sid="your_api_key_sid",
    api_key_secret="your_api_key_secret",
    resources=["messages", "calls"],
)

load_info = pipeline.run(source)
print(load_info)

See each source's README for detailed usage and authentication options.

Features

All sources share these common features:

• Incremental loading where applicable
• Automatic token/auth refresh
• Rate limit handling with exponential backoff
• Works with any dlt destination

Error handling

Sources distinguish two kinds of resources:

• Auxiliary resources (metadata: campaigns, ads, audiences, ...) skip expected client errors (HTTP 400/403/404) per endpoint — some APIs return 4xx for valid requests on accounts without certain features. Every skip is logged with the response body.
• Primary resources (the fact data a source exists for: report, insights) never skip. Failures raise PrimaryResourceTerminalError (request rejected — retrying will not help; 4xx other than 408/429) or PrimaryResourceTransientError (timeout, rate limit, provider-side job failure), so a load never "succeeds" with silently missing data. Both mix in dlt's TerminalException / TransientException. Note that dlt wraps resource exceptions in PipelineStepFailed / ResourceExtractionError, so retry helpers that inspect only one level (such as dlt.pipeline.helpers.retry_load) cannot see the classification — use contains_terminal_exception, which walks the exception chain:

from tenacity import Retrying, retry_if_exception, stop_after_attempt
from dlt_community_sources._utils import contains_terminal_exception

for attempt in Retrying(
    stop=stop_after_attempt(3),
    retry=retry_if_exception(lambda e: not contains_terminal_exception(e)),
    reraise=True,
):
    with attempt:
        pipeline.run(source)

To load a subset of resources, either pass the source-specific resources argument or use dlt's native selection: source.with_resources("report").

Operating in production

Lessons from running these sources on daily schedules:

• Isolate failures per account. When one runner processes many accounts/advertisers (one dlt.pipeline per account), wrap each account — and each discovery call — in its own try/except, collect failures, and exit non-zero at the end. A single account's terminal failure (e.g. an account put on hold by the provider) should not abort the remaining accounts, and one discovery failure (e.g. an MCC your API user is not invited to yet) should not kill the whole job.
• Tune connection retries for your network. dlt's requests client retries ConnectionError/Timeout with request_max_attempts=5 and request_backoff_factor=1 by default — it gives up in roughly fifteen seconds. If your egress path has transient blackouts, raise them via dlt config (environment variables RUNTIME__REQUEST_MAX_ATTEMPTS, RUNTIME__REQUEST_BACKOFF_FACTOR, RUNTIME__REQUEST_MAX_RETRY_DELAY).
• Alert on job failures. With this library's fail-loud behavior, data loss shows up as a non-zero exit instead of a green run with missing rows — but only if something watches the exit status. Wire your scheduler's failure signal (e.g. Cloud Run Job completed_execution_count{result:failed}) to an alert before relying on it.

Development

# Fork the repository first, then:
git clone https://github.com/YOUR_USERNAME/dlt-community-sources.git
cd dlt-community-sources
uv sync --group dev
uv run pytest -v
uv run ruff check .

AI-assisted development

This repository is set up for AI coding assistants with two layers of AI context:

• Project rules (.ai/rules.md) — coding conventions, testing patterns, and structure specific to this repo
• dltHub AI workbench — dlt ecosystem knowledge, MCP server, and guided skills for pipeline development

Both are synced to tool-specific locations (Claude Code, Cursor, Codex, etc.) via a single command:

bash scripts/sync-ai-rules.sh

The dltHub workbench provides an MCP server for source search (9,700+ APIs), pipeline inspection, and data exploration — plus skills that guide you through the full pipeline development workflow. See CONTRIBUTING.md for details.

Edit .ai/ only for project rules, then re-run the sync script. CI checks that files are in sync.

Contributing

See CONTRIBUTING.md.

License

MIT