Skip to main content

A framework for writing Airbyte Connectors.

Project description

Connector Development Kit (Python)

The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for:

  • HTTP APIs (REST APIs, GraphQL, etc..)
  • Singer Taps
  • Generic Python sources (anything not covered by the above)

The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate.

This document is a general introduction to the CDK. Readers should have basic familiarity with the Airbyte Specification before proceeding.

Getting Started

Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run

cd airbyte-integrations/connector-templates/generator
./generate.sh

then follow the interactive prompt. Next, find all TODOs in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.

Additionally, you can follow this tutorial for a complete walkthrough of creating an HTTP connector using the Airbyte CDK.

Concepts & Documentation

See the concepts docs for a tour through what the API offers.

Example Connectors

HTTP Connectors:

Singer connectors:

Simple Python connectors using the barebones Source abstraction:

Contributing

First time setup

We assume python points to python >=3.9.

Setup a virtual env:

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]" # [dev] installs development-only dependencies

Iteration

  • Iterate on the code locally
  • Run tests via pytest -s unit_tests
  • Perform static type checks using mypy airbyte_cdk. MyPy configuration is in .mypy.ini.
  • The type_check_and_test.sh script bundles both type checking and testing in one convenient command. Feel free to use it!

Testing

All tests are located in the unit_tests directory. Run pytest --cov=airbyte_cdk unit_tests/ to run them. This also presents a test coverage report.

Publishing a new version to PyPi

  1. Bump the package version in setup.py
  2. Open a PR
  3. An Airbyte member must comment /publish-cdk dry-run=true to publish the package to test.pypi.org or /publish-cdk dry-run=false to publish it to the real index of pypi.org.

Coming Soon

  • Full OAuth 2.0 support (including refresh token issuing flow via UI or CLI)
  • Airbyte Java HTTP CDK
  • CDK for Async HTTP endpoints (request-poll-wait style endpoints)
  • CDK for other protocols
  • Don't see a feature you need? Create an issue and let us know how we can help!

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airbyte-cdk-0.1.66.tar.gz (116.2 kB view details)

Uploaded Source

Built Distribution

airbyte_cdk-0.1.66-py3-none-any.whl (192.1 kB view details)

Uploaded Python 3

File details

Details for the file airbyte-cdk-0.1.66.tar.gz.

File metadata

  • Download URL: airbyte-cdk-0.1.66.tar.gz
  • Upload date:
  • Size: 116.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.11

File hashes

Hashes for airbyte-cdk-0.1.66.tar.gz
Algorithm Hash digest
SHA256 165018d8a2d6a286b43af3d765ffb61341f2e98d1ed29bf5d64c1159581c3bd7
MD5 e33f7210896d44f42dfd44f779fe05ea
BLAKE2b-256 c1180b98d5f120c4a73c643079c14c078ea5eee2dc99730e88d4f5617ddae588

See more details on using hashes here.

File details

Details for the file airbyte_cdk-0.1.66-py3-none-any.whl.

File metadata

  • Download URL: airbyte_cdk-0.1.66-py3-none-any.whl
  • Upload date:
  • Size: 192.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.11

File hashes

Hashes for airbyte_cdk-0.1.66-py3-none-any.whl
Algorithm Hash digest
SHA256 977de1020a3508ca4d03abf9cc94ec64cfd68724da86b1e900ac7af0f6148b67
MD5 4ed08986736fd75eaa2373f5246eaed6
BLAKE2b-256 69de16f61e86273dc26bde1f432ee9e1577bf7b5c56e55f864d03e0701a94486

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page