Connector Development Kit (Python CDK)
The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for:
- HTTP APIs (REST APIs, GraphQL, etc..)
- Singer Taps
- Generic Python sources (anything not covered by the above)
The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate.
This document is a general introduction to the CDK. Readers should have basic familiarity with the Airbyte Specification before proceeding.
Getting Started
Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run
cd airbyte-integrations/connector-templates/generator
npm run generate
then follow the interactive prompt. Next, find all TODOs in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.
Additionally, you can follow this tutorial for a complete walkthrough of creating an HTTP connector using the Airbyte CDK.
Concepts & Documentation
See the concepts docs for a tour through what the API offers.
Example Connectors
HTTP Connectors:
Singer connectors:
Simple Python connectors using the barebones Source abstraction:
Contributing
First time setup
We assume python points to python >=3.7.
Setup a virtual env:
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]" # [dev] installs development-only dependencies
Iteration
- Iterate on the code locally
- Run tests via
pytest -s unit_tests - Perform static type checks using
mypy airbyte_cdk.MyPyconfiguration is in.mypy.ini. - The
type_check_and_test.shscript bundles both type checking and testing in one convenient command. Feel free to use it!
Testing
All tests are located in the unit_tests directory. Run pytest --cov=airbyte_cdk unit_tests/ to run them.
This also presents a test coverage report.
Publishing a new version to PyPi
- Bump the package version in
setup.py - Open a PR
- An Airbyte member must comment
/publish-cdk --dry-run=<true or false>. Dry runs publish to test.pypi.org.
Coming Soon
- Full OAuth 2.0 support (including refresh token issuing flow via UI or CLI)
- Airbyte Java HTTP CDK
- CDK for Async HTTP endpoints (request-poll-wait style endpoints)
- CDK for other protocols
- General CDK for Destinations
- Don't see a feature you need? Create an issue and let us know how we can help!
1 maintainer
Release files for airbyte-cdk 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| airbyte-cdk-0.1.1.tar.gz | 26.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| airbyte_cdk-0.1.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size:72.6 kB
Release files / airbyte-cdk-0.1.1.tar.gz
| Download URL | airbyte-cdk-0.1.1.tar.gz |
|---|---|
| Size | 26.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5c3df85462517140f42214d8d30b964dc1f561ab3930508c4edc8e0f94a3278d
|
|
BLAKE2b-256 checksum How to use checksums |
3b5723b99c38c1df9b2f4de0472b016c735d92ade6f751812a329e5e41edbd56
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
Release files / airbyte_cdk-0.1.1-py3-none-any.whl
| Download URL | airbyte_cdk-0.1.1-py3-none-any.whl |
|---|---|
| Size | 46.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8cebc5fb256db80749b4d91c64207972eb59e50fbf3c204da059833995d35cdf
|
|
BLAKE2b-256 checksum How to use checksums |
01b5db3205e41d3b296a150b2a1df5312929e8c0ee4db88fe831c6016ea57ecb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
