A framework for writing Airbyte Connectors.
Project description
Connector Development Kit (Python)
The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for:
- HTTP APIs (REST APIs, GraphQL, etc..)
- Singer Taps
- Generic Python sources (anything not covered by the above)
The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate.
This document is a general introduction to the CDK. Readers should have basic familiarity with the Airbyte Specification before proceeding.
Getting Started
Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run
cd airbyte-integrations/connector-templates/generator
./generate.sh
then follow the interactive prompt. Next, find all TODO
s in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.
Additionally, you can follow this tutorial for a complete walkthrough of creating an HTTP connector using the Airbyte CDK.
Concepts & Documentation
See the concepts docs for a tour through what the API offers.
Example Connectors
HTTP Connectors:
Singer connectors:
Simple Python connectors using the barebones Source
abstraction:
Contributing
First time setup
We assume python
points to python >=3.9.
Setup a virtual env:
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]" # [dev] installs development-only dependencies
Iteration
- Iterate on the code locally
- Run tests via
python -m pytest -s unit_tests
- Perform static type checks using
mypy airbyte_cdk
.MyPy
configuration is in.mypy.ini
. - The
type_check_and_test.sh
script bundles both type checking and testing in one convenient command. Feel free to use it!
Autogenerated files
If the iteration you are working on includes changes to the models, you might want to regenerate them. In order to do that, you can run:
SUB_BUILD=CONNECTORS_BASE ./gradlew format --scan --info --stacktrace
This will generate the files based on the schemas, add the license information and format the code. If you want to only do the former and rely on pre-commit to the others, you can run the appropriate generation command i.e. ./gradlew generateProtocolClassFiles
or ./gradlew generateComponentManifestClassFiles
.
Testing
All tests are located in the unit_tests
directory. Run python -m pytest --cov=airbyte_cdk unit_tests/
to run them. This also presents a test coverage report.
Building a connector with your local CDK
When developing a new feature in the CDK, you may find it helpful to run a connector that uses that new feature. You can test this in one of two ways:
- Running a connector locally
- Building and running a source via Docker
Installing your local CDK into a local Python connector
In order to get a local Python connector running your local CDK, do the following.
First, make sure you have your connector's virtual environment active:
# from the `airbyte/airbyte-integrations/connectors/<connector-directory>` directory
source .venv/bin/activate
# if you haven't installed dependencies for your connector already
pip install -e .
Then, navigate to the CDK and install it in editable mode:
cd ../../../airbyte-cdk/python
pip install -e .
You should see that pip
has uninstalled the version of airbyte-cdk
defined by your connector's setup.py
and installed your local CDK. Any changes you make will be immediately reflected in your editor, so long as your editor's interpreter is set to your connector's virtual environment.
Building a Python connector in Docker with your local CDK installed
Create a symlink that connects <connector-directory>/airbyte-cdk
to your local CDK installation:
# from the `airbyte/airbyte-integrations/connectors/<connector-directory>` directory
ln -s ../../../airbyte-cdk/python airbyte-cdk
Now that you have a symbolic link in your source, you will nedd to install it in the docker image. Depending on how the docker image is build you will have two possibilities:
- If image building process uses a temporary image called
builder
, add the following just after the copy of the installed packages to the folder/usr/local
:
COPY airbyte-cdk airbyte-cdk
RUN PYTHONUSERBASE=/usr/local pip install -e ./airbyte-cdk
- Add the following lines to your connector's
Dockerfile
, before the line that installs dependencies viapip install -e .
:
COPY airbyte-cdk airbyte-cdk
RUN pip install -e ./airbyte-cdk
Add the following to your connectors build.gradle
file:
airbyteDocker {
followSymlinks = true
}
You should be able to build your connector with
# from the airbytehq/airbyte base directory
./gradlew :airbyte-integrations:connectors:<connector-directory>:airbyteDocker
and the installation should use your local CDK. Note that the local CDK is injected at build time, so if you make changes, you will have to run the build command again to see them reflected.
Note: if your connector uses a .dockerignore
file, it cannot have exclude-all
or exclude-except
patterns, i.e. the .dockerignore
must specifically say which files to ignore without using any regex.
Publishing a new version to PyPi
- Bump the package version in
setup.py
- Open a PR
- An Airbyte member must comment
/publish-cdk dry-run=true
to publish the package to test.pypi.org or/publish-cdk dry-run=false
to publish it to the real index of pypi.org.
Coming Soon
- Full OAuth 2.0 support (including refresh token issuing flow via UI or CLI)
- Airbyte Java HTTP CDK
- CDK for Async HTTP endpoints (request-poll-wait style endpoints)
- CDK for other protocols
- Don't see a feature you need? Create an issue and let us know how we can help!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for airbyte_cdk-0.18.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8a2f602a9bb93a73094352fcb46faaf8c83552ae4d069a1a45f0beee4ed0fc27 |
|
MD5 | badd8480be984bb27fb777aed841206c |
|
BLAKE2b-256 | ae6fce7bd2a315cdaaf128e8ac6cbdf0f5cf4180e94e2f131d14fdd44ec434a4 |