Base source implementation for low-code sources.
Project description
Declarative-Manifest source connector
This is the repository for the Declarative-Manifest source connector, written in Python. The declarative manifest source connector is a special connector that can create an arbitrary source connector from a declarative manifest file. This allows users to create a source connector without writing any code.
Note: This connector is managed by the Airbyte Python CDK release process. It can be run as a standalone connector in Docker and PyAirbyte, but is not yet meant to be run in the platform as a standalone connector. This source is an interface to the low-code CDK and as such, should not be modified without a corresponding CDK change.
Local development
Prerequisites
- Python (~=3.9)
- Poetry (~=1.7) - installation instructions here
Installing the connector
From this connector directory, run:
poetry install
Create credentials
The credentials for source-declarative-manifest are a little different. Your config
will need to contain the
injected declarative manifest, as indicated in the spec
. It will also need to contain the fields that the spec
coming out of the manifest requires. An example is available in integration_tests/pokeapi_config.json
. To use
this example in the following instructions, copy this file to secrets/config.json
.
Locally running the connector
poetry run source-declarative-manifest spec
poetry run source-declarative-manifest check --config secrets/config.json
poetry run source-declarative-manifest discover --config secrets/config.json
poetry run source-declarative-manifest read --config secrets/config.json --catalog sample_files/configured_catalog.json
Running unit tests
To run unit tests locally, from the connector directory run:
poetry run pytest unit_tests
Building the docker image
- Install
airbyte-ci
- Run the following command to build the docker image:
airbyte-ci connectors --name=source-declarative-manifest build
An image will be available on your host with the tag airbyte/source-declarative-manifest:dev
.
Running as a docker container
Then run any of the connector commands as follows:
docker run --rm airbyte/source-declarative-manifest:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-declarative-manifest:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-declarative-manifest:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-declarative-manifest:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
Running our CI test suite
You can run our full test suite locally using airbyte-ci
:
airbyte-ci connectors --name=source-declarative-manifest test
This source does not currently pass the full test suite.
Dependency Management
The manifest declarative source is built to be an interface to the low-code CDK source. This means that this source should not have any production dependencies other than the Airbyte Python CDK. If for some reason you feel that a new dependency is needed, you likely want to add it to the CDK instead. It is expected that a given version of the source-declarative-manifest connector corresponds to the same version in its CDK dependency.
Publishing a new version of the connector
New versions of this connector should only be published (automatically) via the manual Airbyte CDK release process. If you want to make a change to this connector that is not a result of a CDK change and a corresponding CDK dependency bump, please reach out to the Connector Extensibility team for guidance.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for airbyte_source_declarative_manifest-0.86.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 18a96c85f1c5e9c15eb1418ab2698baecbc0f953421bad89953cb09e909d9582 |
|
MD5 | 499d3ef18e7bcb9e524c9f7a4f678b16 |
|
BLAKE2b-256 | c3cae203aaada195e776225c5c2b583693200668eb7ac3f15ac7b43b73a4d163 |
Hashes for airbyte_source_declarative_manifest-0.86.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4b68ef3071550f2272a85058bd9a289e6c31a602f587d51a5670abd06347b6b4 |
|
MD5 | 84a1f518ff2ba9c4add3352275c69609 |
|
BLAKE2b-256 | b2c588765127f3d39ae1ce342caf804e3f2f63ff72e173e12ed9c1fc3fb29811 |