Base source implementation for low-code sources.
Project description
Declarative-Manifest source connector
This is the repository for the Declarative-Manifest source connector, written in Python. The declarative manifest source connector is a special connector that can create an arbitrary source connector from a declarative manifest file. This allows users to create a source connector without writing any code.
Note: This connector is managed by the Airbyte Python CDK release process. It can be run as a standalone connector in Docker and PyAirbyte, but is not yet meant to be run in the platform as a standalone connector. This source is an interface to the low-code CDK and as such, should not be modified without a corresponding CDK change.
Local development
Prerequisites
- Python (~=3.9)
- Poetry (~=1.7) - installation instructions here
Installing the connector
From this connector directory, run:
poetry install
Create credentials
The credentials for source-declarative-manifest are a little different. Your config
will need to contain the
injected declarative manifest, as indicated in the spec
. It will also need to contain the fields that the spec
coming out of the manifest requires. An example is available in integration_tests/pokeapi_config.json
. To use
this example in the following instructions, copy this file to secrets/config.json
.
Locally running the connector
poetry run source-declarative-manifest spec
poetry run source-declarative-manifest check --config secrets/config.json
poetry run source-declarative-manifest discover --config secrets/config.json
poetry run source-declarative-manifest read --config secrets/config.json --catalog sample_files/configured_catalog.json
Running unit tests
To run unit tests locally, from the connector directory run:
poetry run pytest unit_tests
Building the docker image
- Install
airbyte-ci
- Run the following command to build the docker image:
airbyte-ci connectors --name=source-declarative-manifest build
An image will be available on your host with the tag airbyte/source-declarative-manifest:dev
.
Running as a docker container
Then run any of the connector commands as follows:
docker run --rm airbyte/source-declarative-manifest:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-declarative-manifest:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-declarative-manifest:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-declarative-manifest:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
Running our CI test suite
You can run our full test suite locally using airbyte-ci
:
airbyte-ci connectors --name=source-declarative-manifest test
This source does not currently pass the full test suite.
Dependency Management
The manifest declarative source is built to be an interface to the low-code CDK source. This means that this source should not have any production dependencies other than the Airbyte Python CDK. If for some reason you feel that a new dependency is needed, you likely want to add it to the CDK instead. It is expected that a given version of the source-declarative-manifest connector corresponds to the same version in its CDK dependency.
Publishing a new version of the connector
New versions of this connector should only be published (automatically) via the manual Airbyte CDK release process. If you want to make a change to this connector that is not a result of a CDK change and a corresponding CDK dependency bump, please reach out to the Connector Extensibility team for guidance.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for airbyte_source_declarative_manifest-3.2.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 21c877b97dcbad317e26053376d271009e3da6ffd97dfd7d7a7ada484b30e4a6 |
|
MD5 | 89e1c2fea7890500776b71c78aba7034 |
|
BLAKE2b-256 | 5f7afe66427ae27454f767b74f488a3849e0cd0edfb66ef85463c8a5d8609075 |
Hashes for airbyte_source_declarative_manifest-3.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0a1b90457bbbcfd67abdad0d438ba4a8922c6d565f0e1c8b48799bbca79e9da0 |
|
MD5 | 4f2a5fc3ba6517bc947996900af99a2e |
|
BLAKE2b-256 | 7efa70f9b8d2cf4bbed6795cb1be49716cf755c678d53d360eed16bb43d35beb |