Base source implementation for low-code sources.
Project description
Declarative-Manifest source connector
This is the repository for the Declarative-Manifest source connector, written in Python. The declarative manifest source connector is a special connector that can create an arbitrary source connector from a declarative manifest file. This allows users to create a source connector without writing any code.
Note: This connector is managed by the Airbyte Python CDK release process. It can be run as a standalone connector in Docker and PyAirbyte, but is not yet meant to be run in the platform as a standalone connector. This source is an interface to the low-code CDK and as such, should not be modified without a corresponding CDK change.
Local development
Prerequisites
- Python (~=3.9)
- Poetry (~=1.7) - installation instructions here
Installing the connector
From this connector directory, run:
poetry install
Create credentials
The credentials for source-declarative-manifest are a little different. Your config
will need to contain the
injected declarative manifest, as indicated in the spec
. It will also need to contain the fields that the spec
coming out of the manifest requires. An example is available in integration_tests/pokeapi_config.json
. To use
this example in the following instructions, copy this file to secrets/config.json
.
Locally running the connector
poetry run source-declarative-manifest spec
poetry run source-declarative-manifest check --config secrets/config.json
poetry run source-declarative-manifest discover --config secrets/config.json
poetry run source-declarative-manifest read --config secrets/config.json --catalog sample_files/configured_catalog.json
Running unit tests
To run unit tests locally, from the connector directory run:
poetry run pytest unit_tests
Building the docker image
- Install
airbyte-ci
- Run the following command to build the docker image:
airbyte-ci connectors --name=source-declarative-manifest build
An image will be available on your host with the tag airbyte/source-declarative-manifest:dev
.
Running as a docker container
Then run any of the connector commands as follows:
docker run --rm airbyte/source-declarative-manifest:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-declarative-manifest:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-declarative-manifest:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-declarative-manifest:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
Running our CI test suite
You can run our full test suite locally using airbyte-ci
:
airbyte-ci connectors --name=source-declarative-manifest test
This source does not currently pass the full test suite.
Dependency Management
The manifest declarative source is built to be an interface to the low-code CDK source. This means that this source should not have any production dependencies other than the Airbyte Python CDK. If for some reason you feel that a new dependency is needed, you likely want to add it to the CDK instead. It is expected that a given version of the source-declarative-manifest connector corresponds to the same version in its CDK dependency.
Publishing a new version of the connector
New versions of this connector should only be published (automatically) via the manual Airbyte CDK release process. If you want to make a change to this connector that is not a result of a CDK change and a corresponding CDK dependency bump, please reach out to the Connector Extensibility team for guidance.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for airbyte_source_declarative_manifest-3.0.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | f123c68c0cf8d24118fa9b33e13fe41f7be5669ec8dc856684958572ba00477a |
|
MD5 | 1793f2a0204e0c06cd1b832149e11aef |
|
BLAKE2b-256 | 9168b9d6bff987f58baeb8c13cc2e188757b8c286d5da2916ee63c3a7ca1a3f5 |
Hashes for airbyte_source_declarative_manifest-3.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 46169f0c18f837788bf54faae60a4b798af242f29673b64e3ad27c69d889de3b |
|
MD5 | ada8e3d1f2e690d0c79f1f969fe8af2e |
|
BLAKE2b-256 | d8479afbf63fa6ff6734c18fa98ddeb5c34f70d7dba7cc111a110efe127afed4 |