Base source implementation for low-code sources.
Project description
Declarative-Manifest source connector
This is the repository for the Declarative-Manifest source connector, written in Python. The declarative manifest source connector is a special connector that can create an arbitrary source connector from a declarative manifest file. This allows users to create a source connector without writing any code.
Note: This connector is managed by the Airbyte Python CDK release process. It can be run as a standalone connector in Docker and PyAirbyte, but is not yet meant to be run in the platform as a standalone connector. This source is an interface to the low-code CDK and as such, should not be modified without a corresponding CDK change.
Local development
Prerequisites
- Python (~=3.9)
- Poetry (~=1.7) - installation instructions here
Installing the connector
From this connector directory, run:
poetry install
Create credentials
The credentials for source-declarative-manifest are a little different. Your config
will need to contain the
injected declarative manifest, as indicated in the spec
. It will also need to contain the fields that the spec
coming out of the manifest requires. An example is available in integration_tests/pokeapi_config.json
. To use
this example in the following instructions, copy this file to secrets/config.json
.
Locally running the connector
poetry run source-declarative-manifest spec
poetry run source-declarative-manifest check --config secrets/config.json
poetry run source-declarative-manifest discover --config secrets/config.json
poetry run source-declarative-manifest read --config secrets/config.json --catalog sample_files/configured_catalog.json
Running unit tests
To run unit tests locally, from the connector directory run:
poetry run pytest unit_tests
Building the docker image
- Install
airbyte-ci
- Run the following command to build the docker image:
airbyte-ci connectors --name=source-declarative-manifest build
An image will be available on your host with the tag airbyte/source-declarative-manifest:dev
.
Running as a docker container
Then run any of the connector commands as follows:
docker run --rm airbyte/source-declarative-manifest:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-declarative-manifest:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-declarative-manifest:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-declarative-manifest:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
Running our CI test suite
You can run our full test suite locally using airbyte-ci
:
airbyte-ci connectors --name=source-declarative-manifest test
This source does not currently pass the full test suite.
Dependency Management
The manifest declarative source is built to be an interface to the low-code CDK source. This means that this source should not have any production dependencies other than the Airbyte Python CDK. If for some reason you feel that a new dependency is needed, you likely want to add it to the CDK instead. It is expected that a given version of the source-declarative-manifest connector corresponds to the same version in its CDK dependency.
Publishing a new version of the connector
New versions of this connector should only be published (automatically) via the manual Airbyte CDK release process. If you want to make a change to this connector that is not a result of a CDK change and a corresponding CDK dependency bump, please reach out to the Connector Extensibility team for guidance.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for airbyte_source_declarative_manifest-4.5.4.dev202408230135.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 012fafabf8c34e4940ae3cfd2e9018d9cbe0cc23f63a5e9fa002d3bfb191bfdc |
|
MD5 | 039ab3974548fec358ba2dfc27c42e92 |
|
BLAKE2b-256 | 0213c4048f303ef4a0f231cccb5746900dc9a1ec103fef07bd3aa46217f2161b |
Hashes for airbyte_source_declarative_manifest-4.5.4.dev202408230135-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b5177fe3c8cea38d9ed66189f597f6194c9bfdfb4c66e909b9642c15d27a2861 |
|
MD5 | e2a5dbfef8880500aa115d58f2775eec |
|
BLAKE2b-256 | d7ce2778bec284d88df76fd4ad046b970fb9efaea10f8458e27ffcb04aa58b67 |