Skip to main content

Base source implementation for low-code sources.

Project description

Declarative-Manifest source connector

This is the repository for the Declarative-Manifest source connector, written in Python. The declarative manifest source connector is a special connector that can create an arbitrary source connector from a declarative manifest file. This allows users to create a source connector without writing any code.

Note: This connector is managed by the Airbyte Python CDK release process. It can be run as a standalone connector in Docker and PyAirbyte, but is not yet meant to be run in the platform as a standalone connector. This source is an interface to the low-code CDK and as such, should not be modified without a corresponding CDK change.

Local development

Prerequisites

  • Python (~=3.9)
  • Poetry (~=1.7) - installation instructions here

Installing the connector

From this connector directory, run:

poetry install

Create credentials

The credentials for source-declarative-manifest are a little different. Your config will need to contain the injected declarative manifest, as indicated in the spec. It will also need to contain the fields that the spec coming out of the manifest requires. An example is available in integration_tests/pokeapi_config.json. To use this example in the following instructions, copy this file to secrets/config.json.

Locally running the connector

poetry run source-declarative-manifest spec
poetry run source-declarative-manifest check --config secrets/config.json
poetry run source-declarative-manifest discover --config secrets/config.json
poetry run source-declarative-manifest read --config secrets/config.json --catalog sample_files/configured_catalog.json

Running unit tests

To run unit tests locally, from the connector directory run:

poetry run pytest unit_tests

Building the docker image

  1. Install airbyte-ci
  2. Run the following command to build the docker image:
airbyte-ci connectors --name=source-declarative-manifest build

An image will be available on your host with the tag airbyte/source-declarative-manifest:dev.

Running as a docker container

Then run any of the connector commands as follows:

docker run --rm airbyte/source-declarative-manifest:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-declarative-manifest:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-declarative-manifest:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-declarative-manifest:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json

Running our CI test suite

You can run our full test suite locally using airbyte-ci:

airbyte-ci connectors --name=source-declarative-manifest test

This source does not currently pass the full test suite.

Dependency Management

The manifest declarative source is built to be an interface to the low-code CDK source. This means that this source should not have any production dependencies other than the Airbyte Python CDK. If for some reason you feel that a new dependency is needed, you likely want to add it to the CDK instead. It is expected that a given version of the source-declarative-manifest connector corresponds to the same version in its CDK dependency.

Publishing a new version of the connector

New versions of this connector should only be published (automatically) via the manual Airbyte CDK release process. If you want to make a change to this connector that is not a result of a CDK change and a corresponding CDK dependency bump, please reach out to the Connector Extensibility team for guidance.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Built Distribution

File details

Details for the file airbyte_source_declarative_manifest-3.1.0.dev202407091452.tar.gz.

File metadata

File hashes

Hashes for airbyte_source_declarative_manifest-3.1.0.dev202407091452.tar.gz
Algorithm Hash digest
SHA256 4929f6f52bddfe8cac0f926866ef8055c52a16089ba915700200cb7c8b4f1024
MD5 caf7a4c18f9b7872d351ff5c71497e2c
BLAKE2b-256 07ed839e2d4e7c127a72c48db578f9539a23fbbf937d114907a090905d6aaea5

See more details on using hashes here.

File details

Details for the file airbyte_source_declarative_manifest-3.1.0.dev202407091452-py3-none-any.whl.

File metadata

File hashes

Hashes for airbyte_source_declarative_manifest-3.1.0.dev202407091452-py3-none-any.whl
Algorithm Hash digest
SHA256 79b1635817c8771dff3a7f232831c0b0b481fa67f6083dbb38d6fe88c42f1bd1
MD5 dd1e40cc892f35229c3d356694a82ac9
BLAKE2b-256 c4b31185d70ff27e04c8d635cc39952172dfdbf4eaed91e444e8954da78f624b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page