Skip to main content

Base source implementation for low-code sources.

Project description

Declarative-Manifest source connector

This is the repository for the Declarative-Manifest source connector, written in Python. The declarative manifest source connector is a special connector that can create an arbitrary source connector from a declarative manifest file. This allows users to create a source connector without writing any code.

Note: This connector is managed by the Airbyte Python CDK release process. It can be run as a standalone connector in Docker and PyAirbyte, but is not yet meant to be run in the platform as a standalone connector. This source is an interface to the low-code CDK and as such, should not be modified without a corresponding CDK change.

Local development

Prerequisites

  • Python (~=3.9)
  • Poetry (~=1.7) - installation instructions here

Installing the connector

From this connector directory, run:

poetry install

Create credentials

The credentials for source-declarative-manifest are a little different. Your config will need to contain the injected declarative manifest, as indicated in the spec. It will also need to contain the fields that the spec coming out of the manifest requires. An example is available in integration_tests/pokeapi_config.json. To use this example in the following instructions, copy this file to secrets/config.json.

Locally running the connector

poetry run source-declarative-manifest spec
poetry run source-declarative-manifest check --config secrets/config.json
poetry run source-declarative-manifest discover --config secrets/config.json
poetry run source-declarative-manifest read --config secrets/config.json --catalog sample_files/configured_catalog.json

Running unit tests

To run unit tests locally, from the connector directory run:

poetry run pytest unit_tests

Building the docker image

  1. Install airbyte-ci
  2. Run the following command to build the docker image:
airbyte-ci connectors --name=source-declarative-manifest build

An image will be available on your host with the tag airbyte/source-declarative-manifest:dev.

Running as a docker container

Then run any of the connector commands as follows:

docker run --rm airbyte/source-declarative-manifest:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-declarative-manifest:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-declarative-manifest:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-declarative-manifest:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json

Running our CI test suite

You can run our full test suite locally using airbyte-ci:

airbyte-ci connectors --name=source-declarative-manifest test

This source does not currently pass the full test suite.

Dependency Management

The manifest declarative source is built to be an interface to the low-code CDK source. This means that this source should not have any production dependencies other than the Airbyte Python CDK. If for some reason you feel that a new dependency is needed, you likely want to add it to the CDK instead. It is expected that a given version of the source-declarative-manifest connector corresponds to the same version in its CDK dependency.

Publishing a new version of the connector

New versions of this connector should only be published (automatically) via the manual Airbyte CDK release process. If you want to make a change to this connector that is not a result of a CDK change and a corresponding CDK dependency bump, please reach out to the Connector Extensibility team for guidance.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Built Distribution

File details

Details for the file airbyte_source_declarative_manifest-4.4.3.dev202408170045.tar.gz.

File metadata

File hashes

Hashes for airbyte_source_declarative_manifest-4.4.3.dev202408170045.tar.gz
Algorithm Hash digest
SHA256 9c4f64c469ccb8920e91a484ea1cec88828fdd3569bd0240c7206ddb83ea54e1
MD5 117520d114835b2ba1d78ed9a4e5c4a6
BLAKE2b-256 09386d6988351870a6472f661f17b907dea6ba42edb83ee517b2b4708d46f521

See more details on using hashes here.

File details

Details for the file airbyte_source_declarative_manifest-4.4.3.dev202408170045-py3-none-any.whl.

File metadata

File hashes

Hashes for airbyte_source_declarative_manifest-4.4.3.dev202408170045-py3-none-any.whl
Algorithm Hash digest
SHA256 290eb2a1f3e79604f843128f8a90c8ebef53af84a74421128c2a0e605a7898a7
MD5 e98d43f94b42caebb034251ef7bb023e
BLAKE2b-256 49624f8d615bcb178705a381a7dba2d3598ef007d8262f764ab9698e2902155c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page