Skip to main content

Versatile Data Kit SDK ingestion plugin to ingest data via http requests.

Project description

VDK-INGEST-HTTP Plugin

monthly download count for vdk-ingest-http

This plugin provides functionality to ingest data over http.

Usage

To use the plugin, just install it, and set the method attribute of send_object_for_ingestion(), and send_tabular_data_for_ingestion() functions to "http".

Example:

def run(job_input: IJobInput):
    # Do something to get data for ingestion
    payload = get_some_data()

    # Ingest the data
    job_input.send_object_for_ingestion(payload=payload,
                                        destination_table="aa_test_table",
                                        method="http",
                                        target="http://example.com/data-source"
                                        )

The above example shows how to ingest json data. In this case, there are three arguments that are required: payload, method and target. This would be fixed in the future, so that only payload would be required.
The payload needs to be a json object, and should contain the destination_table inside, using the @table key. For example:

{
    "@table": "destination_table_name",
    "column1": "value1",
    "column2": "value2",
    "column3": "value3",
}

The method attribute needs to be provided for the time being.
The target attribute should specify the url endpoint, where the data will be ingested.

Testing

To develop or test locally (from the current directory)

# install dev and test dependencies
pip install -r requirements
# install the plugin in editable mode
pip install -e .
# run the tests
pytest

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

vdk_ingest_http-0.2.1431637373.tar.gz (7.5 kB view details)

Uploaded Source

File details

Details for the file vdk_ingest_http-0.2.1431637373.tar.gz.

File metadata

File hashes

Hashes for vdk_ingest_http-0.2.1431637373.tar.gz
Algorithm Hash digest
SHA256 0e144f8cd6aca0d263f665021f0094040b5da999e6eccc0c454eaca9b5f56388
MD5 aa8272d403e988614114202c9a50bc0f
BLAKE2b-256 3e873395bb15bc28c7f5c5f70216d164c8a11d8d0c801a6117ebd1eee90a269e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page