Skip to main content

Singer tap for StackExchange, built with the Meltano SDK for Singer Taps.

Project description

tap-stackexchange

Test Maintainability Rating Security Rating Duplicated Lines (%)

Singer tap for the StackExchange API.

Built with the Meltano SDK for Singer Taps and Targets.

Capabilities

  • sync
  • catalog
  • state
  • discover

Settings

Setting Required Default Description
key False None Pass this to receive a higher request quota
site False stackoverflow.com StackExchange site
tags False None Question tags
start_date False None The earliest record date to sync

A full list of supported settings and capabilities is available by running: tap-stackexchange --about

Custom filter

The StackExchange API supports a number of custom filters that can be used to include or exclude certain fields from the response objects. This application has a baked-in filter with the following parameters:

  • include

    Parameter Description
    question.comment_count The number of comments on the question
    tag.last_activity_date The date of the last activity on the tag
  • unsafe=false

Update the baked-in filter

To update the baked-in filter, edit the FILTER_ID constant in tap_stackexchange/tap.py.

To generate a new filter, use the Try It button on the StackExchange API documentation, using the baked-in filter as the base parameter.

Use a custom filter

To use a custom filter, set the filter setting to the filter ID. Note that if you use a custom filter, you will need to use a custom catalog that includes the fields you want to sync. That is you will need to

  1. Write the default catalog to a file: tap-stackexchange --discover > catalog.json
  2. Edit the catalog file to include the fields not included by the default API filter
  3. Run the tap with the custom catalog: tap-stackexchange --config config.json --catalog catalog.json

Installation

pipx install git+https://github.com/edgarrmondragon/tap-stackexchange.git

Source Authentication and Authorization

Register a new application on Stack Apps and copy the generated key.

Usage

You can easily run tap-stackexchange by itself or in a pipeline using Meltano.

Executing the Tap Directly

tap-stackexchange --version
tap-stackexchange --help
tap-stackexchange --config CONFIG --discover > ./catalog.json

Initialize your Development Environment

pipx install poetry
poetry install

Testing with Meltano

Note: This tap will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.

Your project comes with a custom meltano.yml project file already created. Open the meltano.yml and follow any "TODO" items listed in the file.

Next, install Meltano (if you haven't already) and any needed plugins:

# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd tap-stackexchange
meltano install

Now you can test and orchestrate using Meltano:

# Test invocation:
meltano invoke tap-stackexchange --version

# OR run a test `elt` pipeline:
meltano elt tap-stackexchange target-sqlite --job_id=stackexchange-sqlite

# Runtime configuration
TAP_STACKEXCHANGE__LOAD_SCHEMA=dragon_ball_gt \
TAP_STACKEXCHANGE_SITE=anime \
TAP_STACKEXCHANGE_TAGS='["dragon-ball-gt"]' \
meltano elt tap-stackexchange target-sqlite

SDK Dev Guide

See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tap_stackexchange-0.0.1b1.tar.gz (12.9 kB view details)

Uploaded Source

Built Distribution

tap_stackexchange-0.0.1b1-py3-none-any.whl (13.1 kB view details)

Uploaded Python 3

File details

Details for the file tap_stackexchange-0.0.1b1.tar.gz.

File metadata

  • Download URL: tap_stackexchange-0.0.1b1.tar.gz
  • Upload date:
  • Size: 12.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for tap_stackexchange-0.0.1b1.tar.gz
Algorithm Hash digest
SHA256 20ea32ff01780f79ac7b31c5628d13c3f44a9b18ac945833a84e9ab911497bcf
MD5 3b019a10c176112d95e7ed86de4280c3
BLAKE2b-256 06abacda390b69ccaf628ae3d19974043b539edc42d35f8f6fd49acf0986c03a

See more details on using hashes here.

Provenance

The following attestation bundles were made for tap_stackexchange-0.0.1b1.tar.gz:

Publisher: build.yaml on MeltanoLabs/tap-stackexchange

Attestations:

File details

Details for the file tap_stackexchange-0.0.1b1-py3-none-any.whl.

File metadata

File hashes

Hashes for tap_stackexchange-0.0.1b1-py3-none-any.whl
Algorithm Hash digest
SHA256 32c35717677057b61914aa19636aa4c0580467ef35e252ff41c996d367532582
MD5 9f0835799a16a4ef36d20c66e3ada1b2
BLAKE2b-256 0f9c5db179ea5897ac8e4b0eb1fee55813d55e1814b05b791f2caea361cee1a3

See more details on using hashes here.

Provenance

The following attestation bundles were made for tap_stackexchange-0.0.1b1-py3-none-any.whl:

Publisher: build.yaml on MeltanoLabs/tap-stackexchange

Attestations:

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page