`tap-stackexchange` is a Singer tap for StackExchange, built with the Meltano SDK for Singer Taps.
Project description
tap-stackexchange
Singer tap for the StackExchange API.
Built with the Meltano SDK for Singer Taps and Targets.
Capabilities
sync
catalog
state
discover
Settings
Setting | Required | Default | Description |
---|---|---|---|
key | False | None | Pass this to receive a higher request quota |
site | False | stackoverflow.com | StackExchange site |
tags | False | None | Question tags |
start_date | False | None | The earliest record date to sync |
A full list of supported settings and capabilities is available by running: tap-stackexchange --about
Custom filter
The StackExchange API supports a number of custom filters that can be used to include or exclude certain fields from the response objects. This application has a baked-in filter with the following parameters:
-
include
Parameter Description question.comment_count
The number of comments on the question tag.last_activity_date
The date of the last activity on the tag -
unsafe=false
Update the baked-in filter
To update the baked-in filter, edit the FILTER_ID
constant in tap_stackexchange/tap.py
.
To generate a new filter, use the Try It button on the StackExchange API documentation, using the baked-in filter as the base
parameter.
Use a custom filter
To use a custom filter, set the filter
setting to the filter ID. Note that if you use a custom filter, you will need to
use a custom catalog that includes the fields you want to sync. That is you will need to
- Write the default catalog to a file:
tap-stackexchange --discover > catalog.json
- Edit the catalog file to include the fields not included by the default API filter
- Run the tap with the custom catalog:
tap-stackexchange --config config.json --catalog catalog.json
Installation
pipx install git+https://github.com/edgarrmondragon/tap-stackexchange.git
Source Authentication and Authorization
Register a new application on Stack Apps and copy the generated key
.
Usage
You can easily run tap-stackexchange
by itself or in a pipeline using Meltano.
Executing the Tap Directly
tap-stackexchange --version
tap-stackexchange --help
tap-stackexchange --config CONFIG --discover > ./catalog.json
Initialize your Development Environment
pipx install poetry
poetry install
Testing with Meltano
Note: This tap will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.
Your project comes with a custom meltano.yml
project file already created. Open the meltano.yml
and follow any "TODO" items listed in
the file.
Next, install Meltano (if you haven't already) and any needed plugins:
# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd tap-stackexchange
meltano install
Now you can test and orchestrate using Meltano:
# Test invocation:
meltano invoke tap-stackexchange --version
# OR run a test `elt` pipeline:
meltano elt tap-stackexchange target-sqlite --job_id=stackexchange-sqlite
# Runtime configuration
TAP_STACKEXCHANGE__LOAD_SCHEMA=dragon_ball_gt \
TAP_STACKEXCHANGE_SITE=anime \
TAP_STACKEXCHANGE_TAGS='["dragon-ball-gt"]' \
meltano elt tap-stackexchange target-sqlite
SDK Dev Guide
See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tap_stackexchange-0.0.1a2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 31ef336a3883b8a1622875554f677a013148b7832d124c01ff039800a3603f67 |
|
MD5 | a9d7e93de944b5b8365f5da1552b9f64 |
|
BLAKE2b-256 | 110d2efd73e7bf2b60375ae770d9dea8ff970165847d63228a53711890e29c91 |
Hashes for tap_stackexchange-0.0.1a2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d4a14331520509b370962ae537d636253f93c68fd5abca64e55e6149d3ca1a1f |
|
MD5 | a97f90c4ab903ea793adb900b4fd44c5 |
|
BLAKE2b-256 | 123041a6f8cbef3cf5ab4b347215d4c6467ec6ddc801d53e767786ea24dbda46 |