Skip to main content

Singer.io tap for extracting data from Wonolo REST API

Project description

tap-wonolo

PyPI version PyPI - Status Build Status PyPI - Python Version License: GPL v3

A Singer tap for extracting data from the Wonolo REST API v2.

Installation

Since package dependencies tend to conflict between various taps and targets, Singer recommends installing taps and targets into their own isolated virtual environments:

Install Wonolo Tap

$ python3 -m venv ~/.venvs/tap-wonolo
$ source ~/.venvs/tap-wonolo/bin/activate
$ pip3 install tap-wonolo
$ deactivate

Install Stitch Target (optional)

$ python3 -m venv ~/.venvs/target-stitch
$ source ~/.venvs/target-stitch/bin/activate
$ pip3 install target-stitch
$ deactivate

Configuration

The tap accepts a JSON-formatted configuration file as arguments. This configuration file has four required fields:

  1. api_key: A valid Wonolo API key.
  2. secret_key: A valid Wonolo secret key.
  3. environment: A valid Wonolo enviroment (either "test" or "production").

An bare-bones Wonolo configuration may file may look like the following:

{
  "api_key": "foo",
  "secret_key": "bar",
  "environment": "test"
}

Authentication

If no auth_token key is supplied in the configuration file, the tap will automatically request one via the API and write it back to the config file:

{
  "api_key": "foo",
  "secret_key": "bar",
  "environment": "test",
  "auth_token": "foobar",
  "auth_token_expires_at": "2019-11-08T00:00:20Z",
}

The tap will then use the aforementioned auth_token to authenticate to the API, until the auth_token becomes invalidated, at which point the tap will automatically request and record a new auth_token from the API.

Granular Stream Configuration

Additionally, you may specify more granular configurations for individual streams. Each key under a stream should represent a valid API request parameter for that endpoint. A more fleshed-out configuration file may look similar to the following:

{
  "api_key": "foo",
  "secret_key": "bar",
  "environment": "test",
  "api_version": "v2",
  "streams": {
    "jobs": {
      "state": "fulfilled",
      "company_id": "1234",
      "classification": "w2"
    },
    "job_requests": {
      "w2_hourly_rate": 21.5,
      "updated_after": "2017-05-24T17:01:19.391-07:00"
    }
  }
}

Streams

The current version of the tap syncs three distinct Streams:

  1. Jobs: Endpoint Documentation
  2. Job Requests: Endpoint Documentation
  3. Users: Endpoint Documentation

Discovery

Singer taps describe the data that a stream supports via a Discovery process. You can run the Dayforce tap in Discovery mode by passing the --discover flag at runtime:

$ ~/.venvs/tap-wonolo/bin/tap-wonolo --config=config/wonolo.config.json --discover

The tap will generate a Catalog to stdout. To pass the Catalog to a file instead, simply redirect it to a file:s

$ ~/.venvs/tap-wonolo/bin/tap-wonolo --config=config/wonolo.config.json --discover > catalog.json

Sync to stdout

Running a tap in Sync mode will extract data from the various selected Streams. In order to run a tap in Sync mode and have messages emitted to stdout, pass a valid configuration file and catalog file:

$ ~/.venvs/tap-wonolo/bin/tap-wonolo --config=config/wonolo.config.json --catalog=catalog.json

The tap will emit occasional Metric, Schema, Record, and State messages. You can persist State between runs by redirecting messages to a file:

$ ~/.venvs/tap-wonolo/bin/tap-wonolo --config=config/wonolo.config.json --catalog=catalog.json >> state.json
$ tail -1 state.json > state.json.tmp
$ mv state.json.tmp state.json

Sync to Stitch

You can also send the output of the tap to Stitch Data for loading into the data warehouse. To do this, first create a JSON-formatted configuration for Stitch. This configuration file has two required fields:

  1. client_id: The ID associated with the Stitch Data account you'll be sending data to.
  2. token The token associated with the specific Import API integration within the Stitch Data account.

An example configuration file will look as follows:

{
  "client_id": 1234,
  "token": "foobarfoobar"
}

Once the configuration file is created, simply pipe the output of the tap to the Stitch Data target and supply the target with the newly created configuration file:

$ ~/.venvs/tap-wonolo/bin/tap-wonolo --config=config/dayforce.config.json --catalog=catalog.json --state=state.json | ~/.venvs/target-stitch/bin/target-stitch --config=config/stitch.config.json >> state.json
$ tail -1 state.json > state.json.tmp
$ mv state.json.tmp state.json

Contributing

The first step to contributing is getting a copy of the source code. First, fork tap-wonolo on GitHub. Then, cd into the directory where you want your copy of the source code to live and clone the source code:

$ git clone git@github.com:YourGitHubName/tap-wonolo.git

Now that you have a copy of the source code on your local machine, you can leverage Pipenv and the corresponding Pipfile to install of the development dependencies within a virtual environment:

$ pipenv install --three --dev

This command will create an isolated virtual environment for your tap-wonolo project and install all the development dependencies defined within the Pipfile inside of the environment. You can then enter a shell within the environment:

$ pipenv shell

Or, you can run individual commands within the environment without entering the shell:

$ pipenv run <command>

For example, to format your code using isort and flake8 before commiting changes, run the following commands:

$ pipenv run make isort
$ pipenv run make flake8

You can also run the entire testing suite before committing using tox:

$ pipenv run tox

Finally, you can run your local version of the tap within the virtual environment using a command like the following:

$ pipenv run tap-wonolo --config=config/dayforce.config.json --catalog=catalog.json

Once you've confirmed that your changes work and the testing suite passes, feel free to put out a PR!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for tap-wonolo, version 0.1.0
Filename, size File type Python version Upload date Hashes
Filename, size tap_wonolo-0.1.0-py3-none-any.whl (22.8 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size tap_wonolo-0.1.0.tar.gz (12.1 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page