A Singer target for writing data to Miso API
Project description
target-miso
This package is a Singer target that sends data to Miso's Data API. Being a singer target, you can integrate it into your data pipeline using your favorite DataOps framework, for instance, Meltano.
Use with Meltano
Follow the steps below to setup target-miso
in your Meltano project. Or see the most essential project example here.
Setup
Setup meltano.yml
like this:
# ...
plugins:
# ...
loaders:
- name: target-miso
namespace: target_miso
pip_url: target-miso
executable: target-miso
config:
template_folder: template
api_server: https://api.askmiso.com
api_key: your_miso_api_key
use_async: false
Make sure to run meltano install
to install the dependency.
Configuration
The config object accepts the following properties:
name | required | explanation |
---|---|---|
template_folder | true | Where you keep the template files. The path is relative to Meltano project directory. |
api_server | true | The Miso API server host. |
api_key | true | Your Miso API key. |
use_async | true | Whether to send request in asynchronous mode. |
Replication methods
Currently, this target supports FULL_TABLE
and INCREMENTAL
replication methods. LOG_BASED
is not yet supported.
Template
To specify how records are transformed into payloads of Miso API, for each stream from the tap, put a corresponding jinja2 template file in your template folder. For example, given a stream product
, put a template file product.jinja
like this:
{
"product_id": "{{ data.uuid }}",
"created_at": "{{ data.created_at | datetime_format }}",
"title": "{{ data.title }}",
"description": "{{ data.description }}",
"categories": "{{ data.category | convert_categories }}",
"custom_attributes": {
"vender": "{{ data.vender if data.vender }}"
}
}
Rules on output data types
Miso takes 3 kinds of data records: user
, product
, and interaction
. A record is classified into one of these type by the following rules:
- If the payload contains the
type
field, it is an interaction record. - If the payload contains the
user_id
field, it is a user record. - If the payload contains the
product_id
field, it is a product record.
Built-in filters
Target Miso comes with a few built-in filters that can be used in template expressions:
datetime_format
Takes a string in any format compatible with dateparser and output in ISO format, which is desired by Miso API.
list_of_str
Wrap a string to a singleton list of string. For example, "apple"
to ["apple"]
.
convert_categories
Wrap a string to a singleton double-layered list of string. For example, "apple"
to [["apple"]]
.
remove_symbol
- Convert an int to string.
- Strip off some special characters from input string, including:
"
,\
,\\N
,“
,\r\n
,\n
,\r
.
split
Split a string into a list by comma.
fix_url
Encode (as URL component) the path component of a URL string.
jsonify
Serialize an obj to a JSON string.
Copyright © 2021 Miso Corp.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for target_miso-0.9.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7f0121d581a044fdd970adc3a04cd7e454c7abdf4ca107077a4b0edd81da279a |
|
MD5 | f051cc8e78df531233e43a436839b21c |
|
BLAKE2b-256 | 76fd2ef4e0d93333f61678f71bb6c792ba821a137a7bcdb63eddf3dc0330d9b6 |