A Singer target for writing data to Miso API
Project description
target-miso
This package is a Singer target that sends data to Miso's Data API. Being a singer target, you can integrate it into your data pipeline using your favorite DataOps framework, for instance, Meltano.
Use with Meltano
Follow the steps below to setup target-miso
in your Meltano project. Or see the most essential project example here.
Setup
Setup meltano.yml
like this:
# ...
plugins:
# ...
loaders:
- name: target-miso
namespace: target_miso
pip_url: target-miso
executable: target-miso
config:
template_folder: template
api_key: your_miso_api_key
Make sure to run meltano install
to install the dependency.
Configuration
The config object accepts the following properties:
name | required | default | explanation |
---|---|---|---|
api_server | no | https://api.askmiso.com | The Miso API server host. |
api_key | yes | Your Miso API key. | |
template_folder | yes | Where you keep the template files. The path is relative to Meltano project directory. | |
use_async | no | False | Whether to send request in asynchronous mode. |
dry_run | no | False | Whether to send request in dry-run mode. |
write_record_limit | no | 100 | The maximum number of records to be written. |
Replication methods
Currently, this target supports FULL_TABLE
and INCREMENTAL
replication methods. LOG_BASED
is not yet supported.
Template
To specify how records are transformed into payloads of Miso API, for each stream from the tap, put a corresponding jinja2 template file in your template folder. For example, given a stream product
, put a template file product.jinja
like this:
{
"product_id": "{{ data.uuid }}",
"created_at": "{{ data.created_at | datetime_format }}",
"title": "{{ data.title }}",
"description": "{{ data.description }}",
"categories": "{{ data.category | convert_categories }}",
"custom_attributes": {
"vender": "{{ data.vender if data.vender }}"
}
}
Rules on output data types
Miso takes 3 kinds of data records: user
, product
, and interaction
. A record is classified into one of these type by the following rules:
- If the payload contains the
type
field, it is an interaction record. - If the payload contains the
user_id
field, it is a user record. - If the payload contains the
product_id
field, it is a product record.
Built-in filters
Target Miso comes with a few built-in filters that can be used in template expressions:
datetime_format
Takes a string in any format compatible with dateparser and output in ISO format, which is desired by Miso API.
list_of_str
Wrap a string to a singleton list of string. For example, "apple"
to ["apple"]
.
convert_categories
Wrap a string to a singleton double-layered list of string. For example, "apple"
to [["apple"]]
.
remove_symbol
- Convert an int to string.
- Strip off some special characters from input string, including:
"
,\
,\\N
,“
,\r\n
,\n
,\r
.
split
Split a string into a list by comma.
fix_url
Encode (as URL component) the path component of a URL string.
jsonify
Serialize an obj to a JSON string.
Copyright © 2021 Miso Corp.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for target_miso-0.9.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cbb083b4364740c4e6aa92dd7900e6bac2dd2115bfb576719723ba6cdee91020 |
|
MD5 | 50f87eb814d2446e24c8d8e4e14dce50 |
|
BLAKE2b-256 | dff59600691d54e65553d6531e0a883223e87d3d7444b3d0fad26f3016d405cf |