This script takes input and lastic and kafka based on the config. if we want to add another plugin to this then add the plugin details in the config file in the plugins and create your new input file in the input folder.
Project description
Python sfapmetl feature
- To run python script, ProjectName and appName should be present.
- Please provide config as mentione below,
key: <profile key>
tags:
Name: <name>
appName: <appName>
projectName: <projectName>
metrics:
plugins:
- name: <PluginName>
enabled: true
url:
job: <job_url>
stage: <stage_url>
task: <task_url>
authkey: <authentication_key_for_the_urls>
buffer_path: <path for bufferfile.json> #provide path for bufferfile which keeps record of unfinished job,stage and tasks.
-
Please mention the config file path in main.py file
-
After this setup, add cronjob into /etc/crontab( Applicable for Linux AWS instance, else run this script as a cron job) ex: To run script every 5 minutes - */5 * * * * root python /home/ubuntu/python_script/utility/main.py
-
please refer this link for cronjob https://www.digitalocean.com/community/tutorials/how-to-use-cron-to-automate-tasks-ubuntu-1804
Note:
- If we want to add another plugin to this, add the plugin details in the config file under the plugins section and create your new input file in the input folder.
- The input python file name should always be same as the plugin name. In this case plugin name is "etl".
- There should be a fuction named "work" with config parameter which will be invoked when we run the script.
- The work fuction should return list of dictionary objects.
Installation
$ pip install sf-apm-etl
Usage
$ sfapmetl <config file path>
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for sfapmetl-0.1.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fe084368ee0bb1e86c0388c950ae32149fdac35f32888d3cb43141a98d1700da |
|
MD5 | 39ef49f4208f35476da162c2ac0b64bb |
|
BLAKE2b-256 | 4f8f459b5eda594679a58a47c4991c1e0a54c6a0edde8eb439e4c9ffe9befe15 |