Streamsets DataCollector API utility
Project description
A set of utilities to help with management of Streamsets pipelines.
Instalation
pip install sdctool
Configuration
sdc-hosts.yml
Create a YAML file called sdc-hosts.yml in the root of the project and fill in with details of the instances you plan to use with these utilities. See the template-sdc-hosts.yml for an example of the required formatting.
template-sdc-hosts.yml:
instances: production: protocol: https host: production-sdc-host port: 18630 verify_ssl: True stage: protocol: https host: stage-sdc-host port: 18630 verify_ssl: True development: protocol: http host: dev-sdc-host port: 18630
creds.yml
Create a YAML file called creds.yml in the root of the project and fill in with credentials of the instances you plan to use with these utilities. See the template-creds.yml for an example of the required formatting.
template-creds.yml:
instances: production: user: admin pass: s3cr3t stage: user: admin pass: s35am3 development: user: admin pass: adm1n15trat0r
Usage
The script sdc-tool includes help docs with details on the script arguments.
Promote Pipeline
Migrate one pipeline from one SDC environment to another. SDC administrator credentials are required to execute these commands. If a destPipelineId is not specified, a new pipeline will be created with the same name and description as the exported pipeline. Currently, origin offset values are not part of the exported configuration, so no manipulation of the destination’s offset is required.
$ sdc-tool pipeline promote --src dev \
--srcPipelineId ESImport77337a4f-74c5-45d1-91fd-7ce746f1bdfd \
--dest stage \
--destPipelineId ESImport48b1200f-c270-4937-a226-b3443ce850f3`
Export Pipeline
Export pipeline to --out path.
$ sdc-tool pipeline export --src dev \
--out sdc.json \
--pipelineId StreamManagerConsumerV06fa7c3d3-458f-4446-9f51-398899118b73
Import Pipeline
Import pipeline from --pipelineJson path.
$ sdc-tool pipeline import --dest production \
--pipelineJson testpipeline.json \
--pipelineId firstpipe
Start Pipeline
$ sdc-tool pipeline start \
--pipelineId firstpipe \
--host production
Stop Pipeline
$ sdc-tool pipeline stop \
--pipelineId firstpipe \
--host production
Validate Pipeline
Validate pipleine and return issues.
$ sdc-tool pipeline validate \
--pipelineId firstpipe \
--host production
Developing
This project depends on Python and Docker Compose Installing required libraries
pip install -r requirements.txt
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.