Skip to main content

Streamsets DataCollector API utility

Project description

A set of utilities to help with management of Streamsets pipelines.


pip install sdctool



Create a YAML file called sdc-hosts.yml in the root of the project and fill in with details of the instances you plan to use with these utilities. See the template-sdc-hosts.yml for an example of the required formatting.


    protocol: https
    host: production-sdc-host
    port: 18630
    verify_ssl: True
    protocol: https
    host: stage-sdc-host
    port: 18630
    verify_ssl: True
    protocol: http
    host: dev-sdc-host
    port: 18630


Create a YAML file called creds.yml in the root of the project and fill in with credentials of the instances you plan to use with these utilities. See the template-creds.yml for an example of the required formatting.


    user: admin
    pass: s3cr3t
    user: admin
    pass: s35am3
    user: admin
    pass: adm1n15trat0r


The script sdc-tool includes help docs with details on the script arguments.

Promote Pipeline

Migrate one pipeline from one SDC environment to another. SDC administrator credentials are required to execute these commands. If a destPipelineId is not specified, a new pipeline will be created with the same name and description as the exported pipeline. Currently, origin offset values are not part of the exported configuration, so no manipulation of the destination’s offset is required.

$ sdc-tool pipeline promote --src dev \
  --srcPipelineId ESImport77337a4f-74c5-45d1-91fd-7ce746f1bdfd \
  --dest stage \
  --destPipelineId ESImport48b1200f-c270-4937-a226-b3443ce850f3`

Export Pipeline

Export pipeline to --out path.

$ sdc-tool pipeline export --src dev \
  --out sdc.json \
  --pipelineId StreamManagerConsumerV06fa7c3d3-458f-4446-9f51-398899118b73

Import Pipeline

Import pipeline from --pipelineJson path.

$ sdc-tool pipeline import --dest production \
  --pipelineJson testpipeline.json \
  --pipelineId firstpipe

Start Pipeline

$ sdc-tool pipeline start \
  --pipelineId firstpipe \
  --host production

Stop Pipeline

$ sdc-tool pipeline stop \
  --pipelineId firstpipe \
  --host production

Validate Pipeline

Validate pipleine and return issues.

$ sdc-tool pipeline validate \
  --pipelineId firstpipe \
  --host production


This project depends on Python and Docker Compose Installing required libraries

pip install -r requirements.txt

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for sdctool, version 0.10.1
Filename, size File type Python version Upload date Hashes
Filename, size sdctool-0.10.1.tar.gz (9.2 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page