Skip to main content
Help us improve PyPI by participating in user testing. All experience levels needed!

Streamsets DataCollector API utility

Project description

A set of utilities to help with management of Streamsets pipelines.

Instalation

pip install sdctool

Configuration

sdc-hosts.yml

Create a YAML file called sdc-hosts.yml in the root of the project and fill in with details of the instances you plan to use with these utilities. See the template-sdc-hosts.yml for an example of the required formatting.

template-sdc-hosts.yml:

instances:
  production:
    protocol: https
    host: production-sdc-host
    port: 18630
    verify_ssl: True
  stage:
    protocol: https
    host: stage-sdc-host
    port: 18630
    verify_ssl: True
  development:
    protocol: http
    host: dev-sdc-host
    port: 18630

creds.yml

Create a YAML file called creds.yml in the root of the project and fill in with credentials of the instances you plan to use with these utilities. See the template-creds.yml for an example of the required formatting.

template-creds.yml:

instances:
  production:
    user: admin
    pass: s3cr3t
  stage:
    user: admin
    pass: s35am3
  development:
    user: admin
    pass: adm1n15trat0r

Usage

The script sdc-tool includes help docs with details on the script arguments.

Promote Pipeline

Migrate one pipeline from one SDC environment to another. SDC administrator credentials are required to execute these commands. If a destPipelineId is not specified, a new pipeline will be created with the same name and description as the exported pipeline. Currently, origin offset values are not part of the exported configuration, so no manipulation of the destination’s offset is required.

$ sdc-tool pipeline promote --src dev \
  --srcPipelineId ESImport77337a4f-74c5-45d1-91fd-7ce746f1bdfd \
  --dest stage \
  --destPipelineId ESImport48b1200f-c270-4937-a226-b3443ce850f3`

Export Pipeline

Export pipeline to --out path.

$ sdc-tool pipeline export --src dev \
  --out sdc.json \
  --pipelineId StreamManagerConsumerV06fa7c3d3-458f-4446-9f51-398899118b73

Import Pipeline

Import pipeline from --pipelineJson path.

$ sdc-tool pipeline import --dest production \
  --pipelineJson testpipeline.json \
  --pipelineId firstpipe

Start Pipeline

$ sdc-tool pipeline start \
  --pipelineId firstpipe \
  --host production

Stop Pipeline

$ sdc-tool pipeline stop \
  --pipelineId firstpipe \
  --host production

Validate Pipeline

Validate pipleine and return issues.

$ sdc-tool pipeline validate \
  --pipelineId firstpipe \
  --host production

Developing

This project depends on Python and Docker Compose Installing required libraries

pip install -r requirements.txt

Project details


Release history Release notifications

History Node

0.11.0

This version
History Node

0.10.1

History Node

0.9.4

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
sdctool-0.10.1.tar.gz (9.2 kB) Copy SHA256 hash SHA256 Source None Oct 12, 2017

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page