Skip to main content

Streamsets DataCollector API utility

Project description

sdctool
=======

A set of utilities to help with management of Streamsets pipelines.

Installation
-----------

``pip install sdctool``

Configuration
-------------

sdc-hosts.yml
^^^^^^^^^^^^^

Create a YAML file called ``sdc-hosts.yml`` in the root of the project
and fill in with details of the instances you plan to use with these
utilities. See the ``template-sdc-hosts.yml`` for an example of the
required formatting.

``template-sdc-hosts.yml``:

::

instances:
production:
protocol: https
host: production-sdc-host
port: 18630
verify_ssl: True
stage:
protocol: https
host: stage-sdc-host
port: 18630
verify_ssl: True
development:
protocol: http
host: dev-sdc-host
port: 18630

creds.yml
^^^^^^^^^

Create a YAML file called ``creds.yml`` in the root of the project and
fill in with credentials of the instances you plan to use with these
utilities. See the ``template-creds.yml`` for an example of the required
formatting.

``template-creds.yml``:

::

instances:
production:
user: admin
pass: s3cr3t
stage:
user: admin
pass: s35am3
development:
user: admin
pass: adm1n15trat0r

Usage
^^^^^

The script ``sdc-tool`` includes help docs with details on the script
arguments.

Promote Pipeline
^^^^^^^^^^^^^^^^

Migrate one pipeline from one SDC environment to another. SDC
administrator credentials are required to execute these commands. If a
``destPipelineId`` is not specified, a new pipeline will be created with
the same name and description as the exported pipeline. Currently,
origin offset values are not part of the exported configuration, so no
manipulation of the destination's offset is required.

.. code:: bash

$ sdc-tool pipeline promote --src dev \
--srcPipelineId ESImport77337a4f-74c5-45d1-91fd-7ce746f1bdfd \
--dest stage \
--destPipelineId ESImport48b1200f-c270-4937-a226-b3443ce850f3`

Export Pipeline
^^^^^^^^^^^^^^^

Export pipeline to ``--out`` path.

.. code:: bash

$ sdc-tool pipeline export --src dev \
--out sdc.json \
--pipelineId StreamManagerConsumerV06fa7c3d3-458f-4446-9f51-398899118b73

Import Pipeline
^^^^^^^^^^^^^^^

Import pipeline from ``--pipelineJson`` path.

.. code:: bash

$ sdc-tool pipeline import --dest production \
--pipelineJson testpipeline.json \
--pipelineId firstpipe

Start Pipeline
^^^^^^^^^^^^^^

.. code:: bash

$ sdc-tool pipeline start \
--pipelineId firstpipe \
--host production

Stop Pipeline
^^^^^^^^^^^^^

.. code:: bash

$ sdc-tool pipeline stop \
--pipelineId firstpipe \
--host production

Validate Pipeline
^^^^^^^^^^^^^^^^^

Validate pipleine and return issues.

.. code:: bash

$ sdc-tool pipeline validate \
--pipelineId firstpipe \
--host production

Developing
----------

This project depends on Python and Docker Compose Installing required
libraries

.. code:: bash

pip install -r requirements.txt

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for sdctool, version 0.11.0
Filename, size File type Python version Upload date Hashes
Filename, size sdctool-0.11.0.tar.gz (8.7 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page