Skip to main content

A Python library for the Demisto SDK

Project description

PyPI version CircleCI Language grade: Python Coverage Status

Demisto SDK

The Demisto SDK library can be used to manage your Demisto content with ease and efficiency. The library uses python 3.7+.

Usage

Installation

  1. Install - pip3 install demisto-sdk

  2. Upgrade - pip3 install --upgrade demisto-sdk

  3. Demisto server demisto-sdk integration - In order that demisto-sdk and Demisto server communicate, perfrom the following steps:

    1. Get an API key for Demisto-server - Settings -> Integrations -> API keys -> Get your Key (copy it, you will be to copy it once)
    2. Add the following parameters to ~/.zshrc and ~/.bash_profile:
    export DEMISTO_BASE_URL=<http or https>://<demisto-server url or ip>:<port>
    export DEMISTO_API_KEY=<API key>
    

    for example:

    export DEMISTO_BASE_URL=http://127.0.0.1:8080
    export DEMISTO_API_KEY=XXXXXXXXXXXXXXXXXXXXXX
    
    1. Reload your terminal before continue.

CLI usage

You can use the SDK in the CLI as follows:

demisto-sdk <command> <args>

For more information, run demisto-sdk -h. For more information on a specific command execute demisto-sdk <command> -h.


Commands

Supported commands:

  1. init
  2. Validate
  3. Lint
  4. Secrets
  5. Unify
  6. Split-yml
  7. Create
  8. Format
  9. Run
  10. Run-playbook
  11. Upload
  12. Generate-docs
  13. Generate-test-playbook
  14. Json-to-outputs
  15. Create-id-set

Init

Create a pack, integration or script template. If --integration and --script flags are not given the command will create a pack.

Examples:

  1. This command will create a new integration template named MyNewIntegration within "path/to/my/dir" directory:

    demisto-sdk init --integration -n MyNewIntegration -o path/to/my/dir
    

For detailed command usage press here


Validate

Makes sure your content repository files are in order and have valid yml file scheme.

Examples:

  1. This command will validate all the files in content repo:

    demisto-sdk validate
    
  2. This will validate the file Integrations/Pwned-V2/Pwned-V2.yml only:

    demisto-sdk validate -p Integrations/Pwned-V2/Pwned-V2.yml
    

For detailed command usage press here


Lint

Run lintings (flake8, mypy, pylint, bandit) and pytest. pylint and pytest will run within all the docker images of an integration/script. Meant to be used with integrations/scripts that use the folder (package) structure. Will lookup up what docker image to use and will setup the dev dependencies and file in the target folder.

Examples:

  1. This command will parallel run the linters, excluding mypy, on the python files inside the "Integrations/PaloAltoNetworks_XDR" and "Scripts/HelloWorldScript" directories, using 2 workers (threads):

    demisto-sdk lint -d Integrations/PaloAltoNetworks_XDR,Scripts/HellowWorldScript --no-mypy -p -m 2
    

For detailed command usage press here


Secrets

Run Secrets validator to catch sensitive data before exposing your code to public repository. Attach full path to whitelist to allow manual whitelists. Default file path to secrets is "./Tests/secrets_white_list.json".

Examples:

  1. This will run the secrets validator on your files:

    demisto-sdk secrets
    

For detailed command usage press here


Unify

Unify the code, image and description files to a single Demisto yaml file.

Examples:

  1. This command will grab the integration components in Integrations/MyInt directory and unify them to a single yaml file that will be created in the "Integrations" directory.

    demisto-sdk unify -i Integrations/MyInt -o Integrations
    

For detailed command usage press here


Split-yml

Extract code, image and description files from a demisto integration or script yml file.

Examples

  1. This command will split the yml file to a directory with the integration components (code, image, description, pipfile etc.

    demisto-sdk split-yml -i Integrations/integration-MyInt.yml -o Integrations/MyInt
    
  2. This command will split the yml file to a directory with the script components (code, description, pipfile etc.)

    demisto-sdk split-yml -i Scripts/script-MyInt.yml -o Scripts/MyInt
    

For detailed command usage press here


Create

Create content artifacts.

Examples:

  1. This command will create content artifacts in the current directory:

    demisto-sdk create -a .
    

For detailed command usage press here


Format

Format your integration/script/playbook yml file according to Demisto's standard automatically.

Examples:

  1. This command will go through the integration file, format it, and override the original file with the necessary changes:

    demisto-sdk format -t integration -s Integrations/Pwned-V2/Pwned-V2.yml
    

For detailed command usage press here


Run

Run an integration command in the playground of a remote Demisto instance and retrieves the output.

In order to run the command, DEMISTO_BASE_URL environment variable should contain the Demisto base URL, and DEMISTO_API_KEY environment variable should contain a valid Demisto API Key.

Examples:

  1. This command will run the query !gct-translate-text text="ciao" target="iw" on the playground of the Demisto instance and print the output:

    demisto-sdk run -q '!gct-translate-text text="ciao" target="iw"
    

For detailed command usage press here


Run-playbook

Run a playbook in a given Demisto instance. DEMISTO_API_KEY environment variable should contain a valid Demisto API Key. You can either specify a URL as an environment variable named: DEMISTO_BASE_URL, or enter it as an argument.

Examples:

  1. This command will run the playbook playbook_name in Demisto instance https://demisto.local and will wait for the playbook to finish its run:

    DEMISTO_API_KEY=<API KEY> demisto-sdk run-playbook -p 'playbook_name' -u 'https://demisto.local'
    

For detailed command usage press here


Upload

Upload an integration to Demisto instance.

In order to run the command, DEMISTO_BASE_URL environment variable should contain the Demisto base URL, and DEMISTO_API_KEY environment variable should contain a valid Demisto API Key.

Examples:

  1. This will upload the integration YML file integration-GoogleCloudTranslate.yml to the Demisto instance:

    demisto-sdk upload -i Integrations/GoogleCloudTranslate/integration-GoogleCloudTranslate.yml
    

generate-docs

Generate documentation file for integration, playbook or script from yaml file.

Arguments:

  • -i, --input Path of the yml file.

  • -o, --output The output dir to write the documentation file into, documentation file name is README.md.

  • -t, --file_type The type of yml file. When the argument is empty, the type will be selected automatically.

  • -e, --examples In order to create example, DEMISTO_BASE_URL environment variable should contain the Demisto base URL, and DEMISTO_API_KEY environment variable should contain a valid Demisto API Key. For integration - Path for file containing command or script examples. Each Command should be in a separate line. For script - the script example surrounded by double quotes. When the argument is empty, the documentation will be generate without examples.

  • -c, --command For integration only - A comma-separated value of commands to replace (e.g panorama-get-address,panorama-create-address). It can be used only with an existing README.md file. If any new command is presented - the new command will be added to the end of the docs file. For an existing command, it will try to replace the existing command in the README.md file. If it will fail, it will add it to the end of the file as well.

  • -id, --id_set Path of updated id_set.json file, used for generates script documentation. When the argument is empty, the documentation will be generate without Used In section.

  • -v, --verbose Verbose output - mainly for debugging purposes.

Examples: demisto-sdk generate-docs -o /Users/Documentations -i /demisto/content/Playbooks/playbook-Block_IP_-_Generic.yml This will generate documentation file to Block IP - Generic playbook in /Users/Documentations/README.md.

demisto-sdk generate-docs -o /Users/Documentations -i /demisto/content/Integrations/Tanium_v2/Tanium_v2.yml -c /Users/tanium_commands.txt This will generate documentation file to Tanium V2 integration in /Users/Documentations/README.md, the file /Users/tanium_commands.txt should contains the example commands to execute.

demisto-sdk generate-docs -o /Users/Documentations -i /demisto/content/Scripts/script-PrintErrorEntry.yml -id /demisto/content/Tests/id_set.json -e "!PrintErrorEntry message=Hi" This will generate documentation file to PrintErrorEntry script in /Users/Documentations/README.md. id_set.json should be updated to gets all the integration that uses this script.

demisto-sdk generate-docs -i /demisto/content/Integrations/Tanium_v2/Tanium_v2.yml -c tn-get-package This will generate the command section for tn-get-package and will replace it in the README.md file located in /demisto/content/Integrations/Tanium_v2/.

create-id-set

Create the content dependency tree by ids.

Arguments:

  • -o, --output The path of the directory in which you want to save the created id set.

Examples: demisto-sdk create-id-set -o . This will create the id set in the current directory.

In the code

You can import the SDK core class in your python code as follows:

from demisto_sdk.core import DemistoSDK

Dev Environment Setup

We build for python 3.7 and 3.8. We use tox for managing environments and running unit tests.

  1. Clone the Demisto-SDK repository (Make sure that you have GitHub account):
    git clone https://github.com/demisto/demisto-sdk

  2. If you are using a default python version 3.7 or 3.8 you can skip this part.

    pyenv is an easy tool to control the versions of python on your environment. Install pyenv and then run:

    pyenv install 3.7.5
    pyenv install 3.8.0
    

    After installing run in {path_to_demisto-sdk}/demisto-sdk:

    cd {path_to_demisto-sdk}/demisto-sdk
    pyenv versions
    

    And you should see marked with asterisks:

    * 3.7.5 (set by /{path_to_demisto-sdk}/demisto-sdk/.python-version)
    * 3.8.0 (set by /{path_to_demisto-sdk}/demisto-sdk/.python-version)
    

    If not, simply run the following command from the Demisto-SDK repository:

    pyenv local 3.7.5 3.8.0
    
  3. Using the terminal go to the Demisto-SDK repository - we will set up the development environment there.

  4. Install tox:

    pip3 install tox
    

    Then setup dev virtual envs for python 3 (will also install all necessary requirements):

    tox
    
  5. Set your IDE to use the virtual environment you created using the following path: /{path_to_demisto-sdk}/demisto-sdk/.tox/py37/bin/python


Generate-test-playbook

Generate Test Playbook from integration/script yml

Examples:

  1. This command will create a test playbook in TestPlaybook folder, with filename TestXDRPlaybook.yml:

    demisto-sdk generate-test-playbook -i Integrations/PaloAltoNetworks_XDR/PaloAltoNetworks_XDR.yml -n TestXDRPlaybook -t integration -o TestPlaybooks`
    

For detailed command usage press here


Convert JSON to Demisto Outputs

Convert JSON format to demisto entry context yaml format.

Examples:

  1. The following command :

    demisto-sdk json-to-outputs -c jira-get-ticket -p Jira.Ticket -i path/to/valid.json
    

    if valid.json looks like

    {
        "id": 100,
        "title": "do something title",
        "created_at": "2019-01-01T00:00:00"
    }
    

    This command will print to the stdout the following:

    arguments: []
    name: jira-get-ticket
    outputs:
    - contextPath: Jira.Ticket.id
      description: ''
      type: Number
    - contextPath: Jira.Ticket.title
      description: ''
      type: String
    - contextPath: Jira.Ticket.created_at
      description: ''
      type: Date
    

For detailed command usage press here


How to run commands in your development environment

In the Demisto-SDK repository while on the git branch you want to activate and run this command to use python 3.7:

source .tox/py37/bin/activate

or this command to use python 3.8:

For detailed command usage press here


Autocomplete

Our CLI supports autocomplete for Linux/MacOS machines, you can turn this feature on by running one of the following: for zsh users run in the terminal

eval "$(_DEMISTO_SDK_COMPLETE=source_zsh demisto-sdk)"

for regular bashrc users run in the terminal

eval "$(_DEMISTO_SDK_COMPLETE=source demisto-sdk)"

License

MIT - See LICENSE for more information.


Contributions

Contributions are welcome and appreciated.
For information regarding contributing, press here. For release guide, press here

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for demisto-sdk, version 0.5.2
Filename, size File type Python version Upload date Hashes
Filename, size demisto-sdk-0.5.2.tar.gz (712.1 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page