spl2-testing-framework 1.8.5

SPL2 Testing Framework

SPL2 Testing Framework

Overview

The SPL2 Testing Framework enables running SPL2 tests both locally (or on any Splunk instance with SPL2 orchestrator), remotely (using external cloud environments) or using cli.

• For Cloud: It uses the Search Service API.
• For Splunk: It uses the Splunk Search API (with SPL2 support)
• For cli - it uses the internal spl2-processor-cli library. This option is not available for public usage, as spl2-processor-cli is a Splunk internal tool.

Prerequisites

1. Install Python and Poetry

1. Ensure python3.x is available
2. Install testing framework:
   • Install poetry and execute the command poetry install to create a virtual environment and install required dependencies.
   • OR
   • Install the library using pip: pip install spl2-testing-framework

2. Set configuration. It may be done in spl2_test_config.json file or environment variables

Note: setting configuration is necessary for running tests using splunk or cloud environment. For running tests using cli no configuration is required. The only requirement is to have this library installed, as described below

spl2_test_config.json - local file present in the current working directory (the directory from where tests are executed)

Configuration for running tests using cloud search client (Ingest processor)

• cloud_instance - address of Cloud host where the tests can be executed
  • e.g.: staging.scs.splunk.com
• tenant - tenant to use for testing
  • e.g.: spl2-content
• bearer_token - token used for authentication. To obtain the token, go to: https://console.[cloud_instance]/[tenant]/settings

Configuration for running tests using splunk search client (Splunk instance)

• host - address of Splunk host where the tests can be executed
  • e.g.: localhost or https://10.202.35.219
• port - port of Splunk host where the tests can be executed
  • usually 8089, but can be different
• user - user to authenticate
• password - password to authenticate

The same configuration can be done using environment variables(however, spl2_test_config.json has higher priority):

cloud search client

• SPL2_TF_CLOUD_INSTANCE => cloud_instance
• SPL2_TF_TENANT => tenant
• SPL2_TF_BEARER_TOKEN => bearer_token

splunk search client

• SPL2_TF_HOST => host
• SPL2_TF_PORT => port
• SPL2_TF_USER => user
• SPL2_TF_PASSWORD => password

3. Installing spl2-processor-cli (Splunk internal tool)

spl2-processor-cli can be installed using brew:

brew install spl2-processor-cli

Before installation, it may be necessary to authenticate to artifactory by running:

okta-artifactory-login -t generic

Running tests

To run tests, execute the command:

spl2_tests_run [cli|splunk|cloud]

In the directory where the tests are located. Test discovery is recursive, so it's possible to run tests even from the root directory of the project.

It is possible to pass more options to the command, which works also with pytest, e.g.:

• -k "filter" - to run only tests which name contains "filter"
• -v[vv] - to see more verbose output
• -n [auto|<number>] - to run tests in parallel
  • auto - to use all available cores,
  • <number> - to use specific number of cores
  • however, it's recommended to run tests in parallel on cli mostly, as running on splunk or cloud doesn't give significant performance improvement
• -x - to stop on first failure
• -pdb - to enter debugger on failure
• ... and much more, whatever is supported by pytest

Additionally, the following options are supported:

• --test_dir - directory where test files (.test.json, .test.spl2, module.json) are located. Defaults to the current working directory. Discovery is recursive.

• --code_dir - directory where SPL2 code modules (.spl2) are located. Defaults to the current working directory. Falls back to --test_dir if the module is not found. See Separate code and test directories for details.

• --ignore_empty_strings - to ignore empty strings in the results

• --ignore_additional_fields_in_actual - to ignore fields present in actual results but not in expected results (useful when actual results contain extra fields that should not affect comparison)

• --create_comparison_sheet - to create a comparison sheet in comparison_box_test folder using actual and expected outputs (works only when running box tests)

• --cli_bench - when running tests with CLI, this option enables benchmarking mode by adding -b and -n flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., --cli_bench=1000 will add -b -n 1000 to the CLI command). In bench mode, all tests always succeed and CLI output is printed to logs - this is useful for performance testing and benchmarking. This is only applicable when using --type cli.

• --lookups_root_dir - root directory passed to spl2-processor-cli as --lookupsRootDir for resolving lookup transforms.conf. Defaults to . (current working directory). For a standard TA layout use --lookups_root_dir=package/default. Only applicable when using --type cli.

• --check_splunk_results - option to validate box test output by sending events to a Splunk instance (using HEC) and querying them back, with support for CIM (Common Information Model) and TA (Technology Add-on) checks.

• --reduction_stats - when set, box tests record a reduction rate per run: input _raw vs output _raw (after pairing input/output events). Rates are grouped by pipeline (the .spl2 code module stem) and sourcetype (from the input event if set, else the output event, else UNKNOWN). Off by default.

• --reduction_stats_file - path to a JSON file written at the end of the pytest session with collected rates (default reduction_stats.json). Only used when --reduction_stats is set.

• --reduction_stats_full - when set together with --reduction_stats, each pipeline’s JSON payload also includes a pandas describe() summary (count, mean, std, min, quartiles, max) per sourcetype column. Only used when --reduction_stats is set.

• Note: The pytest.ini.sample file allows you to define command parameters. Just update the configurations, rename the file by removing the .sample extension, and execute the command.

Run tests in IDE [PyCharm]

It's also possible to run tests in PyCharm. To do this, it's necessary to set Run Configurations

Sample configuration which may be used:

• Run configuration
  • Type: Python test
  • Module: spl2_testing_framework.test_runner
  • Parameters: --type [cli | splunk | cloud] --test_dir /tests/resources -o log_cli=true --log-cli-level=INFO --verbose
  • Optional: append --reduction_stats (and --reduction_stats_file, --reduction_stats_full if needed) to collect box-test reduction JSON as described above.
  • If test dir is not specified, current working directory will be used
  • Use --code_dir if SPL2 code modules are in a separate directory from tests
  • If necessary another pytest options can be added

Note: It's necessary to set "pytest" as default test runner in PyCharm settings

Separate code and test directories

By default, the framework expects SPL2 code modules (.spl2 files) to live alongside their test files. The --code_dir option allows you to keep code and tests in separate directory trees.

How it works

What is searched	Primary directory	Fallback directory
Code modules (.spl2)	--code_dir	--test_dir
Test files (.test.json, .test.spl2)	--test_dir	--code_dir

Both searches are recursive. If the primary directory has no results, the framework falls back to the other directory and logs the action.

When multiple files with the same name are found within a directory, a warning is logged and the lexicographically first match is used.

Example project layout

my-project/
├── src/                          # --code_dir
│   ├── network_traffic.spl2
│   └── dns_lookup.spl2
└── tests/                        # --test_dir
    ├── network_traffic.test.json
    ├── dns_lookup.test.json
    └── dns_lookup.test.spl2

Run with:

spl2_tests_run cli --test_dir tests --code_dir src

Backward compatibility

When --code_dir is not specified (or both flags point to the same directory), the framework behaves exactly as before — code modules are found next to their test files via the same recursive search.

Applies to all test types

The --code_dir option works for box tests, unit tests, and single SPL2 file execution.

Executing a spl2 file

This framework also supports executing a single spl2 file and prints the results in command line as well as a log file. This will help developers to get the results of the spl2 pipeline as and when they are developing the pipeline.

It requires 3 additional parameters:

• --template_file
• --sample_file
• --sample_delimiter

It will execute the template_file provided in the --template_file parameter. It will read samples if --sample_file parameter is provided and will separate the samples by using --sample_delimiter. If --sample_file is not provided, then it will look for the samples in the respective module.json file corresponding to the template_file.

To run a single spl2 file, execute the command:

single_spl2_file_run [cli|splunk|cloud]

It is possible to pass more options to the command, which works also with pytest, e.g.:

• --test_dir - Path in which the template file and module.json are available. If not provided, it will look for the current directory for the template file and module.json file

• --code_dir - Path where SPL2 code modules are located, if different from --test_dir. Falls back to --test_dir when not set. See Separate code and test directories.

• --template_file - The spl2 template file to execute

• --sample_file - A file containing all the samples required for the template file. If not provided, it will look for the samples in module.json file of the corresponding template file

• --sample_delimiter - Separator for separating the samples provided in the sample file. If not provided, it will use newline as a default separator.

• --limit_tests - Limit number of tests to execute from the unit, sample file or module.json. If not provided, all will be executed. This can be used for quick testing of spl2 files during development.

• ... and much more, whatever is supported by pytest

Note: The pytest.ini.sample file allows you to define command parameters. Just update the configurations, rename the file by removing the .sample extension, and execute the command.

Performance check

It is possible to measure execution time of spl2 pipeline, or even do more advanced time checks using flag:

• --performance_check=time - to run basic time measurements - time of execution of spl2 pipeline will be printed to stdout
• --performance_check=detailed_time - to do more advanced time checks which injects more timestamps into spl2 pipeline.

Running detailed_time check creates text file with spl2 pipeline code with injected timestamps after every command ("|") Content of this file will also be printed to stdout.

This checks can be applied only to box tests, as assertions which are used in unit tests may impact spl2 pipeline performance.

Reduction rate statistics (box tests)

Optional telemetry for how much each pipeline shrinks event payloads (length of input _raw vs matched output _raw), broken down by pipeline and sourcetype.

Enable

• Pass --reduction_stats to pytest (or add it in pytest.ini / your IDE run configuration).

Output

• During the session, each box test appends one or more floating-point rates into an in-memory structure: pipeline_name → sourcetype → [rates…].
• After all tests finish, if --reduction_stats was set, the framework logs a short pandas summary per pipeline and writes --reduction_stats_file (default reduction_stats.json).
• With --reduction_stats_full, the JSON file also embeds per-pipeline describe() statistics for each sourcetype column (handy for dashboards or quick distribution checks).

How rates are computed (high level)

• Input events are parsed from the box test source string; output rows come from the job’s destination results.
• Input/output rows are paired using similarity on _raw (with a threshold and a same-length positional fallback). Only matched pairs contribute rates.
• For each pair, lengths use a canonical JSON length when _raw parses as JSON or a Python literal; otherwise the raw string length is used, so identical logical content can yield a 0 reduction rate even if string formatting differs.

Scope

• Applies to box tests only. Single-SPL2 runs and unit tests do not populate reduction stats.

Check splunk results

When set (--check_splunk_results=<option>), box tests (after running the SPL2 pipeline) send pipeline output to the configured Splunk instance and validate results there instead of comparing only in-memory expected vs actual.

Value	Behavior
CIM	For each output event, ingest via HEC, load back from Splunk, then check CIM fields from expected_cim_fields.cim_fields with validate_compatibility(actual, expected). Skips if the box test has no cim_fields.
TA	For each output event, ingest via HEC, load back from Splunk, then assert equality with the expected row for that index.

You can name top-level fields to drop on both the pipeline/actual row and each matching expected_destination_result row before they are compared by adding ignore_fields_in_destination_compare: an object mapping field name → short reason. Use the same key under test in each entry in *.test.json (same file as expected_destination_result):

"test": {
  "source": "...",
  "expected_destination_result": [ ... ],
  "ignore_fields_in_destination_compare": {
    "_raw": "Raw event differs after Splunk indexing",
    "_time": "Timestamp may vary due to ingestion latency"
  }
}

If the key is omitted, no fields are ignored. It applies to the default in-memory box test assert (no --check_splunk_results) and to --check_splunk_results=TA. It does not apply to --check_splunk_results=CIM. If the file or key is missing, no fields are ignored.

Skipping a test for specific run modes

Use skip_on in the test block to skip an individual test entry depending on how the framework is invoked:

"test": {
  "source": "...",
  "expected_destination_result": [ ... ],
  "skip_on": ["splunk_ta", "splunk_cim"]
}

Value	Skipped when
"standard"	Running without --check_splunk_results (default in-memory compare)
"splunk_ta"	Running with --check_splunk_results=TA
"splunk_cim"	Running with --check_splunk_results=CIM

A single value can be given as a plain string instead of a list. Unrecognised values are ignored with a warning.

Configuration:

SPLUNK_INSTANCE in spl2_test_config.json

New top-level key for the Splunk instance used by --check_splunk_results:

{
  "SPLUNK_INSTANCE": {
    "ip": "<hostname-or-ip>",
    "port": 8088,
    "api_port": 8089,
    "username": "<splunk-user>",
    "password": "<splunk-password>",
    "index": "<index-name>",
    "hec_token": "<hec-token>"
  }
}

• ip – Splunk server host (e.g. localhost or hostname).
• port – HEC port (e.g. 8088).
• api_port – Management/API port (e.g. 8089) for Search API and index creation.
• username / password – Used for Search API and index creation (HTTP Basic).
• index – Index where HEC events are sent and then searched.
• hec_token – Token for HEC (Authorization: Splunk <token>).

Do not commit real credentials. Prefer environment variables (below) or a local config that is not in version control.

Environment variables

Any SPLUNK_INSTANCE field can be set via the environment; config file values override these only when non-empty.

Variable	Maps to
SPL2_TF_SPLUNK_INSTANCE_IP	ip
SPL2_TF_SPLUNK_INSTANCE_PORT	port
SPL2_TF_SPLUNK_INSTANCE_API_PORT	api_port
SPL2_TF_SPLUNK_INSTANCE_USERNAME	username
SPL2_TF_SPLUNK_INSTANCE_PASSWORD	password
SPL2_TF_SPLUNK_INSTANCE_INDEX	index
SPL2_TF_SPLUNK_INSTANCE_HEC_TOKEN	hec_token

Starting a local Splunk instance for TA testing

spl2_splunk_start spins up a splunk/splunk Docker container with the TA mounted as an installed app and writes a ready-to-use spl2_test_config.json.

spl2_splunk_start --ta-path /path/to/ta

The command:

1. Detects the TA app directory — prefers <ta-path>/package/ (standard TA layout), falls back to <ta-path> itself.
2. Reads app.manifest to derive the Splunk app name.
3. Starts the container with the app directory mounted as a volume.
4. Waits for Ansible provisioning (PLAY RECAP) to complete, then polls the HEC health endpoint until Splunk is ready.
5. Writes spl2_test_config.json in the current directory with all connection details pre-filled.

Options

Option	Default	Description
--ta-path	.	Path to the TA root directory.
--splunk-version	10.4	Splunk Docker image tag.
--no-config	off	Skip writing spl2_test_config.json.
--config-dest	spl2_test_config.json	Where to write the config file.

Once the container is up, run your tests with:

spl2_tests_run splunk --check_splunk_results=TA

Requirements: Docker must be running. The container exposes ports 8088 (HEC) and 8089 (API). If a container named spl2-splunk already exists with a matching app mount and Splunk version it is reused (started if stopped); if it exists with different parameters it is removed and recreated.

Override the default credentials via environment variables if needed:

Variable	Default	Purpose
SPL2_TF_LOCAL_SPLUNK_PASSWORD	Chang3d!	Splunk admin password
SPL2_TF_LOCAL_SPLUNK_HEC_TOKEN	(built-in token)	HEC token

Format all files

To format all files run:

black spl2_testing_framework tests