spark-rapids-user-tools

User tools to help with the adoption, installation, execution, and tuning of the NVIDIA cuDF plugin for Apache Spark.

The wrapper improves end-user experience within the following dimensions:

Qualification: Educate the CPU customer on the cost savings and acceleration potential of the cuDF plugin. The output shows a list of apps recommended for the cuDF plugin with estimated savings and speed-up.
Tuning: Tune the cuDF plugin configs based on initial job run leveraging Spark event logs. The output shows recommended per-app cuDF plugin configuration settings.
Diagnostics: Run diagnostic functions to validate the Dataproc environment with the cuDF plugin to make sure the cluster is healthy and ready for Spark jobs.
Prediction: Predict the speedup of running a Spark application with the cuDF plugin on GPUs.
Train: Train a model to predict the performance of a Spark job with the cuDF plugin. The output shows the model file that can be used to predict the performance of a Spark job.

Getting started

Set up a Python environment with a version between 3.10 and 3.12

Run the project in a virtual environment. Note, .venv is the directory created to put the virtual env in, so modify if you want a different location.
```
$ python -m venv .venv
$ source .venv/bin/activate
```
Install spark-rapids-user-tools
- Using released package.
```
$ pip install spark-rapids-user-tools
```
- Install from source.
```
$ pip install -e .
```
  Note:
  - To install dependencies required for running unit tests, use the optional test parameter: pip install -e '.[test]'
  - To install dependencies required for QualX training, use the optional qualx parameter pip install -e '.[qualx]'
  - To install all the required dependencies, use the optional dev-env parameter: pip install -e '.[dev-env]'
- Using wheel package built from the repo (see the build steps below).
```
$ pip install <wheel-file>
```
Make sure to install CSP SDK if you plan to run the tool wrapper.

Building from source

Set up a Python environment similar to the steps above.

Create a virtual environment. Note, .venv is the directory created to put the virtual env in, so modify if you want a different location.
```
$ python -m venv .venv
$ source .venv/bin/activate
```
Run the provided build script to compile the project.
```
$> ./build.sh
```
Fat Mode: Similar to fat jar in Java, this mode solves the problem when web access is not available to download resources having Url-paths (http/https). The command builds the tools jar file and downloads the necessary dependencies and packages them with the source code into a single 'wheel' file.
```
$> ./build.sh fat
```

Logging Configuration

The core tools project uses Log4j for logging. Default log level is set to INFO. You can configure logging settings in the log4j.properties file located in the src/spark_rapids_pytools/resources/dev/ directory. This is applicable when you clone the project and build it from source. To change the logging level, modify the log4j.rootLogger property. Possible levels include DEBUG, INFO, WARN, ERROR.

Usage and supported platforms

Please refer to spark-rapids-user-tools guide for details on how to use the tools and the platform.

Please refer to qualx guide for details on how to use the QualX tool for prediction and training.

What's new

Please refer to CHANGELOG.md for our latest changes.

Release files for spark-rapids-user-tools 26.8.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for spark-rapids-user-tools 26.8.2
File	Interpreter	ABI	Platform	Reset
spark_rapids_user_tools-26.8.2-471_a0c0d63f-py3-none-any.whl 5.5 MB Sep 15, 2026	Python 3	none	any	Details

Release files / spark_rapids_user_tools-26.8.2-471_a0c0d63f-py3-none-any.whl

Download URL	spark_rapids_user_tools-26.8.2-471_a0c0d63f-py3-none-any.whl
Size	5.5 MB
Tags	Python 3
SHA-256 checksum How to use checksums	`845cb3f09b0bece0195768ddde7d4739415d61b8b5ade36bec29cf822c0a682e`
BLAKE2b-256 checksum How to use checksums	`32323329eee0f15264bd0f29eb2ea8956c8e09f61b11bb4b1170b01d7f51b35b`
Upload date	Sep 15, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.10.17`