whitebox-workflows 0.9.8

whitebox_workflows is a Python library for advanced spatial analysis.

Whitebox Workflows for Python

What is Whitebox Workflows?

Whitebox Workflows (WbW) is a Python library for advanced geoprocessing, including many GIS and remote sensing analysis operations for manipulating common types of raster, vector and lidar geospatial data.

Why Whitebox Workflows when there already is WhiteboxTools Open Core?

Whitebox Workflows (WbW) is based on the WhiteboxTools Open Core (WbOC) open-source codebase. While the two products share many characteristics and functionality, there are important differences.

The WhiteboxTools Open Core is a command line back-end program that interfaces with various front-end applications, such as QGIS, ArcGIS and the R and Python scripting languages. Front-end/back-end communication is very limited. Front-ends can only communicate with WbOC by passing text-based commands and receive text-based outputs. Data files are provided as file names and are read into memory during tool operation and output data are written to disc. This design allows WbOC to be readily integrated into other projects. However, it doesn't allow front-ends to directly interact with Whitebox data, and it isn't well suited to longer geoprocessing workflows. Tools in a WbOC based geoprocessing script are essentially run independently of other tools in the same workflow.

By comparison, Whitebox Workflows is a native Python extension library; it has been designed to work with Python, providing a geoprocessing scripting environment. Like the open-core, WbW is developed using the fast programming language Rust, but is compiled to a shared library that can be directly imported into Python scripts, much like NumPy and other common Python scientific libraries.

Each of the more than 400 geoprocessing tools that users love about the WbOC are also found in WbW. The library design of WbW affords a much more intimate level of communication between it and your Python geoprocessing script. For instance, with WbW you can directly manipulate raster, vector, and lidar data objects, to perform low-level geoprocessing in a way that is impossible with the open-core. For example, below we manipulate raster data directly in Python using WbW:

import whitebox_workflows

wbw = whitebox_workflows.WbEnvironment()
dem = wbw.read_raster('/path/to/data/dem.tif')
high_areas = wbw.new_raster(dem.configs)

print("Finding high elevations")

for row in range(dem.configs.rows):
  for col in range(dem.configs.columns):
    elev = dem[row, col]
    if elev != dem.configs.nodata and elev > 1000.0:
      high_areas[row, col] = 1.0
    else:
      high_areas[row, col] = 0.0

wbw.write_raster(high_areas, 'high_areas.tif', compress=True)

Where tools in the open-core take file name strings as inputs, the WbW equivalent functions take in-memory geospatial objects as input parameters. WbW functions also return output objects. This means that for a typical geoprocessing workflow there is significantly less reading/writing of data to the disc. There is no performance cost incurred by read/write operations during the intermediate processing. WbW has been designed to meet enterprise-scale geoprocessing workflow needs.

The following example Python script interpolates a lidar file to a digital elevation model (DEM), performs some common pre-processing steps on the DEM, and then runs a flow accumulation operation, before outputting the flow accumulation grid to file:

import whitebox_workflows

wbw = whitebox_workflows.WbEnvironment()

lidar = wbw.read_lidar('/path/to/data/myfile.laz')
dem = wbw.lidar_tin_gridding(input_lidar=lidar, returns_included='last', cell_size=1.0)
dem_nodata_filled = wbw.fill_missing_data(dem, filter_size=21)
dem_breached = wbw.breach_depressions_least_cost(dem_nodata_filled, fill_deps=True)
dem_smoothed = wbw.feature_preserving_smoothing(dem_breached, filter_size=15)
flow_accum = wbw.dinf_flow_accum(dem_smoothed, log_transform=True)
wbw.write_raster(flow_accum, "flow_accumulation_grid.tif", compress=True)

Notice how each of the five tool functions return data objects that then serve as the inputs for later operations. While there's only one read operation and one write operation in the script, an equivalent WbOC workflow would result in 10 individual read/write operations. This characteristic can result in significant gains in overall workflow performance. It is often the case that read/write operations can be the bottle-neck in geoprocessing performance. Fewer read/write operations also means significantly less wear on your hardware.

The design of WbW also allows for more natural geoprocessing of data objects. For example, rather than using individual raster math tools (e.g. Add, Divide, Sin etc.), with WbW, you can often treat raster objects like any other numerical variables in scripts--with WbW, Python becomes your raster calculator!

new_raster = 1.0 - (raster1 - raster2) / (raster1 + raster2)
new_raster = dem > 1000.0   # Note: This single line is equivalent to one of the example Python scripts above
new_raster = raster.sin().to_degrees()
new_raster = raster1.max(raster2)   # You can use a number instead of a raster as the parameter, e.g. raster.max(100.0)

Overall, if you're using the Whitebox platform to develop Python scripts for geoprocessing tasks, Whitebox Workflows is the clear winner. It provides easier-to-write and faster-running scripting with less strain on your expensive hardware. Simply put, it's a more productive geoprocessing environment.

There is, however, one small downside to using WbW over WbOC. Developing WbW was not a matter of simply compiling the existing WbOC codebase as a library; it took a substantial development effort to create this great product. Whitebox Workflows is not free. You need to purchase a valid license activation code to use WbW. The good news is, annual licenses for WbW are very reasonably priced--only about $10. We want as many people using this wonderful product as possible!

Installation

If you have Python installed on your machine, simply type pip install whitebox-workflows at the command prompt. At the moment, Windows (64-bit) and Mac (Intel and ARM) is supported. Linux support will follow soon.

If you have installed whitebox-workflows Python package before and want to upgrade to the latest version, you can use the following command:

pip install whitebox-workflows -U

It is recommended that you use a Python virtual environment to test the whitebox-workflows package.

Usage

import whitebox_workflows

##########################
# Set up the environment #
##########################
wbw = whitebox_workflows.WbEnvironment() # A WbEnvironment object is needed to read/write data and run tools.
wbw.verbose = True # Determines if tools output to std::out
wbw.max_procs = -1
wbw.working_directory = '/path/to/my/data'

############################
# Read some data from disc #
############################
dem = wbw.read_raster('DEM_5m.tif')
points = wbw.read_vector('points.shp')
lidar = wbw.read_lidar('my_lidar_tile.laz')

######################
# Run some functions #
#######################
hillshade_raster = wbw.hillshade(dem, azimuth=270.0)
pointer = wbw.d8_pointer(dem)
watersheds = wbw.watershed(pointer, points)

###########################
# Write some data to disc #
###########################
wbw.write_raster(hillshade_raster, 'hillshade.tif', compress=True)
wbw.write_raster(watersheds, 'watersheds.tif', compress=True)

######################
# Raster map algebra #
######################
elev_in_ft = dem * 3.28084
high_in_watershed = (dem > 500.0) * (watersheds == 1.0)
tan_elev = dem.tan()
dem += 10.0 
raster3 = raster1 / raster2

###############################
# Manipulate lidar point data #
###############################
import whitebox_workflows

wbw = whitebox_workflows.WbEnvironment()
lidar = wbw.read_lidar('/path/to/data/lidar_tile.laz')
lidar_out = wbw.new_lidar(lidar.header) # Create a new file

print('Filtering point data...')
for a in range(lidar.header.number_of_points):
    (point_data, time, colour, waveform) = lidar.get_point_record(a)
    if point_data.is_first_return() or point_data.is_intermediate_return():
        lidar_out.add_point(point_data, time)

wbw.write_lidar(lidar_out, "new_lidar.laz")