D3DTOOLS

A collection of Python tools for working with shapefiles and converting them for Delft3D modeling.

CAUTION: The ncrain function currently only works for Taiwan data in EPSG:3826 projection.

GDAL Installation: GDAL is required for this package. For conda environments, use conda install gdal to install GDAL. For non-conda environments, download the appropriate wheel file from https://github.com/cgohlke/geospatial-wheels/releases to install GDAL.

Installation

pip install d3dtools

Features

This package provides several utilities for converting shapefiles to various formats used in Delft3D modeling:

• ncrain: Generate a NetCDF file from rainfall data and thiessen polygon shapefiles
• snorain: Process rainfall scenario data and generate time series CSV files
• shp2ldb: Convert boundary line shapefiles to LDB files
• shpbc2pli (alias: shp2pli): Convert boundary line shapefiles to PLI files
• shpblock2pol (alias: shp2pol): Convert shapefile blocks to POL files
• shpdike2pliz (alias: shp2pliz): Convert bankline shapefiles to PLIZ files
• shp2xyz: Convert point shapefiles to XYZ files
• evaluate: Calculate flood simulation accuracy metrics by comparing simulated and observed flood extents
• evaluate_sensor: Calculate flood simulation accuracy metrics by comparing simulated flood extents with point-based sensor data (with configurable buffer radius and depth threshold)
• evaluate_sensor2 (alias: eval_iot): Calculate flood simulation accuracy metrics using sensor data with dual-threshold shapefiles (separate low and high depth threshold simulations)
• sensor: Extract time series data from Delft3D FM NetCDF files at observation points
• getfacez: Extract Mesh2d_face_z values (bed level/bathymetry) from Delft3D FM NetCDF files at observation points. Uses a spatial index (shapely STRtree for point-in-polygon matching, scipy cKDTree for nearest-neighbor matching) instead of scanning every mesh face for every observation point, which is much faster on large meshes. Supports -if/--id-field to specify which shapefile field to use for point names
• getfacez2: Original brute-force implementation of getfacez (no spatial index), kept as a fallback. Same CLI arguments, Python API, and output format as getfacez, including -if/--id-field
• fou2shp: Reconstruct Delft3D FM 2D mesh face polygons from a FOU (Fourier) NetCDF output file and export threshold-filtered shapefiles; supports -r/--remove to remove output polygons that intersect mask shapefiles (filtered copies written to <output-folder>_RM/)
• pliz2shp: Convert Delft3D/D-Flow FM .pliz weir/dike polyline files (with Z) to 3D ESRI Shapefiles
• pli2shp: Convert Delft3D polyline files (.pli/.ldb) to ESRI Shapefiles
• pol2shp: Convert Delft3D/D-Flow FM .pol polygon files to ESRI Shapefiles
• xyz2shp: Convert XYZ point files (.xyz/.csv) to ESRI Shapefiles
• rmgrid: Remove (clear) the 2D computational mesh and 1D2D links from a D-Flow FM .dsproj project while preserving the 1D network (pipes/branches)
• rsgrid: Restore the 2D computational mesh (including Mesh2d_face_z bed levels) into a D-Flow FM .dsproj project by cloning it from a source project, while preserving the target's 1D network. The inverse of rmgrid

Usage Examples

Process and generate rainfall scenario data

from d3dtools import snorain

# Process a scenario rainfall CSV file
snorain.generate(
    input_file='rainfall_scenarios.csv',
    output_folder='custom/TAB',
    verbose=True
)

Generate NetCDF from rainfall data (with unit of mm/hr)

from d3dtools import ncrain

# Default usage - processes first CSV file in the input folder
ncrain.generate()

# With custom parameters
ncrain.generate(
    input_shp_folder='custom/SHP',
    input_tab_folder='custom/TAB',
    output_nc_folder='custom/NC',
    intermediate_ras_folder='custom/RAS_RAIN',
    intermediate_shp_folder='custom/SHP_RAIN',
    clean_intermediate=True,
    raster_resolution=320
)

# Process a specific CSV file
ncrain.generate(
    input_tab_folder='custom/TAB',
    rainfall_file='specific_rainfall.csv',
    verbose=True
)

# Process all CSV files in the input folder
ncrain.generate_all(
    input_shp_folder='custom/SHP',
    input_tab_folder='custom/TAB',
    output_nc_folder='custom/NC',
    verbose=True
)

Convert boundary shapefiles to PLI

from d3dtools import shpbc2pli

# Default usage
shpbc2pli.convert()

# With custom parameters
shpbc2pli.convert(
    input_folder='custom/SHP_BC',
    output_folder='custom/PLI_BC'
)

# With custom ID field name
shpbc2pli.convert(
    input_folder='custom/SHP_BC',
    output_folder='custom/PLI_BC',
    id_field='BoundaryName'  # Use 'BoundaryName' column instead of default 'ID'/'Id'/'id'/'iD'
)

Convert block shapefiles to POL

from d3dtools import shpblock2pol

# Default usage
shpblock2pol.convert()

# With custom parameters
shpblock2pol.convert(
    input_folder='custom/SHP_BLOCK',
    output_folder='custom/POL_BLOCK'
)

Convert dike shapefiles to PLIZ

from d3dtools import shpdike2pliz

# Default usage
shpdike2pliz.convert()

# With custom parameters
shpdike2pliz.convert(
    input_folder='custom/SHP_DIKE',
    output_folder='custom/PLIZ_DIKE',
    output_filename='CustomDike'
)

# With custom ID field name
shpdike2pliz.convert(
    input_folder='custom/SHP_DIKE',
    output_folder='custom/PLIZ_DIKE',
    output_filename='CustomDike',
    id_field='DikeName'  # Use 'DikeName' column instead of default 'ID'/'Id'/'id'/'iD'
)

Convert boundary shapefiles to LDB

from d3dtools import shp2ldb

# Default usage
shp2ldb.convert()

# With custom parameters
shp2ldb.convert(
    input_folder='custom/SHP_LDB',
    output_folder='custom/LDB'
)

# With custom ID field name
shp2ldb.convert(
    input_folder='custom/SHP_LDB',
    output_folder='custom/LDB',
    id_field='BoundaryName'  # Use 'BoundaryName' column instead of default 'ID'/'Id'/'id'/'iD'
)

Convert point shapefiles to XYZ

from d3dtools import shp2xyz

# Default usage
shp2xyz.convert()

# With custom parameters
shp2xyz.convert(
    input_folder='custom/SHP_SAMPLE',
    output_folder='custom/XYZ_SAMPLE'
)

# With custom Z-field name
shp2xyz.convert(
    input_folder='custom/SHP_SAMPLE',
    output_folder='custom/XYZ_SAMPLE',
    z_field='ELEVATION'  # Use 'ELEVATION' column instead of default Z-field detection
)

Extract time series data from NetCDF files

from d3dtools import sensor

# Extract data from NetCDF file at observation points
data = sensor.getdata(
    nc_file='path/to/model_output.nc',
    obs_shp='path/to/observation_points.shp',
    output_csv='water_depth.csv',
    output_excel='water_depth.xlsx',
    plot=True  # Display a plot of the time series
)

# Process the data further if needed
print(data.head())
stats = data.describe().transpose()
print(stats)

Extract Mesh2d_face_z values from NetCDF files (spatial-index accelerated)

from d3dtools import getfacez

# Extract bed level/bathymetry data from NetCDF file at observation points.
# Uses an STRtree (point-in-polygon) or cKDTree (nearest neighbor) spatial index
# instead of a per-point full mesh scan, so it stays fast on large meshes.
data = getfacez.extract_mesh2d_face_z(
    nc_file='path/to/model_output.nc',
    obs_shp='path/to/observation_points.shp',
    output_csv='bathymetry.csv',
    output_excel='bathymetry.xlsx',
    id_field='StationName',  # Optional; field to use for point names (default: auto-detect)
    verbose=True  # Display additional information during processing
)

# Process the data further if needed
print(data.head())
print(f"Bathymetry range: {data['Mesh2d_face_z'].min():.3f} to {data['Mesh2d_face_z'].max():.3f}")

Extract Mesh2d_face_z values from NetCDF files (original brute-force fallback)

from d3dtools import getfacez2

# Same signature and output as getfacez, but uses the original per-point full mesh
# scan (no spatial index). Kept as a fallback in case the spatial-index approach
# ever misbehaves on unusual mesh data.
data = getfacez2.extract_mesh2d_face_z(
    nc_file='path/to/model_output.nc',
    obs_shp='path/to/observation_points.shp',
    output_csv='bathymetry.csv',
    output_excel='bathymetry.xlsx',
    id_field='StationName',  # Optional; field to use for point names (default: auto-detect)
    verbose=True  # Display additional information during processing
)

# Process the data further if needed
print(data.head())
print(f"Bathymetry range: {data['Mesh2d_face_z'].min():.3f} to {data['Mesh2d_face_z'].max():.3f}")

Calculate flood simulation accuracy using sensor data

from d3dtools import evaluate_sensor

# Compare simulated flood extents with sensor observations
results = evaluate_sensor.confusion_matrix(
    sim_path='path/to/simulated_flood.shp',
    obs_path='path/to/sensor_observations.shp',
    buffer_radius=30,               # Buffer radius around sensor points in meters (default: 30)
    depth_threshold=30,             # Water depth threshold in centimeters (default: 30)
    output_csv='sensor_accuracy.csv'
)

print(f"Accuracy: {results['accuracy']:.2f}%")
print(f"Recall (Catch Rate): {results['recall']:.2f}%")

Calculate flood simulation accuracy using sensor data with dual thresholds

from d3dtools import evaluate_sensor2

# Compare simulated flood extents (low/high threshold) with sensor observations
results = evaluate_sensor2.confusion_matrix(
    low_threshold_sim_path='path/to/simulated_flood_low.shp',
    high_threshold_sim_path='path/to/simulated_flood_high.shp',
    obs_path='path/to/sensor_observations.shp',
    buffer_radius=30,               # Buffer radius around sensor points in meters (default: 30)
    depth_threshold=30,             # Water depth threshold in centimeters (default: 30)
    output_csv='sensor_accuracy2.csv'
)

print(f"Accuracy: {results['accuracy']:.2f}%")
print(f"Recall (Catch Rate): {results['recall']:.2f}%")

Reconstruct FOU mesh faces as threshold shapefiles

# Run via command line (recommended)
# fou2shp --input NC/FlowFM_fou.nc -of SHP
# fou2shp --input NC/FlowFM_fou.nc --var Mesh2d_fourier002_max_depth --output-folder output

# Remove polygons intersecting a mask shapefile; filtered copies go to SHP_RM/
# fou2shp --input NC/FlowFM_fou.nc -r SHP/EXCLUDE.shp
# fou2shp --input NC/FlowFM_fou.nc -r SHP/*.shp
# fou2shp --input NC/FlowFM_fou.nc --remove SHP/ROAD.shp SHP/BUILDING.shp

Convert PLIZ files to Shapefiles

from d3dtools import pliz2shp

# Convert a single .pliz file
pliz2shp.pliz_to_shp(
    input_file='PLIZ/MyDike.pliz',
    output_dir='SHP_LINES3D',       # Optional; default: SHP_LINES3D
    crs='EPSG:3826'                 # Optional; default: EPSG:3826
)

# Batch convert via CLI (recommended for multiple files)
# pliz2shp -i Dike001.pliz
# pliz2shp -if custom/PLIZ -of custom/SHP

Convert PLI/LDB files to Shapefiles

from d3dtools import pli2shp

# Convert a single .pli or .ldb file
pli2shp.polyline_to_shp(
    input_file='PLI/boundary.pli',
    output_dir='SHP_LINES',         # Optional; default: SHP_LINES
    crs='EPSG:3826'                 # Optional; default: EPSG:3826
)

# Batch convert via CLI (recommended for multiple files)
# pli2shp -i boundary.pli
# pli2shp -if custom/PLI -of custom/SHP

Convert POL files to Shapefiles

from d3dtools import pol2shp

# Convert a single .pol file
pol2shp.pol_to_shp(
    input_file='POL/POL_001.pol',
    output_dir='SHP_POLYGONS',      # Optional; default: SHP_POLYGONS
    crs='EPSG:3826'                 # Optional; default: EPSG:3826
)

# Batch convert via CLI (recommended for multiple files)
# pol2shp -i POL_001.pol
# pol2shp -if custom/POL -of custom/SHP

Convert XYZ/CSV point files to Shapefiles

from d3dtools import xyz2shp

# Convert a single .xyz or .csv point file
xyz2shp.xyz_to_shp(
    input_file='XYZ/XYZ_001.xyz',
    output_dir='SHP_XYZ',           # Optional; default: SHP_XYZ
    crs='EPSG:3826',                # Optional; default: EPSG:3826
    dimension='3'                   # Optional; '3' for x,y,z points, '2' for x,y only
)

# Batch convert via CLI (recommended for multiple files)
# xyz2shp -i XYZ_001.xyz
# xyz2shp -if custom/XYZ -of custom/SHP

Remove the 2D mesh from a D-Flow FM project

# Recommended usage via the command line (operates on a .dsproj project)
# rmgrid                                  # Auto-detect the .dsproj in the current folder
# rmgrid -i MyProject.dsproj              # Specify the project explicitly
# rmgrid -i MyProject.dsproj --force-backup  # Overwrite an existing .nc.bak
# rmgrid -i MyProject.dsproj --restore    # Restore the original net file from .nc.bak

The tool empties the 2D mesh in the project's UGRID NetCDF net file while preserving the 1D network (pipes/branches), strips 2D-specific blocks from the IniFieldFile, and creates a <name>.nc.bak backup so the change can be reverted with --restore.

Restore the 2D mesh into a D-Flow FM project

# Recommended usage via the command line (operates on .dsproj projects)
# rsgrid -s Intact.dsproj                   # Restore into first .dsproj in cwd
# rsgrid -i Stripped.dsproj -s Intact.dsproj
# rsgrid -i target_net.nc -s source_net.nc

The tool clones the 2D mesh (including Mesh2d_face_z bed levels) from a source project's net file into the target's net file, keeping the target's own 1D network, coordinate system, and other settings intact. It backs up the target net file with a timestamped copy before overwriting. This is the inverse of rmgrid.

Calculate flood simulation accuracy

from d3dtools import evaluate

# Compare simulated and observed flood extents
results = evaluate.confusion_matrix(
    sim_path='path/to/simulated_flood.shp',
    obs_path='path/to/observed_flood.shp',
    output_path='accuracy_results.csv'
)

print(f"Accuracy: {results['accuracy']:.2f}%")
print(f"Recall (Catch Rate): {results['recall']:.2f}%")

Command-line Usage

d3dtools-info: Access Tool Information

The package provides the d3dtools-info command-line utility that serves as a central information hub for all available tools:

# Display the package version
d3dtools-info --version
d3dtools-info -v

# Get help on d3dtools-info itself
d3dtools-info --help

# Display description of all available tools
d3dtools-info

# Display detailed information about a specific tool
d3dtools-info ncrain
d3dtools-info snorain
d3dtools-info shp2ldb
d3dtools-info shp2pli
d3dtools-info shp2pliz
d3dtools-info shp2pol
d3dtools-info shp2xyz
d3dtools-info shpbc2pli
d3dtools-info shpblock2pol
d3dtools-info shpdike2pliz
d3dtools-info sensor
d3dtools-info evaluate
d3dtools-info evaluate_sensor
d3dtools-info evaluate_sensor2
d3dtools-info eval_iot
d3dtools-info getfacez
d3dtools-info getfacez2
d3dtools-info fou2shp
d3dtools-info pliz2shp
d3dtools-info pli2shp
d3dtools-info pol2shp
d3dtools-info xyz2shp
d3dtools-info rmgrid
d3dtools-info rsgrid

# Display help for specific tools
ncrain --help
snorain --help
shp2ldb --help
shp2pli --help
shp2pliz --help
shp2pol --help
shp2xyz --help
shpbc2pli --help
shpblock2pol --help
shpdike2pliz --help
sensor --help
evaluate --help
evaluate_sensor --help
evaluate_sensor2 --help
eval_iot --help
getfacez --help
getfacez2 --help
fou2shp --help
pliz2shp --help
pli2shp --help
pol2shp --help
xyz2shp --help
rmgrid --help
rsgrid --help

The d3dtools-info tool helps you discover available functionality, learn about tool options, and access usage examples without having to remember all command-line parameters.

The package also provides command-line utilities for each specific tool:

# Generate NetCDF from rainfall data
ncrain                      # Process all CSV files in the input folder
ncrain --shp-folder custom/SHP --tab-folder custom/TAB --nc-folder custom/NC --resolution 320
ncrain --verbose            # Display additional processing information
ncrain --no-clean           # Keep intermediate files
ncrain --single rainfall.csv  # Process only a specific CSV file

# Process rainfall scenario data
snorain -i rainfall_scenarios.csv -of custom/TAB
snorain --input rainfall_scenarios.csv --output-folder custom/TAB --verbose

# Convert boundary shapefiles to LDB
shp2ldb
shp2ldb -i custom/SHP_LDB -of custom/LDB  # Specify input and output folders
shp2ldb --id_field BoundaryName  # Specify custom ID field

# Convert boundary shapefiles to PLI
shpbc2pli  # or use the alias: shp2pli
shpbc2pli --id_field BoundaryName  # Specify custom ID field

# Convert block shapefiles to POL
shpblock2pol  # or use the alias: shp2pol
shpblock2pol -i custom/SHP_BLOCK -of custom/POL_BLOCK  # Specify input and output folders

# Convert dike shapefiles to PLIZ
shpdike2pliz  # or use the alias: shp2pliz
shpdike2pliz --id_field DikeName  # Specify custom ID field

# Convert point shapefiles to XYZ
shp2xyz
shp2xyz -i custom/SHP_SAMPLE -of custom/XYZ_SAMPLE  # Specify input and output folders
shp2xyz --z_field ELEVATION  # Specify custom Z-field name

# Extract time series data at observation points
sensor --nc-file path/to/model_output.nc --obs-shp path/to/observation_points.shp
sensor --nc-file path/to/model_output.nc --obs-shp path/to/observation_points.shp --output-csv water_depth.csv --output-excel water_depth.xlsx --plot
sensor --verbose  # Display additional processing information

# Calculate flood simulation accuracy metrics
evaluate --sim path/to/simulated_flood.shp --obs path/to/observed_flood.shp
evaluate --sim path/to/simulated_flood.shp --obs path/to/observed_flood.shp --output accuracy_results.csv

# Calculate flood simulation accuracy using sensor data
evaluate_sensor --sim path/to/simulated_flood.shp --obs path/to/sensor_points.shp
evaluate_sensor --sim path/to/simulated_flood.shp --obs path/to/sensor_points.shp --buffer 30 --threshold 30 --output sensor_accuracy.csv

# Calculate flood simulation accuracy using sensor data with dual-threshold shapefiles
evaluate_sensor2 --sim-low SHP/SIM_thrd125.shp --sim-high SHP/SIM_thrd475.shp --obs SHP/OBS_SENSOR.shp
evaluate_sensor2 --sim-low SHP/SIM_thrd125.shp --sim-high SHP/SIM_thrd475.shp --obs SHP/OBS_SENSOR.shp --buffer 50 --threshold 20 --output sensor_accuracy2.csv
eval_iot --sim-low SHP/SIM_thrd125.shp --sim-high SHP/SIM_thrd475.shp --obs SHP/OBS_SENSOR.shp  # Alias for evaluate_sensor2
eval_iot --sim-low SHP/SIM_thrd125.shp --sim-high SHP/SIM_thrd475.shp --obs SHP/OBS_SENSOR.shp --buffer 30 --threshold 20 --output sensor_accuracy2.csv

# Extract Mesh2d_face_z values at observation points (spatial-index accelerated)
getfacez --nc-file path/to/model_output.nc --obs-shp path/to/observation_points.shp
getfacez --nc-file path/to/model_output.nc --obs-shp path/to/observation_points.shp --output-csv bathymetry.csv --output-excel bathymetry.xlsx
getfacez --nc-file path/to/model_output.nc --obs-shp path/to/observation_points.shp -if StationName  # Specify custom id field
getfacez --verbose  # Display additional processing information

# Extract Mesh2d_face_z values at observation points (original brute-force fallback)
getfacez2 --nc-file path/to/model_output.nc --obs-shp path/to/observation_points.shp
getfacez2 --nc-file path/to/model_output.nc --obs-shp path/to/observation_points.shp --output-csv bathymetry.csv --output-excel bathymetry.xlsx
getfacez2 --nc-file path/to/model_output.nc --obs-shp path/to/observation_points.shp -if StationName  # Specify custom id field
getfacez2 --verbose  # Display additional processing information

# Reconstruct FOU mesh faces as threshold-filtered shapefiles
fou2shp                                         # Use defaults (NC/FlowFM_fou.nc -> SHP/)
fou2shp --input NC/FlowFM_fou.nc -of SHP        # Specify input and output directory
fou2shp --input NC/FlowFM_fou.nc --var Mesh2d_fourier002_max_depth --output-folder output
fou2shp --input NC/FlowFM_fou.nc -r SHP/EXCLUDE.shp             # Remove polygons intersecting a mask; output -> SHP_RM/
fou2shp --input NC/FlowFM_fou.nc -r SHP/*.shp                   # Glob pattern for multiple masks
fou2shp --input NC/FlowFM_fou.nc --remove SHP/ROAD.shp SHP/BUILDING.shp  # Multiple explicit masks

# Convert a Delft3D/D-Flow FM .pliz file (weir/dike polyline with Z) to a 3D ESRI Shapefile
pliz2shp -i Dike001.pliz
pliz2shp -i Dike001.pliz -of output --crs EPSG:4326  # Specify output folder and CRS
pliz2shp -if custom/PLIZ -of custom/SHP              # Convert every .pliz file in a folder
pliz2shp --help

# Convert a Delft3D polyline file (.pli/.ldb) to an ESRI Shapefile
pli2shp -i boundary.pli
pli2shp -i LDB_001.ldb -of output --crs EPSG:4326
pli2shp -if custom/PLI -of custom/SHP
pli2shp --help

# Convert a Delft3D/D-Flow FM .pol file to a polygon ESRI Shapefile
pol2shp -i POL_001.pol
pol2shp -i POL_001.pol -of output --crs EPSG:4326
pol2shp -if custom/POL -of custom/SHP
pol2shp --help

# Convert an XYZ/CSV point file to an ESRI Shapefile
xyz2shp -i XYZ_001.xyz
xyz2shp -i XYZ_001.csv -of output --crs EPSG:4326
xyz2shp -i XYZ_001.xyz -d 2                         # Write 2D (x,y) points instead of 3D
xyz2shp -if custom/XYZ -of custom/SHP
xyz2shp --help

# Remove the 2D computational mesh from a D-Flow FM .dsproj project
rmgrid                                # Auto-detect the .dsproj in the current folder
rmgrid -i MyProject.dsproj            # Specify the project explicitly
rmgrid -i MyProject.dsproj --force-backup  # Overwrite an existing .nc.bak
rmgrid -i MyProject.dsproj --restore  # Restore the original net file from .nc.bak

# Restore the 2D computational mesh into a D-Flow FM .dsproj project
rsgrid -s Intact.dsproj                   # Restore into first .dsproj in cwd
rsgrid -i Stripped.dsproj -s Intact.dsproj # Specify target and source explicitly
rsgrid -i target_net.nc -s source_net.nc  # Operate directly on net files

Changelog

See CHANGELOG.md for the full version history.

Requirements

• numpy>=1.20.0
• pandas>=1.3.0
• geopandas>=0.10.0
• rasterio>=1.2.0
• netCDF4>=1.5.0
• pyproj>=3.0.0
• shapely>=2.0.0
• scipy>=1.7.0
• matplotlib>=3.4.0
• openpyxl>=3.0.0

License

MIT