Skip to main content

ras-commander-fast-dss

A native HEC-DSS engine for Python, written in Rust. It reads and writes DSS version 6 and version 7 files (catalogs, regular and irregular time series, paired data, grids, profiles and location records) without Java, HEC Monolith or any HEC native library. It also imports gridded data (GRIB2, NCEP Stage IV GRIB1, NetCDF, GeoTIFF/ASCII rasters, PRISM BIL and ZIP packages, SNODAS and Zarr v2) into DSS grids, the job HEC-Vortex does for HEC-HMS and HEC-RAS.

ras-commander-fast-dss is the DSS backend of RAS Commander and hms-commander. It is called FAST-DSS on the RAS Commander website.

pip install ras-commander-fast-dss

Wheels are abi3 (one wheel covers CPython 3.10 and later) for Windows x86_64, for Linux x86_64 and aarch64 (manylinux and musllinux), and, from 0.1.3, for macOS arm64 and x86_64 (separate wheels, not universal2). Routine CI builds and tests on Linux and Windows. The macOS wheels are built and tested in the release workflow on macOS 15, and the engine's killed-process matrix was qualified on macOS 15 arm64 and x86_64 before release. The source distribution is published with every release: other platforms can build it with a Rust toolchain; the build has no C dependencies.

On macOS, the extension module itself targets macOS 11.0 (arm64) and 10.12 (x86_64) and links only system libraries. Its dependencies set their own requirements: recent numpy releases publish macOS wheels for macOS 14 and later, and so does your Python build. On an older macOS, pip may choose an older compatible numpy or pandas, or none; only macOS 15 is tested.

Quick example

import numpy as np
import ras_commander_fast_dss as dss

with dss.open("model.dss") as f:
    print(f.version, f.version_string)        # 7 '7-IU'
    catalog = f.catalog()                     # DataFrame, one row per record
    flow = f.read_timeseries("/BASIN/LOC/FLOW//1HOUR/OBS/")

rating = dss.read_paired_data("model.dss", "/BASIN/LOC/STAGE-FLOW///RATING/")
grid = dss.read_grid("model.dss", "/SHG/BASIN/PRECIP/01JAN2020:0000/01JAN2020:0100/RADAR/")

# Write a regular time series into a new DSS7 file.
dss.create("out.dss", version=7)
dss.write_timeseries(
    "out.dss",
    "/BASIN/LOC/FLOW/01JAN2020/1HOUR/SIM/",
    np.array([10.0, 12.5, 15.0]),
    start="2020-01-01 01:00",
    units="CFS",
    data_type="INST-VAL",
)

Time series come back as pandas DataFrames with HEC missing markers mapped to NaN; paired data and grids come back as NumPy arrays with their metadata. Errors are typed (ras_commander_fast_dss.exceptions), and a file that HEC software holds open for writing is refused rather than written (DssLockedError). The gridded importer and tools are in ras_commander_fast_dss.gridded.

Interrupted writes, sync=, DssFile sessions and repair

Writers order their steps so that a DSS7 or DSS6 file that was consistent before a write can be reconciled after the writing process dies part-way through; the next native write does this from the file itself. The crash model is process death, not power loss. The format notes (dss7.md, dss6.md) give the exact steps, what is and is not guaranteed, and the known limitations.

  • sync= on every writer (write_timeseries, write_paired_data, write_grid, write_profile, write_location, create, repair, the gridded importer, the grid tools and gridded.grid_to_point) chooses when the write calls fsync:

    • "end" (the default since 0.1.2; True means the same): one flush at the end of each write call. A call that stores many records flushes once, after the last.
    • "step": a flush at every flush point between the steps of the write, the default up to 0.1.1.
    • "none" (False means the same): no flush.

    The three make the same writes in the same order, so the process-death recovery above is the same in every mode. No mode claims durability on power loss.

  • A DssFile keeps one read+write handle from its first write until it is closed, so consecutive writes through it do not open the file again. Every write still takes and releases the HEC lock, and the handle is checked against the file the path names before and after each write. Close the DssFile before renaming, replacing or deleting its file: a write that loses a race with such a change raises OSError, and on Windows os.replace onto a file held by a session fails.

  • write_batch(path, records) and DssFile.write_batch(records) store many time-series and paired-data records in one call: one open, one lock test held until the last record, and with the default sync="end" one flush after the last record. Each record is written by the same transaction as its own write_timeseries / write_paired_data call, so a process killed during a batch is recovered as for separate calls.

  • create and repair build a new file under a unique temporary name beside the destination and give it its name by a hard link (or, on glibc Linux, an atomic no-replace rename), so a killed process leaves the destination absent or complete. Where neither is possible (FAT, exFAT, some network shares) they raise DssPublishError and create nothing; create the file on a filesystem with hard links and copy it.

  • repair(src, dest) (DSS7 only) copies the readable records of a damaged DSS7 file into a new file: every readable record that its documented discovery procedure reaches (dss7.md, copy-out repair). It does not find everything a catalog might: a bin block whose declared span does not fit the file is reported as unclassified rather than salvaged, while the catalog walk salvages the whole bins of such a partial block. src is opened read-only, each copied record is verified against the source, and an existing dest is never replaced. Always check the result: when the recovery is partial, complete is False and not_copied lists what was left out.

report = dss.repair("damaged.dss", "recovered.dss")
if not report["complete"]:
    print(report["not_copied"])

FEATURES lists optional capabilities by name, so callers can test for them. Flags added in 0.1.2: write_sync_modes, gridded_threads_all_cores, dssfile_held_handle, gridded_grib1_stage4, gridded_prism_bil, gridded_snodas and write_ts_dss6_irregular_pattern_opt_in. Added in 0.1.3: write_batch. The package version is ras_commander_fast_dss.__version__.

repair is for DSS7 files only. A DSS6 file is not repaired by a copy: the next native write to it reconciles an interrupted write in place.

DSS6 irregular patterns (record type 111) are written only with allow_unconfirmed_dss6_irregular_pattern=True, on a path or through a DssFile session, because no HEC reader has confirmed such records; without the flag the write is refused and the file is unchanged. Pathnames with bytes outside printable ASCII are refused for writing.

With ras-commander and hms-commander

RAS Commander reads DSS boundary data through RasDss. With this package installed, select the native backend:

from ras_commander import RasDss

RasDss.backend = "native"     # or set RAS_COMMANDER_DSS_BACKEND=native
catalog = RasDss.get_catalog("model.dss")

hms-commander uses the same package for its DSS reads. See the native engine documentation for the backend options and the differences from the Java path.

Qualification

The engine is a clean-room implementation of the DSS formats, written from HEC's MIT-licensed hec-dss source and the format notes in docs/format/, which cite the heclib file and function behind every rule. HEC software is used only as a test oracle; none of its code is copied, linked or shipped.

  • Reads were compared with HEC Monolith through ras-commander's Java path on the 53 public RasExamples DSS files (11,045 comparisons): every non-missing value is bit-identical and every missing position agrees; the remaining differences are documented corrections.
  • Writers are checked in this repository's tests against files written by HEC's own C library and by HEC Monolith (fixtures/public/): for the uncompressed records tested, the native writer, given the same input, produces word-identical records apart from write times and the program name. DSS6 regular time-series merges and extensions into existing records are checked against HEC Monolith and HEC's DSS6 C library. Uncompressed merged records are word-identical apart from addresses and write stamps. Where HEC stored a block compressed, HEC keeps it compressed and the native writer stores it uncompressed: their decoded values are compared, and the representation differs (addresses, stamps, KINDAT, KICOMP and KINCHE). Repeat-only compression (scheme 1) decodes to the same values. One deliberate value difference is documented: a merge into a block HEC stored with delta compression keeps the stored values outside the written window, where HEC re-quantizes the whole block (docs/format/dss6.md §9.4).
  • Gridded import is checked against HEC-Vortex 1.1.0 output on public NOAA, PRISM and NSIDC products and Vortex's own test resources, bit for bit where the fixtures say so. Where this package differs from Vortex on purpose, the difference is documented: SNODAS times (Vortex's timing is the explicit option snodas_times="vortex") and SNODAS missing cells (masked here, kept as numbers by Vortex). Two historical NOHRSC header conventions are imported as their headers declare them: melt and sublimation dated before 2019-02-05 (06:00 to 05:00 UTC) and precipitation dated before 2008-10-23 (07:00 to 06:00 UTC). These are 23-hour DSS periods, so consecutive daily records leave one hour uncovered; nothing is filled or shifted (see the CHANGELOG).
  • Hostile input: parsers bounds-check every offset and length and return errors; dss-core forbids unsafe code; fuzz and allocation-limit tests run in CI.

This is qualification for the files and versions named, not a claim that every possible DSS record is covered. Comparison with HEC software does not imply HEC or USACE endorsement.

Performance

Performance is qualified internally against HEC's DSS C library, pydsstools, hecdss, HEC Monolith and HEC-Vortex on a private corpus of production model files; those files and the detailed results are not published. The statements below summarize those evaluations. To measure on your own machine, run benchmarks/bench_public.py, which generates its own data and checks every result. Absolute times depend on storage, antivirus scanning and file layout.

  • Reads and catalogs are on par with HEC's C library and pydsstools. On very large DSS7 catalogs pydsstools is somewhat faster, because FAST-DSS validates and normalizes every pathname it returns.
  • Compared with the Java path (HEC Monolith) and hecdss, whole-dataset reads are many times faster, because FAST-DSS decodes records directly into NumPy arrays without a per-record Python or Java crossing.
  • Gridded import is on par with HEC-Vortex for continental grids and faster for smaller imports, with no JVM start-up. The grid tools and grid-to-point are slower than Vortex, because each output record is written separately.
  • Writes are durable by default and therefore slower. By default, FAST-DSS flushes to disk at the end of every write call, so that a killed process leaves a file HEC can read. The other tools leave flushing to the operating system. For bulk loads, write inside a DssFile session with sync="none". Bulk series writes are then on par with the C-based tools. Writing many small records or paired-data records one call at a time remains noticeably slower than the C libraries, because of per-call overhead; reducing it is planned.

Repository layout

Path Contents
crates/dss-core The format engine: DSS6 and DSS7 readers and writers (#![forbid(unsafe_code)])
crates/dss-vortex Gridded import and grid tools
crates/dss-py PyO3 binding, built with maturin into ras_commander_fast_dss._native
crates/dss-wasm Read-only WebAssembly binding
python/ras_commander_fast_dss The Python API
vendor/ hdf5-reader and netcdf-reader 0.12.1 with local patches (PATCHES.md)
fixtures/public/ Public and synthetic test files, each with its provenance in README.md
docs/format/ DSS6 and DSS7 format specifications
benchmarks/ A reproducible cross-tool benchmark on generated data (README.md)

Some comments, test names and synthetic fixture pathnames carry development labels (P20, P46, ...) that date when a rule or test was added. They are labels only: the rules themselves are stated in the code, the committed tests and docs/format/.

Build from source

cargo test --locked                                   # dss-core
cargo test --locked --workspace --exclude dss-py
python -m pip install maturin
python -m maturin build --release --locked --out dist
python -m pip install dist/*.whl pytest
python -I -m pytest                                   # tests/python against the installed wheel

dss-vortex needs Rust 1.92 or later; the other crates build with 1.80.

Attributions and acknowledgments

FAST-DSS (ras-commander-fast-dss) is an independent, pure-Rust implementation of the HEC-DSS file format, versions 6 and 7. It could not have been built without the open-source software and data published by the U.S. Army Corps of Engineers Hydrologic Engineering Center (HEC), and by the wider water-resources community.

How it was developed

The engine was written from HEC's published hec-dss C and Fortran source, which is MIT licensed. Every format rule in the code cites the HEC source file and routine it was derived from. No HEC code is copied into the engine, and no HEC library is linked or bundled: the engine has no C dependencies at build or run time.

HEC's own software served as the reference ("oracle") throughout development. Each reader and writer was compared against it on public example files, synthetic files, and a private corpus of real project files. Files written by the engine were also read back with HEC software:

  • HEC-DSS C library (hec-dss, through HEC's hecdss Python package): the reference for DSS7 reads and writes, and for file checks such as zcheckFile.
  • HEC Monolith, the Java DSS library used by HEC-RAS, HEC-HMS and HEC-DSSVue: the reference for DSS6, paired data, grids, and Java-path parity in ras-commander and hms-commander.
  • HEC-Vortex (MIT), versions 1.1.0 and 0.13.3: the reference for the gridded importer and grid tools. Its bundled PROJ 7.2.0 and GDAL 3.2.1 were the reference for projections and resampling.
  • HEC-DSSVue, HEC-RAS and HEC-HMS example data (public USACE data): the source of the real-data test fixtures.
  • HEC-RAS and HEC-HMS: model runs through ras-commander and hms-commander confirmed that the software reads DSS files written by the engine.

Independent validation with pydsstools

As a final check, outputs were validated with pydsstools (MIT, https://github.com/gyanz/pydsstools), a widely used third-party Python library for HEC-DSS. The comparison covered 4,727 records (21.6 million values) from 122 DSS6 and DSS7 files, covering regular and irregular time series, patterns, paired data, and HRAP, Albers and specified grids. 4,080 records were identical bit for bit and 182 more under a documented rule (for example, pydsstools returns paired data as float32). Of the 196 records that differed, 183 differed only in metadata. pydsstools could not read 209 records that the engine read, the engine could not read 1 record type that is out of its scope (irregular profiles), and 32 corrupt or zeroed records were unreadable by both. Every difference was traced to pydsstools, to behaviour HEC's own readers share, or to a documented engine convention, with HEC Monolith used to break ties. In the write tests, each library read back exactly every time-series and paired-data value the other wrote. The check found three minor engine issues, all in edge cases of files not written by HEC software and none changing a value read from a HEC-written file; all three were fixed before this release to follow HEC's behaviour (DSS6 units and type text, DSS6 specified-grid header byte counts, and the record types used for time-invariant grids).

Thanks

Thanks to the HEC developers for publishing hec-dss, HEC-Vortex and their example data as open source, and to the pydsstools maintainers for a library the industry relies on.

FAST-DSS is an independent open-source project. It is not affiliated with, endorsed by or supported by HEC or USACE. HEC-DSS, HEC-RAS, HEC-HMS, HEC-DSSVue and HEC-Vortex are products of the U.S. Army Corps of Engineers.

License

MIT; see LICENSE. Some modules port code under other permissive licences (BSD-3-Clause, MIT): NOTICE names each one and carries its notice, and THIRD_PARTY_NOTICES.md covers the Rust dependencies.

HEC-DSS, HEC-RAS, HEC-HMS, HEC-DSSVue and HEC-Vortex are products of the US Army Corps of Engineers Hydrologic Engineering Center. This project is independent of HEC and USACE.

If you use this package in published work, please consider citing RAS Commander: https://rascommander.info/ras/cite/.

Metadata

Release files for ras-commander-fast-dss 0.1.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ras-commander-fast-dss 0.1.3
File Size Uploaded Provenance Actions
ras_commander_fast_dss-0.1.3.tar.gz 2.7 MB Without provenance Details for ras_commander_fast_dss-0.1.3.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for ras-commander-fast-dss 0.1.3
File Actions
ras_commander_fast_dss-0.1.3-cp310-abi3-win_amd64.whl CPython 3.10 abi3 Windows x86-64 Details for ras_commander_fast_dss-0.1.3-cp310-abi3-win_amd64.whl
ras_commander_fast_dss-0.1.3-cp310-abi3-musllinux_1_2_x86_64.whl CPython 3.10 abi3 Linux musl 1.2+ x86-64 Details for ras_commander_fast_dss-0.1.3-cp310-abi3-musllinux_1_2_x86_64.whl
ras_commander_fast_dss-0.1.3-cp310-abi3-musllinux_1_2_aarch64.whl CPython 3.10 abi3 Linux musl 1.2+ ARM64 Details for ras_commander_fast_dss-0.1.3-cp310-abi3-musllinux_1_2_aarch64.whl
ras_commander_fast_dss-0.1.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.10 abi3 Linux glibc 2.17+ x86-64 Details for ras_commander_fast_dss-0.1.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
ras_commander_fast_dss-0.1.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl CPython 3.10 abi3 Linux glibc 2.17+ ARM64 Details for ras_commander_fast_dss-0.1.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
ras_commander_fast_dss-0.1.3-cp310-abi3-macosx_11_0_arm64.whl CPython 3.10 abi3 macOS 11.0+ ARM64 Details for ras_commander_fast_dss-0.1.3-cp310-abi3-macosx_11_0_arm64.whl
ras_commander_fast_dss-0.1.3-cp310-abi3-macosx_10_12_x86_64.whl CPython 3.10 abi3 macOS 10.12+ x86-64 Details for ras_commander_fast_dss-0.1.3-cp310-abi3-macosx_10_12_x86_64.whl

Total release size: 25.2 MB

Release files / ras_commander_fast_dss-0.1.3.tar.gz

Download URL ras_commander_fast_dss-0.1.3.tar.gz
Size 2.7 MB
Tags Source
SHA-256 checksum
How to use checksums
1ba35aa13ea0ae5667e2667f894816dd1a3db4b0939fc4f0bc3ff47e1ff5dadd
BLAKE2b-256 checksum
How to use checksums
8d41f8a4ecede5f14be3c7a88c1efb2b6bc9ac0abc2701e3ba1c823f2e36dee9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / ras_commander_fast_dss-0.1.3-cp310-abi3-win_amd64.whl

Download URL ras_commander_fast_dss-0.1.3-cp310-abi3-win_amd64.whl
Size 3.2 MB
Tags CPython 3.10 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
2a2c94dc8bc36da11aa7fefcc8177de8c3a088ac2882e92da6785ff68456e124
BLAKE2b-256 checksum
How to use checksums
dcc5e7c41d96681e1d4f60662922c7b38b29a02aca80b2d6d2f6026f51bbadd3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / ras_commander_fast_dss-0.1.3-cp310-abi3-musllinux_1_2_x86_64.whl

Download URL ras_commander_fast_dss-0.1.3-cp310-abi3-musllinux_1_2_x86_64.whl
Size 3.5 MB
Tags CPython 3.10 Linux musl 1.2+ x86-64 abi3
SHA-256 checksum
How to use checksums
127607ef16bcea193bc66df8344d000bb594a145f68472b19e03108a8e91210b
BLAKE2b-256 checksum
How to use checksums
3fa827315fe12eefc6bc858f0b3d4dd6f0c6fb9e0f7f1c9ddf27f635b0622263
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / ras_commander_fast_dss-0.1.3-cp310-abi3-musllinux_1_2_aarch64.whl

Download URL ras_commander_fast_dss-0.1.3-cp310-abi3-musllinux_1_2_aarch64.whl
Size 3.3 MB
Tags CPython 3.10 Linux musl 1.2+ ARM64 abi3
SHA-256 checksum
How to use checksums
41c46e117c1bdd6d268efd3e5b9f27a68322a40a4db546bd2dda593269c34dec
BLAKE2b-256 checksum
How to use checksums
dfcd9774bf502f2fc7f25b9b420ac7aa21d592ce288b7cf1d9b8782995977f7f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / ras_commander_fast_dss-0.1.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL ras_commander_fast_dss-0.1.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 3.3 MB
Tags CPython 3.10 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
e9c8c52fce5f85336d9e533632d586d755d3ca1f16e928efc84746779257bfb5
BLAKE2b-256 checksum
How to use checksums
ae98c3775909e31d808571f29059231cee13b29595650eea653698f1f671f690
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / ras_commander_fast_dss-0.1.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL ras_commander_fast_dss-0.1.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size 3.1 MB
Tags CPython 3.10 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
0b9d69e357310eb6a468431a730ecad1177867fc96b9e2005e03e39ba86bda88
BLAKE2b-256 checksum
How to use checksums
20215478f52645f9ac9edd259c94fc31e79d23497167081a4bb4c42629226ff0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / ras_commander_fast_dss-0.1.3-cp310-abi3-macosx_11_0_arm64.whl

Download URL ras_commander_fast_dss-0.1.3-cp310-abi3-macosx_11_0_arm64.whl
Size 3.0 MB
Tags CPython 3.10 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
b299d8c1608aedcf3cfc6b551e12adb25e0591f941edb2f65ba6c5dda905ea64
BLAKE2b-256 checksum
How to use checksums
b5f634c6b44ec4f6406038d640e3aa630975a0cc84b66e3aa6ea1e36615c54c6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / ras_commander_fast_dss-0.1.3-cp310-abi3-macosx_10_12_x86_64.whl

Download URL ras_commander_fast_dss-0.1.3-cp310-abi3-macosx_10_12_x86_64.whl
Size 3.2 MB
Tags CPython 3.10 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
271c4a35ccd262b59bc76796ff81988f2f4a83d3d8323298f6fddf533a908868
BLAKE2b-256 checksum
How to use checksums
531bc9f31d38855c3d5801cfc746e5ea1c07c726a6b378c656f33ec9f322acd2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Security for ras-commander-fast-dss 0.1.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page