hiproc 0.6.0

A library to help process HiRISE EDRs with ISIS.

hiproc

A library to help process HiRISE EDRs with ISIS.

• Free software: Apache 2 License

Features

• TODO: Complete testing against Perl Pipelines.

External Dependencies

These programs use as much ‘vanilla’ Python 3 as possible.

However, it does depend on the following:

• pvl library (https://pvl.readthedocs.io)

• kalasiris library (https://kalasiris.readthedocs.io)

• gdal

• numpy

• scipy

• matplotlib

The HiJACK program also requires an external resolveJitter program that has not been publicly released, but isn’t that far off. There is a MATLAB version that has the appropriate licensing, and there is a C++ version. The C++ version could also be made to have the appropriate licensing, it just hasn’t gone through a release process. Maybe I’ll write it in Python, too, and distribute it here.

Warning !

The algorithms based on the HiRISE Processing Pipelines were emulated and tested locally, but the results of each pipeline have not been tested directly against the results of the HiRISE Processing Pipelines, and this warning will remain until I have done so. As a result, I would not particularly ‘trust’ anything produced by these programs at this time, and consider these algorithms a work-in-progress.

These programs have been tested against their upstream Perl counterparts:

• EDR_Stats: Verified!

  Really just runs hi2isis so no surprise here.

• HiCal: Verified. (if HiGainFx() enabled)

  Upstream is undergoing change. HiGainFx really shouldn’t be applied, so it is commented out here. We’re also working to integrate the bitflip cleaning into the upstream Perl, so this is in flux.

Details

The image processing pipelines that the HiRISE team operates internally to produce higher order products are more than just the ‘simple’ programs available in ISIS. Those processes that run in the HiRISE Operations Center (HiROC) are a complicated dance of primarily Perl and ISIS run by a custom job management system, all of which interacts with the HiRISE catalog (HiCat) database.

This makes the HiROC system excellent for processing the Gigabytes of new data that arrive daily from Mars, and allows the team to perform massive reprocessing of the entire data set, as needed, and to produce on a large scale a variety of derived data products.

However, that same complexity makes it difficult to reproduce exactly what that system is doing on a small scale.

The programs here are meant to replicate the HiRISE processing chain on a local scale, so that individual algorithms and processes can be investigated, without needing a massive data processing system and lots of infrastructure.

The programs here have similar names to HiRISE pipelines (hence the perhaps strange intercapped naming conventions), but only focus on the data processing. The HiRISE pipeline programs do a lot of other tasks relevant to being part of a massive ground data system, and clearly, those functionalities aren’t replicated here.

The HiROC system begins by watching the MRO project’s raw data server for new products with the FEI_Watchdog program, and then the HiDog pipeline fetches those products down to HiROC and the the EDRgen pipeline converts them into .img EDR products.

Since that is the most basic form of the data available from the PDS, we will start there, and assume that you have downloaded a set of EDR .img files from the PDS.

As a final note, this library currently uses .json files to manage passing information between programs, instead of a relational database system.

Contributing

Feedback, issues, and contributions are always gratefully welcomed. See the contributing guide for details on how to help and setup a development environment.

Naming

The ISIS software has a number of “proc” programs (mocproc, thmproc, etc.) that are meant to be run to process raw images to higher-level, more usable versions. Naming this library hiproc is an echo to that, and while there is a hiproc program that is available, this library provides a great deal more.

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

When updating this file, please add an entry for your change under Unreleased and one of the following headings:

• Added - for new features.

• Changed - for changes in existing functionality.

• Deprecated - for soon-to-be removed features.

• Removed - for now removed features.

• Fixed - for any bug fixes.

• Security - in case of vulnerabilities.

If the heading does not yet exist under Unreleased, then add it as a 3rd level heading, underlined with pluses (see examples below).

When preparing for a public release add a new 2nd level heading, underlined with dashes under Unreleased with the version number and the release date, in year-month-day format (see examples below).

Unreleased

0.6.0 (2021-03-18)

Added

• More complete documentation for the programs and their parameters.

Changed

• Implemented better handling for configuration files so that they will get distributed with the package.

0.5.0 (2021-03-05)

Added

• lisfix: Added the lisfix module.

• bitflips: Sometimes the very end of the histogram (although not a formal minima) is the appropriate choice, and is now considered.

• bitflips: Added capability to ignore minor maxima at the ends of the histogram to “roll down” to a better solution.

Changed

• name change of project from pyrise to hiproc.

• bitflips: Changed the default medstd_limit from 300 to 400 DN.

Fixed

• bitflips: There were a variety of edge cases that resulted in errors. The appropriate guardrails, handlers, and recovery logic has now been added.

• HiStitch: The equalize and balance parameters cannot both be true.

0.4.0 (2020-09-22)

• Tremendous amount of re-working in bitflips to improve performance.

• Format cleanup

0.3.0 (2020-05-16)

• Confirmed that EDR_Stats and HiCal produce identical output cubes.

0.2.0 (2020-05-06)

• Updated with bit-flip correction.

0.1.0 2020-03-21

• First shared on GitHub