py-tools-ds 0.16.7

A collection of Python tools by Daniel Scheffler.

py_tools_ds

A collection of Python tools by Daniel Scheffler.

• Free software: GNU General Public License v3 or later (GPLv3+)
• Documentation: https://danschef.gitext-pages.gfz-potsdam.de/py_tools_ds/doc/

Status

See also the latest coverage report and the nosetests HTML report.

Features

• TODO

Installation

py_tools_ds depends on some open source packages which are usually installed without problems by the automatic install routine. However, for some projects, we strongly recommend resolving the dependency before the automatic installer is run. This approach avoids problems with conflicting versions of the same software. Using conda, the recommended approach is:

via conda + pip

# create virtual environment for py_tools_ds, this is optional
conda create -y -q -c conda-forge --name py_tools_ds python=3
conda activate py_tools_ds
conda install -c conda-forge numpy gdal 'pyproj>=2.1.0' shapely scikit-image pandas

Then install py_tools_ds using the pip installer:

pip install py_tools_ds

via conda channel (currently only for Linux-64)

# create virtual environment for py_tools_ds, this is optional
conda create -y -q --name py_tools_ds python=3
conda activate py_tools_ds
conda install -c danschef -c conda-forge -c defaults py_tools_ds

Credits

The py_tools_ds package was developed within the context of the GeoMultiSens project funded by the German Federal Ministry of Education and Research (project grant code: 01 IS 14 010 A-C).

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

History

0.16.7 (2020-12-15)

• Fixed issue #16 (WKT2EPSG() returns None in case of bound CRS.).

0.16.6 (2020-12-10)

• Updated installation procedure documentation.

0.16.5 (2020-12-09)

• Added URL checker CI job and fixed all dead URLs.
• Removed travis related files.
• Re-ordered dependencies.
• Added Python 3.9 classifier to setup.py and removed those for <3.6.

0.16.4 (2020-12-07)

• Fixed issue #15 (boxObj.get_coordArray_MapXY() returns faulty y-coordinate array).
• Test_boxObj.test_get_coordArray_MapXY() now also validates the output of boxObj.get_coordArray_MapXY().

0.16.3 (2020-12-03)

• Fixed an issue causing geotransform2mapinfo() return a ‘South’ mapinfo instead of a ‘North’ one for a GeoTransform in the West of the central meridian.

0.16.2 (2020-11-18)

• Fixed issue of remaining coverage artifacts after running ‘make clean-test.
• Revised coord_trafo.py. This fixes an issue that caused pixelToLatLon() to return Lon/Lat instead of Lat/Lon.
• Fixed mapXY2imXY() and imXY2mapXY().
• Added Test_mapXY2imXY(), Test_imXY2mapXY(), Test_pixelToLatLon(), Test_latLonToPixel().
• Removed GDAL dataset input parameters from some functions.
• Revised code style and some docstrings. Added some typehints.
• Bugfix for always raising a RuntimeWarning in fill_holes_within_poly().

0.16.1 (2020-11-03)

• Removed anaconda recipe and deploy_anaconda CI job because py_tools_ds is now on conda-forge anyways.

0.16.0 (2020-11-02)

• Removed similarity module as this is only used by arosics (moved SSIM computation there).

0.15.11 (2020-11-02)

• Replaced deprecated osgeo imports.

0.15.10 (2020-10-30)

• Bumped version due to wrong PyPI upload.

0.15.9 (2020-10-30)

• Fixed issue #13 (TypeError when a MultiPolygon instance is passed to fill_holes_within_poly()).
• Added tests for fill_holes_within_poly() and get_overlap_polygon()
• Tests are now also executable on Windows.

0.15.8 (2020-10-26)

• Replaced deprecated ‘source activate’ with ‘conda activate’.
• Bugfix for not rounding off negative coordinates in get_smallest_boxImYX_that_contains_boxMapYX() and get_smallest_shapelyImPolyOnGrid_that_contains_shapelyImPoly().

0.15.7 (2020-10-12)

• Exclude tests from being installed via ‘pip install’.
• Use SPDX license identifier and set all files to GLP3+ to be consistent with license headers in the source files.

0.15.6 (2020-09-15)

• Replaced deprecated HTTP links.

0.15.5 (2020-09-02)

• Revised build_testsuite_image.sh
• Added docker-pull-policy to register command.
• Added some comments to build_testsuite_image.sh and set concurrency to 4. Updated deprecated code in .gitlab_ci.yml.

0.15.4 (2020-09-01)

• Fixed wrong output of reproject_shapelyGeometry() in case of EPSG 4326.

0.15.3 (2020-09-01)

• Fixed wrong output of transform_any_prj() in case of EPSG 4326.
• Updated minimal version of pyproj to 2.2.0.

0.15.2 (2020-08-27)

• Fixed a race condition within geotransform2mapinfo() and mapinfo2geotransform() when running in multiprocessing.

0.15.1 (2020-08-27)

• Fixed exceptions caused by WKT2 strings passed to GDAL<3.

0.15.0 (2020-08-26)

• Replaced old code with simple pyproj>2.1 calls. This fixes a PermissionError within WKT2EPSG on Windows.
• Added minimal version of pyproj.
• Replaced deprecated code in reproject_shapelyGeometry().
• geotransform2mapinfo() should now be compatible with gdal<3.

0.14.36 (2020-08-21)

• test_py_tools_ds_install now runs within conda-forge environment. Updated installation instructions.

0.14.35 (2020-08-21)

• Re-upload to PyPi.

0.14.34 (2020-08-21)

• Added ‘tolerance’ keyword to geo.vector.topology.polyVertices_outside_poly() to avoid wrong return values due to float uncertainties.

0.14.33 (2020-08-18)

• Removed keywords from gdal_array.OpenArray() call to ensure downwards compatibility.

0.14.32 (2020-08-18)

• Fixed DeprecationWarning related to skimage.measure.compare_ssim. Added minimal version of scikit-image.

0.14.31 (2020-08-18)

• Moved skimage import to functional level to avoid static TLS import error.

0.14.30 (2020-08-17)

• Removed deprecated gdal backports (compatibility module).
• Added minimal version of gdal.
• Cleaned up some unused code.

0.14.29 (2020-05-19)

• Fixed HISTORY.rst and PyPI upload.

0.14.28 (2020-05-19)

• Fixed pyproj deprecation warning.
• Generated HISTORY.rst.

0.14.27 (2020-03-24)

• Merge branch ‘bugfix/fix_issue10’ into ‘master’
• Fix for incompatibility of round_shapelyPoly_coords() with shapely>1.6.4.

0.14.26 (2020-03-10)

• Fix for returning white spaces at the end of a Proj4 string. Updated version info.

0.14.25 (2020-01-08)

• Merge branch ‘enhancement/clean_deps’ into ‘master’
• Re-added dependency scikit-image.
• The rasterio dependency is now optional.
• Added conda update to test_py_tools_ds_install.
• Removed not needed dependencies pyqt, scikit-image, lxml and ipython.

0.14.24 (2020-01-07)

• Merge branch ‘enhancement/remove_sensormapgeo_code’ into ‘master’
• Set conda channel priority to ‘strict’ to avoid using wrong channels.
• Fixed broken badge.
• Fixed deprecation warning.
• Removed all functionality related to transforming sensor to map geometry or vice-versa as this was moved to the new package ‘sensormapgeo’.
• Added downloads badge.

0.14.23 (2019-10-17)

• Merge branch ‘bugfix/fix_bilinear_sensorgeo_gauss_3D_trafo’ into ‘master’
• Fixed issue where SensorMapGeometryTransformer raised an exception when trying to resample a 3D input array.
• Fixed SensorMapGeometryTransformer.to_sensor_geometry() not working for resamp_alg=’bilinear’.
• The test_reproject module now tests all resampling algorithms instead of only one per test.

0.14.22 (2019-10-16)

• Merge branch ‘bugfix/fix_3D_input’ into ‘master’
• Fixed incompatibility of SensorMapGeometryTransformer with 3D input arrays. Updated version info.

0.14.21 (2019-08-14)

• Replaced deprecated PyPi upload commands by twine. Updated version info.

0.14.20 (2019-07-26)

• Merge branch ‘bugfix/fix_importerror_spectral’ into ‘master’
• Moved import of ‘spectral’ package to function level.
• Merge branch ‘enhancement/add_license_texts’ into ‘master’

0.14.19 (2019-07-22)

• Added license notes. Added funding note. Updated version info.
• Merge branch ‘bugfix/fix_windows_geometrytrafo’ into ‘master’

0.14.18 (2019-06-14)

• Fixed Windows incompatibility of SensorMapGeometryTransformer. danschef 6/13/19, 4:46 PM
• Merge branch ‘bugfix/fix_mp_deadlock’ into ‘master’

0.14.17 (2019-03-22)

• Restricted mp_alg overriding to nosetest console calls only.
• Also include unittest in mp_alg overriding.
• Force mp_alg = ‘tiles’ in case SensorMapGeometryTransformer3D is called within a nosetest.
• Added parameter ‘mp_alg’.
• Merge branch ‘enhancement/share_mp_variables’ into ‘master’

0.14.16 (2019-03-22)

• Share large arrays between multiprocessing workers in SensorMapGeometryTransformer3D.
• Merge branch ‘bugfix/fix_singleprocessing’ into ‘master’

0.14.15 (2019-03-22)

• Fixed bug that caused to use multiprocessing.Pool if number of CPUs is set to 1.
• Merge branch ‘bugfix/implement_pyresample_changes’ into ‘master’

0.14.14 (2019-03-21)

• Fixed test_reproject module.
• Fixed linting.
• Removed deprecated install command.
• Inherit from gms_base_centos:0.3.
• Updated docker container setup.

0.14.13 (2019-03-21)

• Declared pyresample as conda dependency.

0.14.12 (2019-02-19)

• Replaced get_area_def by AreaDefinition.
• Implemented pyresample changes and added minimal version of pyresample.
• Fixed issue #7 (only appearing on Windows).
• Merge branch ‘bugfix/fix_static_TLS_import_error’ into ‘master’

0.14.11 (2019-02-19)

• Cleaned up __init__.py files.
• Moved rasterio imports to function level to avoid static TLS conflict with pykdtree which is a dependency of pyresample.
• Merge branch ‘bugfix/fix_ETRS_projection_incompatibility’ into ‘master’

0.14.10 (2019-02-19)

• Reverted pyresample imports. Fixed style issues.
• Added workaround for ETRS/LAEA projection incompatibility + tests.
• Moved all pyresample imports to the top of the module to avoid Exception: dlopen: cannot load any more object with static TLS.
• Moved all pyresample imports to methods to avoid Exception: dlopen: cannot load any more object with static TLS.
• Fix for Exception: dlopen: cannot load any more object with static TLS.
• Fixed multiprocessing issue related to OpenMP multiprocessing within pykdtree as called by pyresample. Fixed type hints.

0.14.9 (2019-02-14)

• Fixed FutureWarning for numpy>1.16: arrays to stack must be passed as a “sequence” type such as list or tuple. Updated version info.
• Renamed test method.

0.14.8 (2019-01-21)

• Fixed gdal v2.4.0 incompatibility and added tests for io.raster.gdal.

0.14.7 (2019-01-09)

• Merged enhancement/add_3D_geolayer_support into master.
• Fixed duplicate return value.
• Fixed sub-multiprocessing error.
• Added SensorMapGeometryTransformer3D + tests.
• Typo fix.

0.14.6 (2018-12-14)

• Fixed faulty output validation.

0.14.5 (2018-12-14)

• Fixed wrong assertion.

0.14.4 (2018-12-14)

• Added default to SensorMapGeometryTransformer.to_map_geometry. Updated version info.

0.14.3 (2018-12-14)

• Revised SensorMapGeometryTransformer + tests.
• Improved output validation.

0.14.2 (2018-12-13)

• Fix.
• Added additional stage to fix deployment order.
• Fixed bad .gitlab-ci.yaml.

0.14.1 (2018-12-13)

• Added type hints. Changed a default value. Fixed deployment order to PyPi, Anaconda.

0.14.0 (2018-12-12)

• Missing test datasets are now versioned.
• Revised SensorMapGeometryTransformer (now fully operable) and added corresponding tests.
• Added boxObj.buffer_mapXY() + test.
• Added type hints.
• Enhanced documentation.
• Fixed docker test run.
• Added resampling algorithms ‘bilinear’ and ‘custom’. Added docstrings and type hints.
• Added first working version of SensorMapGeometryTransformer.

0.13.7 (2018-12-03)

• Added a tolerance to get_smallest_boxImYX_that_contains_boxMapYX() to avoid float coordinate rounding issues.

0.13.6 (2018-12-03)

• Added tests for geo.vector.geometry module. Bugfixes for boxObj().

0.13.5 (2018-12-03)

• Fixed a wrongly raised warning within warp_ndarray(). Updated version info.

0.13.4 (2018-12-03)

• Bugfix for find_nearest(). Updated version info.

0.13.3 (2018-12-03)

• Fixed linting.

0.13.2 (2018-12-03)

• Added version file.
• Fixed linting.
• Added tolerance parameter to find_nearest().
• Added tests for find_nearest().
• Docker CI image now inherits from gms_base_centos:0.2.
• CI setup now updates ci_env environment installed via docker_pyenvs instead of creating an independent environment.
• CI Python environment is not separate from the base env. Fixed mixed channels for gdal and libgdal causing libkea issue during CI.
• Capped version of pycodestyle to <2.4.0 due to bug if used together with flake8.
• Added kealib to requirementsand to test_py_tools_ds_install to fix import error after install.
• Updated docker runner build script.

0.13.1 (2017-12-06)

• Merge branch ‘bugfix/fix_decompress’

0.13.0 (2017-12-06)

• Updated version info.
• Updated README.
• Updated .gitlab-ci.yml.
• Updated .gitlab-ci.yml.
• Updated .gitlab-ci.yml. Updated README.
• Updated .gitlab-ci.yml. Updated README.
• Revised compression.decompress.decompress.
• Added missing anaconda-client. Added conda_build_config.yaml
• Bugfix.
• Added conda recipe. Removed superfluous packages from environment*.yml. Updated .gitlab-ci.yml.
• Updated docker installer and environment*.yml

0.13.0 (2017-12-06)

• Added SQL db tools.

0.12.5 (2017-11-30)

• Removed buggy assertion.

0.12.4 (2017-11-20)

• Updated version info.

0.12.3 (2017-11-20)

• Removed deprecated make rule.
• Moved docker setups for basic gms dependencies to external project.
• Removed additional env ‘py3’.

0.12.2 (2017-11-18)

• Beautified docker installer workflow.
• Updated env settings within gitlab_ci.yml
• Changed tag.
• Changed used environment within gitlab CI.
• Changed basic image name for py_tools_ds_ci.docker.
• Revised docker installer (now uses a basic conda environment and the gitlab runner container on top).
• Moved geopandas to pip packages within docker container setup.

0.12.1 (2017-11-16)

• Replaced pandas by geopandas within CI installer test.

0.12.0 (2017-11-16)

• Added spectral to dependencies. Added many functions from arosics:
• Added comment.
• Updated pip requirements.
• Added badges. Updated pip requirements.
• Added badges.

0.11.1 (2017-11-15)

• Updated version info.

0.11.0 (2017-11-15)

• Merge branch ‘feature/add_rotation_support’
• Removed print statement.
• Completed implementation of geo.map_info.Geocoding class. Added tests for rotated datasets.
• Added support for rotated datasets: Implemented class geo.map_info.Geocoding(). Reimplemented geotransform2mapinfo()
• and mapinfo2geotransform(). Bugfix for geo.projection.isLocal()

0.10.1 (2017-11-09)

• Progressbar now prints to sys.stderr to avoid conflicts with sys.stdout.
• Added FIXME.

0.10.0 (2017-11-02)

• Merge branch ‘feature/add_get_array_tilebounds’
• Revised get_array_tilebounds() and added tests.

0.9.4 (2017-11-02)

• Allowed tuple objects to be passed to warp_ndarray().
• Fixed warp_ndarray() in case a list of ndarrays is provided.
• Added function numeric.array.get_array_tilebounds + tests.
• Added function numeric.array.get_array_tilebounds + tests.
• Added requirements_pip.txt.
• Added pandas to packages installed by conda during CI.

0.9.3 (2017-10-12)

• Bugfix warp_ndarray.
• Changed downsampling threshold of geo.raster.conversion.raster2polygon.

0.9.1 (2017-10-11)

• Fixed pages.
• Updated ‘pages’ CI job.
• Renamed CI job ‘deploy_pages’ tp ‘pages’.
• Changed deploy_pages CI job to make pages work again.
• Updated Anaconda version within docker setup. Updated runner version. Revised .gitlab-ci.yml.
• Changed warp_ndarray projection defaults. Added geo.projection.isLocal(). Added module numeric.numbers. Updated version info.

0.9.0 (2017-10-09)

• Merge remote-tracking branch ‘remotes/origin/feature/add_localCS_compatibility’

0.8.4 (2017-10-06)

• Updated Test_move_shapelyPoly_to_image_grid.
• mapinfo2geotransform(): Fix for asserting a map_info with 8 elements in case of arbitrary coordinates. Added test_coord_grid module.

0.8.3 (2017-10-06)

• geotransform2mapinfo, mapinfo2geotransform: added compatibility to local coordinate systems. Added test_map_info module.
• Added badge for Anaconda cloud.

0.8.2 (2017-09-25)

• Fixed issue#3 (typing).

0.8.1 (2017-09-22)

• Removed tarfile and zipfile from requirements as they are system libs. Updated version info.

0.8.0 (2017-09-22)

• Added module ‘compression’. Updated requirements.

0.7.4 (2017-09-20)

• geo.raster.raster2polygon(): Added auto-downscaling of input array and updated version info.

0.7.3 (2017-09-20)

• Suppressed inspection.
• Fix mapinfo2geotransform for asserting wrong length of map info in case of geographic coordinates.

0.7.2 (2017-12-19)

• Added type hint.
• PEP8 editing. Added linting.

0.7.1 (2017-09-13)

• Merge branch ‘bugfix/fix_get_overlap_polygon’
• Merge branch ‘enhancement/add_auto_setter_GDAL_DATA’
• Fix issue #5 (get_overlap_polygon() did not return geometry type ‘Polygon’ but GeometryCollection.”).

0.7.0 (2017-09-17)

• Fix nosetests.
• Added link for nosetests HTML reports to README.rst.
• Added nosetests. Activated test test artifacts for failed pipelines. Added test_py_tools_ds_install.
• Updated docker container setup and test requirements.

0.6.0 (2017-09-11)

• Added environment module containing auto-setter for GDAL_DATA variable.

0.5.0 (2017-09-11)

• Added os compatibility module. Added future imports to ensure Python 2.7 compatibility.
• Updated README.rst.

0.4.6 (2017-09-11)

• Fixed EPSG2WKT returning None in case GDAL_DATA environment variable is not set. Added Test_EPSG2WKT.

0.4.5 (2017-09-11)

• Bugfix prj_equal: Removed superfluous projection comparison, improved type hint.

0.4.4 (2017-09-09)

• Revised geo.projection.WKT2EPSG and added _find_epsgfile() to increase operation system compatibility.
• Added module test_projection. Updated version info.
• Added dummy code for conda deployment for other Python versions.
• Removed logout.
• Added token.
• Added another logout.
• Always log out from anaconda.
• Fix.
• Install patch.
• Fix.
• Fix.
• Fix.
• Changed conda skeleton output dir.
• Fix.
• First setup for Anaconda CD.

0.4.3 (2017-08-20)

• Fixed missing dependency for scikit-image.

0.4.2 (2017-08-19)

• Completely excluded geoarray from py_tools_ds (solves circular dependency).

0.4.1 (2017-07-05)

• updated __version__ and __versionalias__

0.4.0 (2017-07-03)

• Added auto-deploy to PyPI; revised badges.

0.3.3 (2017-07-03)

• updated setup.py

0.3.2 (2017-07-03)

• Updated links within documentation. Updated setup requirements.
• Bugfix for SystemError: <built-in function Band_SetNoDataValue> returned a result with an error set
• Implemented XY-getters for boxObj.
• Changed license to GPL v3.
• Updated README.rst
• Added requirements.txt
• Bugfix
• Revision of CI setup,
• First setup of CI runner.
• Removed osr from setup.py because its included in gdal.
• Updated README.
• Added subpackages to setup.py
• Changed import statements in __init__.py
• Changed import statements in __init__.py

0.1.0 (2017-06-09)

• First release on PyPI.
• Changed module name from ptds to py_tools_ds to fix bug of pip installer. Changed license.
• Merged complete package content of py_tools_ds into a cookiecutter package.
• Updated a deprecated function call, added new submodule “network”.
• moved GeoArray to a new separate library called ‘geoarray’, added convenience module;

20170331_01

• added keywords vmin, vmax
• some improvements
• added keyword to GeoArray.show_map() to make figure zoomable
• added functions

GMS_BETA

• Bugfix
• new keyword for GeoArray.get_mapPos()
• Bugfix for GeoArray.metadata.setter
• Added output verification for get_overlap_polygon()
• Bugfixes
• Bugfixes
• Bugfix
• added progress keyword to GeoArray.get_mapPos() and get_array_at_mapPos()
• updated assertion from last commit
• added assertion
• Bugfix

20170123_01

• Bugfix
• added array caching: GeoArray and all subclasses now remember the last position read from disk and return it from
• memory -> speed improvement
• Merge remote-tracking branch ‘origin/master’
• geo.coord_grid: - is_coord_grid_equal(): added keyword ‘tolerance’; added type hint and docstring - is_point_on_grid():
• added keyword ‘tolerance’; added type hint and updated docstring
• geo.coord_grid: - is_coord_grid_equal(): added keyword ‘tolerance’; added type hint and docstring - is_point_on_grid():
• added keyword ‘tolerance’; added type hint and updated docstring

20170119_02

• revised GeoArray.__getitem__()
• geo.vector.geometry: - fixed some broken type hints

20170104_01

• geo,map_info: - geotransform2mapinfo(): bugfix
• Bugfix
• added verbose mode to GeoArray.get_mapPos()
• Fix for copied memory address within GeoArray
• Bugfix for not silencing GeoArray.reproject_to_new_grid() in quit mode

20161125_01

• added function for quickly reprojecting GeoArray to a given pixel grid; mask_nodata is now a subclass of GeoArray
• Bugfix for rejecting GeoArray subclasses by GeoArray

20161122_01

• added new property ‘mask_baddata’ to GeoArray; some bugfixes and further developments; new dtypes package
• simplified GeoArray.__init__(); some bugfixes and further developments
• added metadata property to GeoArray
• added holoviews visualization for exploring bands-axis in GeoArray
• updated __version__

20161112_01

• implemented point-wise read processes in GeoArray

20161112_01

• some further developments and bug fixes
• updated __version__
• Bugfix for footprint_poly; revised progress bar handling; bugfix for not resetting timeout start time; some further developments

20161108_01

• edited some assertion messages and docstrings, updated __version__
• bugfix for raster2polygon
• bugfix for missing ‘progress’ attribute of GeoArray, bugfix for note resetting start time in raster2polygon
• bugfix for running raster2polygon without timeout
• bugfix for overwriting user defined nodata value within GeoArray
• added warning if automatic nodata value detection returns unreliable value; bugfix for overwriting user defined attributes of GeoArray
• bugfix for error in warp_ndarray assertion; bugfix for wrong array slicing
• some bugfixes and further developments

20161102_01

• updated __version__
• Bugfix for wrong output geotransform within GeoArray._get_plotable_image()
• added two functions to GeoArray
• fixed a circular import issue
• added a lot of feature improvements and further developments
• updated GDAL version check, __version__

20161029_01

• Bugfix for returning wrong array shape when warping a 3D array
• Bugfix for raising a warning although everything is fine
• added Python 2.7 compatibilty; added GDAL downwards compatibility
• added warpMemoryLimit to warp_ndarray
• fixed missing function transform_any_prj() that was caused by a copy paste error
• added functions to visualize GeoArray as map; improved GeoArray.show(); some new functions
• implemented multiprocessing and GCP based warping into warp_ndarray
• modified docstring of calc_FullDataset_corner_positions(); added TODO to boxObj()
• some bugfixes and improvements
• Bugfix within GeoArray; added functions
• geo.raster.reproject: - added a new version of warp_ndarray and renamed the old one to warp_ndarray_OLD: much faster
• than the old version and no issues when warping 3D arrays - some modifications to warp_ndarray_OLD -
• added get_GDAL_ds_inmem() - added get_GeoArray_from_GDAL_ds() - added warp_GeoArray(): a function to warp GeoArray objects
• added a lot of functions, checked importability, refactored map module to ‘geo’
• Initial commit