DPS_Util

This repository contain all util.

Install:

Get at https://pypi.org/project/dpsutil/

pip install dpsutil

Change log:

Ver 1.0.0:

Compression -> DONE
KafkaWrapper -> DONE
RedisWrapper -> DONE
Vector -> DONE
Media -> DONE
Computer Vision -> DONE
Distance -> DONE
Hashing -> DONE

Ver 1.0.1:

Added image_info -> Get (format, width, height) of image without decoding.
Upgrade imdecode -> Auto decode image but need not provide encode_type.

Ver 1.0.2:

Change 'numpy.pool' to 'vector.pool'
Fix bugs

Ver 1.0.3:

Compression:
Support list compression: compress_list, decompres_list
Distance:
Change distance.function -> vector.distance

Ver 1.1.0:

Added AttrDict, FixedDict, FixedTypeDict

Ver 1.1.1-1.1.7:

Fix bugs
Added UniqueTypeDict
Fix bugs
Changed module fixdict -> defaultdict
Changed FixedDict -> DefaultDict
Changed FixedTypeDict -> DefaultTypeDict
Fix bugs of attrdict.AttrDict, attrdict.UniqueTypeDict, attrdict.DefaultDict
Fix bugs of compress_list, decompress_list

Ver 1.1.8:

Added Environment

Ver 1.1.9-1.1.18:

Added Environment.to_lower -> Useful when extracting to kwarg.
Fix bugs attrdict
Changed hashfunc default from sha1 -> md5
Support call of DefaultDict, which update data from buffer.
Fix bugs Environment

Ver 1.1.19:

Added pre-define output array of decompress_ndarray
Support add new key environment.

Ver 1.1.20-1.1.21:

Edit logger
Fix bugs vector.pool
Changed VectorPool base on numpy.mmap -> VectorPoolMMap
Added VectorPool base on numpy.ndarray
Restructure VectorPoolBase -> Speed up, fix bugs
Added VectorPoolBase.insert

Ver 1.1.22:

Add document of compression, media module.
Fix bugs image
Change media.image.scale -> media.image.zoom

Ver 1.1.23-24:

Fix bugs log
Fix bugs VectorPoolMMap.MIN_SIZE

Ver 1.1.25-26:

Fix bugs attrdict
Added add, remove method attrdict
Rename variable of pool.VectorPoolBase

Ver 1.1.27:

Change namespace attrdict.UniqueTypeDict -> attrdict.TypedDict
Restructure attdict.AttrDict, attdict.DefaultDict, attrdict.TypedDict
Support annotations alias
Speed up attrdict
Support initial attrdict with iterable, generator.
Support to_buffer method with optimize compression.
Support recreate dict with from_buffer.
Lightweight than pickle.dumps

Ver 1.2.0:

CV:
- Added Face Aligner
- Transforms image methods
media.image:
- Added scale, crop_center, flip
- Fix bugs
media.video: support FPS report.
triangle (new)

Ver 1.3.0:

Support decorator attrdict
Support decorator environment
Implemented Confluent-Kafka with:
- Iterator method
- Close event.
- Auto_commit
- Optimize configs for ThroughOut
- Fix bug KafkaError

Todo:

Implement Configs librdkafka
Sort -> implement natsort - https://github.com/SethMMorton/natsort

Features

Compression Lossless:

Support type: ndnumpy, bytes
Compress by blosc. It support multi compressor and multi-thread.

KafkaWrapper:

Wrapping Consumer and Producer with default setting and security.

RedisWrapper

Wrapping Redis Connector with default setting and security.

Vector Pool:

VectorPoolBase handle numpy.ndarray that save a lot of RAM by re-use existed memory space.

Easier than numpy.ndarray. Likely list.

Reduce RAM
High Speed IO
Auto Scale
Support backup and recovery data in a second.

Media:

Implemented OpenCV with:

TurboJPEG (https://github.com/lilohuang/PyTurboJPEG)
FFmpeg(https://github.com/kkroening/ffmpeg-python)

To: improve read & write (image, video) IO speed.

Faster than OpenCV:
- 2-6x with JPEG Encoder
- 1.1x with others.
Added some function which used frequently.
More info: find in dpsutil.media

Computer Vision (cv):

Added Face Align with five landmark.

Distance:

Added cosine_similarity
Added cosine
Added euclidean_distance
Added convert distance functions

*Note: all function execute in numpy.

Attributes dict:

AttrDict: will help you get value of key via attributes. Implement attrdict.AttrDict
DefaultDict: help cover your dict with (keys, values) that was defined before. Implement from AttrDict
TypedDict: help cover your dict when set item. Implement from AttrDict.
DefaultTypeDict: dict only access one type for all element.
Support initial attrdict with iterable, generator.
Support to_buffer method with optimize compression.
Support recreate dict with from_buffer.
Lightweight than pickle.dumps

Environement:

Environment: Auto pair environment parameters fit with default, which provided before.

Other:

Hashing
Sort

Issue:

Cmake error during install blosc

Firstly, you need install scikit

$ pip install scikit-build

After that, follow instuction to install Cmake:

https://cliutils.gitlab.io/modern-cmake/chapters/intro/installing.html

Python

$ pip install cmake>=3.12

Not found 'FFmpeg':

Find FFmppeg lib at:

https://www.ffmpeg.org/download.html

Linux:

sudo apt-get install ffmpeg

Not found 'libjpeg-turbo':

Find FFmppeg lib at:

https://libjpeg-turbo.org/Documentation/OfficialBinaries

Linux:

$ sudo apt install libturbojpeg

Not found Redis or Kafka server:

Make sure your Redis or Kafka server started.
Make sure correct username & password.

Not found 'Could NOT find OpenSSL':

SSL development libraries have to be installed

https://github.com/openssl/openssl#build-and-install

Ubuntu:

$ apt-get install libssl-dev libffi-dev

Develop by: TinDang
OS: Ubuntu Bionic 18.04
Python: 3.6.9 stable

Metadata

Release files for dpsutil 1.4.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dpsutil 1.4.1
File	Size	Uploaded	Provenance
dpsutil-1.4.1.tar.gz	50.1 kB	Oct 5, 2020	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for dpsutil 1.4.1
File	Interpreter	ABI	Platform	Reset
dpsutil-1.4.1-py3-none-any.whl 59.4 kB Oct 5, 2020 Without provenance	Python 3	none	any	Details

Total release size: 109.5 kB

Release files / dpsutil-1.4.1.tar.gz

Download URL	dpsutil-1.4.1.tar.gz
Size	50.1 kB
Tags	Source
SHA-256 checksum How to use checksums	`337fb6cfb2c3a03c17c3365386bc18b09a66fa877550fc505cab9a41bb64c067`
BLAKE2b-256 checksum How to use checksums	`5583ab807a9752543140c5ece8d6e732c02685ddc0369b699768204467f67f0c`
Upload date	Oct 5, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.2.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/50.3.0 requests-toolbelt/0.9.1 tqdm/4.50.0 CPython/3.8.2`

Release files / dpsutil-1.4.1-py3-none-any.whl

Download URL	dpsutil-1.4.1-py3-none-any.whl
Size	59.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`29f3451dc6bf57863437ae07505e8d9fa4e854d5ba1d7954662e35a93fc34125`
BLAKE2b-256 checksum How to use checksums	`71720485fb84466c5d7b67d5611747d5aded346b03688eacd38a939abee7b07a`
Upload date	Oct 5, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.2.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/50.3.0 requests-toolbelt/0.9.1 tqdm/4.50.0 CPython/3.8.2`

Security for dpsutil 1.4.1

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.