Skip to main content

This repository contain all util

Project description

DPS_Util

This repository contain all util.

Install:

Get at https://pypi.org/project/dpsutil/

pip install dpsutil

Change log:

Ver 1.0.0:

  • Compression -> DONE
  • KafkaWrapper -> DONE
  • RedisWrapper -> DONE
  • Vector -> DONE
  • Media -> DONE
  • Computer Vision -> DONE
  • Distance -> DONE
  • Hashing -> DONE

Ver 1.0.1:

  • Added image_info -> Get (format, width, height) of image without decoding.
  • Upgrade imdecode -> Auto decode image but need not provide encode_type.

Ver 1.0.2:

  • Change 'numpy.pool' to 'vector.pool'
  • Fix bugs

Ver 1.0.3:

  • Compression:
  • Support list compression: compress_list, decompres_list
  • Distance:
  • Change distance.function -> vector.distance

Ver 1.1.0:

  • Added AttrDict, FixedDict, FixedTypeDict

Ver 1.1.1-1.1.7:

  • Fix bugs
  • Added UniqueTypeDict
  • Fix bugs
  • Changed module fixdict -> defaultdict
  • Changed FixedDict -> DefaultDict
  • Changed FixedTypeDict -> DefaultTypeDict
  • Fix bugs of attrdict.AttrDict, attrdict.UniqueTypeDict, attrdict.DefaultDict
  • Fix bugs of compress_list, decompress_list

Ver 1.1.8:

  • Added Environment

Ver 1.1.9-1.1.18:

  • Added Environment.to_lower -> Useful when extracting to kwarg.
  • Fix bugs attrdict
  • Changed hashfunc default from sha1 -> md5
  • Support call of DefaultDict, which update data from buffer.
  • Fix bugs Environment

Ver 1.1.19:

  • Added pre-define output array of decompress_ndarray
  • Support add new key environment.

Ver 1.1.20-1.1.21:

  • Edit logger
  • Fix bugs vector.pool
  • Changed VectorPool base on numpy.mmap -> VectorPoolMMap
  • Added VectorPool base on numpy.ndarray
  • Restructure VectorPoolBase -> Speed up, fix bugs
  • Added VectorPoolBase.insert

Ver 1.1.22:

  • Add document of compression, media module.
  • Fix bugs image
  • Change media.image.scale -> media.image.zoom

Ver 1.1.23-24:

  • Fix bugs log
  • Fix bugs VectorPoolMMap.MIN_SIZE

Ver 1.1.25-26:

  • Fix bugs attrdict
  • Added add, remove method attrdict
  • Rename variable of pool.VectorPoolBase

Ver 1.1.27:

  • Change namespace attrdict.UniqueTypeDict -> attrdict.TypedDict
  • Restructure attdict.AttrDict, attdict.DefaultDict, attrdict.TypedDict
  • Support annotations alias
  • Speed up attrdict
  • Support initial attrdict with iterable, generator.
  • Support to_buffer method with optimize compression.
  • Support recreate dict with from_buffer.
  • Lightweight than pickle.dumps

Ver 1.2.0:

  • CV:
    • Added Face Aligner
    • Transforms image methods
  • media.image:
    • Added scale, crop_center, flip
    • Fix bugs
  • media.video: support FPS report.
  • triangle (new)

Ver 1.3.0:

  • Support decorator attrdict
  • Support decorator environment
  • Implemented Confluent-Kafka with:
    • Iterator method
    • Close event.
    • Auto_commit
    • Optimize configs for ThroughOut
    • Fix bug KafkaError

Todo:


Features


Compression Lossless:

  • Support type: ndnumpy, bytes
  • Compress by blosc. It support multi compressor and multi-thread.

KafkaWrapper:

  • Wrapping Consumer and Producer with default setting and security.

RedisWrapper

  • Wrapping Redis Connector with default setting and security.

Vector Pool:

VectorPoolBase handle numpy.ndarray that save a lot of RAM by re-use existed memory space.

Easier than numpy.ndarray. Likely list.

  • Reduce RAM
  • High Speed IO
  • Auto Scale
  • Support backup and recovery data in a second.

Media:

Implemented OpenCV with:

To: improve read & write (image, video) IO speed.

  • Faster than OpenCV:

    • 2-6x with JPEG Encoder
    • 1.1x with others.
  • Added some function which used frequently.

  • More info: find in dpsutil.media

Computer Vision (cv):

  • Added Face Align with five landmark.

Distance:

  • Added cosine_similarity
  • Added cosine
  • Added euclidean_distance
  • Added convert distance functions

*Note: all function execute in numpy.

Attributes dict:

  • AttrDict: will help you get value of key via attributes. Implement attrdict.AttrDict
  • DefaultDict: help cover your dict with (keys, values) that was defined before. Implement from AttrDict
  • TypedDict: help cover your dict when set item. Implement from AttrDict.
  • DefaultTypeDict: dict only access one type for all element.
  • Support initial attrdict with iterable, generator.
  • Support to_buffer method with optimize compression.
  • Support recreate dict with from_buffer.
  • Lightweight than pickle.dumps

Environement:

  • Environment: Auto pair environment parameters fit with default, which provided before.

Other:

  • Hashing
  • Sort

Issue:


Cmake error during install blosc

Firstly, you need install scikit

$ pip install scikit-build

After that, follow instuction to install Cmake:

https://cliutils.gitlab.io/modern-cmake/chapters/intro/installing.html

Python

$ pip install cmake>=3.12   

Not found 'FFmpeg':

Find FFmppeg lib at:

https://www.ffmpeg.org/download.html

Linux:

sudo apt-get install ffmpeg

Not found 'libjpeg-turbo':

Find FFmppeg lib at:

https://libjpeg-turbo.org/Documentation/OfficialBinaries

Linux:

$ sudo apt install libturbojpeg

Not found Redis or Kafka server:

  • Make sure your Redis or Kafka server started.
  • Make sure correct username & password.

Not found 'Could NOT find OpenSSL':

SSL development libraries have to be installed

https://github.com/openssl/openssl#build-and-install

Ubuntu:

$ apt-get install libssl-dev libffi-dev

Develop by: TinDang
OS: Ubuntu Bionic 18.04
Python: 3.6.9 stable

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dpsutil-1.4.1.tar.gz (50.1 kB view hashes)

Uploaded source

Built Distribution

dpsutil-1.4.1-py3-none-any.whl (59.4 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page