Skip to main content

AlexBerUtils is collection of the small utilities

Project description

AlexBerUtils

AlexBerUtils is collection of the small utilities. See CHANGELOG.md for detail description.

Getting Help

QuickStart

python3 -m pip install -U alex-ber-utils

Installing from Github

python3 -m pip install -U https://github.com/alex-ber/AlexBerUtils/archive/master.zip

Optionally installing tests requirements.

python3 -m pip install -U https://github.com/alex-ber/AlexBerUtils/archive/master.zip#egg=alex-ber-utils[tests]

Or explicitly:

wget https://github.com/alex-ber/AlexBerUtils/archive/master.zip -O master.zip; unzip master.zip; rm master.zip

And then installing from source (see below).

Installing from source

python3 -m pip install . # only installs "required"
python3 -m pip install .[tests] # installs dependencies for tests
python3 -m pip install .[md]   # installs multidispatcher (used in method_overloading_test.py)
python3 -m pip install .[fabric]   # installs fabric (used in fabs.py)
python3 -m pip install .[yml]   # installs Yml related dependencies 
                                # (used in ymlparsers.py, init_app_conf.py, deploys.py;
                                # optionally used in ymlparsers_extra.py, emails.py)
python3 -m pip install .[env]   # installs pydotenv (optionally used in deploys.py and mains.py)

Alternatively you install install from requirements file:

python3 -m pip install -r requirements.txt # only installs "required"
python3 -m pip install -r requirements-tests.txt # installs dependencies for tests
python3 -m pip install -r requirements-md.txt   # installs multidispatcher (used in method_overloading_test.py)
python3 -m pip install -r requirements-fabric.txt   # installs fabric (used in fabs.py)
python3 -m pip install -r requirements-yml.txt   # installs Yml related dependencies 
                                                 # (used in ymlparsers.py, init_app_conf.py, deploys.py;
                                                 # optionally used in ymlparsers_extra.py, emails.py)
python3 -m pip install -r requirements-env.txt   # installs pydotenv (optionally used in deploys.py)

From the directory with setup.py

python3 setup.py test #run all tests

or

pytest

Installing new version

See https://docs.python.org/3.1/distutils/uploading.html

python3 setup.py sdist upload

Requirements

AlexBerUtils requires the following modules.

  • Python 3.6+

  • PyYAML==5.1

Changelog

All notable changes to this project will be documented in this file.

#https://pypi.org/manage/project/alex-ber-utils/releases/

[Unrelased]

see [https://github.com/alex-ber/AlexBerUtils/issues/5]

Added

  • optional Dockerfile
  • optioanl .env.docker for refference.
  • Support of Python 3.8

[0.6.0a] - 02/11/2020

Changed

  • requirements*.txt - dependencies version changed, see https://github.com/alex-ber/AlexBerUtils/issues/6
  • Because of pytest upgrade conftest.py was changed: pytest_configure() was added to support dynamically used marks.
  • In ymlparsers_test.py deprecation warning removed (it will be error in Python 3.9) collections.Mapping was changed to collections.abc.Mapping.

[0.5.3] - 10/09/2020

Changed

  • alexber.utils.emails.initConfig is fixed. Before this default variables where ignored.
  • 2 Unit tests for init_app_conf are fixed. These fix are minors.

Documentation

  • importer module [https://medium.com/analytics-vidhya/how-to-write-easily-customizable-code-8b00b43406b2]
  • fixabscwd() function in mains module. [https://medium.com/@alex_ber/making-relative-path-to-file-to-work-d5d0f1da67bf]
  • My parser module [https://medium.com/analytics-vidhya/my-parser-module-429ed1457718]
  • My ymlparsers module [https://medium.com/analytics-vidhya/my-ymlparsers-module-88221edf16a6]
  • My major init_app_conf module [https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c]
  • My deploys module [https://medium.com/analytics-vidhya/my-ymlparsers-module-88221edf16a6 for documentation]
  • My emails module [https://medium.com/analytics-vidhya/my-emails-module-3ad36a4861c5]
  • My processinvokes module [https://medium.com/analytics-vidhya/my-processinvokes-module-de4d301518df]

[0.5.2] - 21/06/2020

Added

  • path() function in mains module. For older Python version uses importlib_resources module. For newer version built in importlib.resources.
  • load_env() function in mains module. Added kwargs forwarding. if dotenv_path or stream is present it will be used. if ENV_PCK is present, dotenv_path will be constructed from ENV_PCK and ENV_NAME. Otherwise, kwargs will be forwarded as is to load_dotenv.
  • fix_env() function in mains module. For each key in ENV_KEYS, this method prepends full_prefix to os.environ[key]. full_prefix is calculated as absolute path of __init__.py of ENV_PCK.

Changed

processinvokes function run_sub_process - documentation typo fixed. Lower Python version to 3.6.

[0.5.1] - 06-05-2020

Added

This method is Linux-like cp command. It copies single file to remote (Posix) machine.

  • Spited dependency list for setup.py req.txt (inexact versions, direct dependency only) and for reproducible installation requirements.txt (exact versions, all, including transitive dependencies).

  • Added req-fabric.txt, requirements-fabric.txt - Fabric, used in fabs module.

  • Added req-yml.txt, requirements-yml.txt - Yml-related dependencies, used in ymlparsers.py and in init_app_conf.py, deploys.py; optionally used in ymlparsers_extra.py, emails.py.

Main dependency is HiYaPyCo. I'm using feature that is availlable in the minimal version.

HiYaPyCo depends upon PyYAML and Jinja2. Limitations for Jinja2 is from HiYaPyCo project.

  • Added req-env.txt, requirements-env.txt - pydotenv, optionally used in deploys.py.

  • Added inspects.has_method(cls, methodName). Check if class cls has method with name methodName directly, or in one of it's super-classes.

  • Added pareser.parse_sys_args function parses command line arguments.

  • Added ymlparsers module - load/safe_dump a Hierarchical Yml files. This is essentially wrapper arround HiYaPyCo project with streamlined and extended API and couple of work-arrounds.

Note: this module doesn't use any package-level variables in hiYaPyCo module, including hiYaPyCo.jinja2env. This module do use Jinja2's Environment.

It also has another defaults for load/safe_dump methods. They can be overridden in initConfig() function.

safe_dump() method supports simple Python objects like primitive types (str, integer, etc), list, dict, OrderedDict.

as_str()- convenient method for getting str representation of the data, for example of dict.

DisableVarSubst - use of this context manager disables variable substation in the load() function.

initConfig - this method reset some defaults. If running from the MainThread, this method is idempotent.

  • Added init_app_conf major module.

The main function is parse_config. This function parses command line arguments first. Than it parse yml files. Command line arguments overrides yml files arguments. Parameters of yml files we always try to convert on best-effort basses. Parameters of system args we try convert according to implicit_convert param.

If you supply implicit_convert=True, than mask_value() will be applied to the flat map (first parameter). Otherwise, implicit_convert wiil have the value that was set in intiConfig(). By default it is True.

Command line key --general.profiles or appropriate key default yml file is used to find 'profiles'. Let suppose, that --config_file is resolved to config.yml. If 'profiles' is not empty, than it will be used to calculate filenames that will be used to override default yml file. Let suppose, 'profiles' resolved to ['dev', 'local']. Than first config.yml will be loaded, than it will be overridden with config-dev.yml, than it will be overridden with config-local.yml. At last, it will be overridden with system args. This entry can be always be overridden with system args.

ymlparsers and parser modules serves as Low-Level API for this module.

mask_value() implemented as a wrapper to parsers.safe_eval() method with support for boolean variables. This implementation is used to get type for arguments that we get from system args. This mechanism can be easily replaced with your own one.

to_convex_map() This method receives dictionary with 'flat keys', it has simple key:value structure where value can't be another dictionary. It will return dictionary of dictionaries with natural key mapping, optionally, entries will be filtered out according to white_list_flat_keys and, optionally, value will be implicitly converted to appropriate type.

In order to simulate dictionary of dictionaries 'flat keys' compose key from outer dict with key from inner dict separated with dot. For example, 'general.profiles' 'flat key' corresponds to convex map with 'general' key with dictionary as value that have one of the keys 'profiles' with corresponding value.

If you supply implicit_convert=True, than mask_value() will be applied to the values of the received flat dictionary. Otherwise, implicit_convert wiil have the value that was set in intiConfig(). By default it is True.

merge_list_value_in_dicts - merges value of 2 dicts. This value represents list of values. Value from flat_d is roughly obtained by flat_d[main_key+'.'+sub_key]. Value from d is roughly obtained by d[main_key][sub_key].

If you supply implicit_convert=True, than mask_value() will be applied to the flat map (first parameter). Otherwise, implicit_convert wiil have the value that was set in intiConfig(). By default it is True.

initConfig - you can set default value of implicit_convert. By default it is True. This parameters is used if implicit_convert wasn't explicitly supplied. This method is idempotent.

  • Added deploys module. This module is usable in your deployment script. See also fabs module.

This method use parsers, ymlparsers,init_app_confas it's low-level API.init_app_conf` usage is limited.

The main function is load_config(). It is simplified method for parsing yml configuration file with optionally overrided profiles only. See init_app_conf.parse_config() for another variant.

split_path - Split filename in 2 part parts by split_dirname. first_part will ends with split_dirname. second_part will start immediately after split_dirname.

add_to_zip_copy_function - Factory method that returns closure that can be used as copy_function param in shutil.copytree().

  • Added emails module. This module contains extensions of the logging handlers. This module optionally depends on ymlparseser module. It is better to use EmailStatus context manager with configured emailLogger. It is intended to configure first your emailLogger with OneMemoryHandler (together with SMTPHandler). Than the code block that you want to aggregate log messages from is better to be enclosed with EmailStatus context manager.

alexber.utils.emails.SMTPHandler is customization of logging.handlers.SMTPHandler. It's purpose is to connect to SMTP server and actually send the e-mail. Unlike logging.handlers.SMTPHandler this class expects for record.msg to be built EmailMessage. You can also change use of underline SMTP class to SMTP_SSL, LMTP, etc. This implementation is thread-safe.

alexber.utils.emails.OneMemoryHandler is variant of logging.handlers.MemoryHandler. This handler aggregates log messages until FINISHED log-level is received or application is going to terminate abruptly (see docstring of calc_abrupt_vars() method for the details) and we have some log messages in the buffer. On such event all messages (in the current Thread) are aggregated to the single EmailMessage. The subject of the EmailMessage is determined by get_subject() method. If you want to change delimeters used to indicate variable declaration inside template, see docstring of the get_subject() method. It is better to use EmailStatus context manager with configured emailLogger. See docstring of EmailStatus.
This implementation is thread-safe.

alexber.utils.emails.EmailStatus - if contextmanager exits with exception (it fails), than e-mail with subject formatted with faildargs and faildkwargs will be send. Otherwise, e-mail with subject formatted with successargs and successkwargs will be send. All messages (in the current Thread) will be aggregated to one long e-mail with the subject described in OneMemoryHandler.get_subject() method.

alexber.utils.emails.initConfig - this method reset some defaults. This method is idempotent. By default, SMTP class from smtplib is used to send actual e-mail. You can change it to SMTP_SSL, LMTP, or another class by specifying default_smpt_cls_name. You can also specified default port for sending e-mails.

processInvokes module has one primary function - run_sub_process() This method run subprocess and logs it's out to the logger. This method is sophisticated decorator to subprocess.run(). It is useful, when your subprocess
run's a lot of time and you're interesting to receive it's stdout and stderr. By default, it's streamed to log. You can easily customize this behavior, see initConig() method.

initConig() This method can be optionally called prior any call to another function in this module. You can use your custom class for the logging. For example, FilePipe.

Changed

  • Spited dependency list for setup.py req.txt (inexact versions, direct dependency only) and for reproducible installation requirements.txt (exact versions, all, including transitive dependencies).

  • README.md changed, added section 'Alternatively you install install from requirements file:'. Some other misc changed done.

  • CHANGELOG.md version 0.4.1 misc changes.

  • Misc improvement in unit tests.

  • Fixed parser.safe_eval - safe_eval('%(message)s') was blow up, now it returns value as is. See https://github.com/alex-ber/AlexBerUtils/issues/2

  • Enhanced importer.importer - added support for PEP 420 (implicit Namespace Packages).
    Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. When interpreted encounter with non-empty path attribute it adds modules found in those locations to the current package. See https://github.com/alex-ber/AlexBerUtils/issues/3

  • In all documentation refference to pip3 was changed to python3 -m pip

[0.4.1] - 2020-04-02

BREAKING CHANGE I highly recommend not to use 0.3.X versions.

Removed

  • module warns is droped

Changed

  • Limitation::

mains module wasn't tested with frozen python script (frozen using py2exe).

  • module mains is rewritten. Function initConf is dropped entirely.
  • module mains now works with logger and with warnings (it was wrong decision to work with warnings).

[0.3.4] - 2020-04-02

Changed

  • CHANGELOG.md fixed
  • warns module bug fixed, now warnings.warn() works.
  • FixabscwdWarning is added to simplify warnings disabling.
  • Changing how mains module use warns.

[0.3.3] - 2020-04-02

Changed

  • CHANGELOG.md fixed

[0.3.2] - 2020-04-01

Changed

  • To REAMDE.md add Installing new version section
  • Fix typo in REAMDE.md (tests, not test).
  • Fixing bug: now, you're able to import package in the Python interpreter (setups.py fixed)
  • Fixing bug: warns module now doesn't change log_level in the preconfigured logger in any cases.
  • BREAKING CHANGE: Inmains module method warnsInitConfig() was renamed to mainInitConfig() Also singature was changed.
  • mains module minor refactored.

Added

  • Unit tests are added for warns module
  • Unit tests are added for mains module

[0.3.1] - 2020-04-01

Changed

  • Tests minor improvements.
  • Excluded tests, data from setup.py (from being installed from the sdist.)
  • Created MANIFEST.in

Added

  • warnsmodule is added:

It provides better integration between warnings and logger. Unlike logging._showwarning() this variant will always go through logger.

warns.initConfig() has optional file parameter (it's file-like object) to redirect warnings. Default value is sys.stderr.

If logger for log_name (default is py.warnings) will be configured before call to showwarning() method, than warning will go to the logger's handler with log_level (default is logging.WARNING).

If logger for log_name (default is py.warnings) willn't be configured before call to showwarning() method, than warning will be done to file (default is sys.stderr) with log_level (default is logging.WARNING).

  • main module is added:

main.fixabscwd() changes os.getcwd() to be the directory of the __main__ module.

main.warnsInitConfig() reexports warns.initConfig() for convenience.

Added

  • Tests for alexber.utils.thread_locals added.

[0.2.5] - 2019-05-22

Changed

  • Fixed bug in UploadCommand, git push should be before git tag.

[0.2.4] - 2019-05-22

Changed

  • Fixed bug in setup.py, incorrect order between VERSION and UploadCommand (no tag was created on upload)

[0.2.1] - 2019-05-22

Changed

  • setup url fixed.
  • Added import of Enum to alexber.utils package.

[0.2.0] - 2019-05-22

Changed

  • setup.py - keywords added.

[0.1.1] - 2019-05-22

Changed

  • README.md fixed typo.

[0.1.0] - 2019-05-22

Changed

  • alexber.utils.UploadCommand - bug fixed, failed on git tag, because VERSION was undefined.

[0.0.1] - 2019-05-22

Added

  • alexber.utils.StrAsReprMixinEnum - Enum Mixin that has str() equal to repr().

  • alexber.utils.AutoNameMixinEnum- Enum Mixin that generate value equal to the name.

  • alexber.utils.MissingNoneMixinEnum - Enum Mixin will return None if value will not be found.

  • alexber.utils.LookUpMixinEnum - Enim Mixin that is designed to be used for lookup by value.

    If lookup fail, None will be return. Also, str() will return the same value as repr().

  • alexber.utils.threadlocal_var, get_threadlocal_var, del_threadlocal_var.

    Inspired by https://stackoverflow.com/questions/1408171/thread-local-storage-in-python

  • alexber.utils.UploadCommand - Support setup.py upload.

    UploadCommand is intented to be used only from setup.py

    It's builds Source and Wheel distribution.

    It's uploads the package to PyPI via Twine.

    It's pushes the git tags.

  • alexber.utils.uuid1mc is is a hybrid between version 1 & version 4. This is v1 with random MAC ("v1mc").

    uuid1mc() is deliberately generating v1 UUIDs with a random broadcast MAC address.

    The resulting v1 UUID is time dependant (like regular v1), but lacks all host-specific information (like v4).

    Note: somebody reported that ran into trouble using UUID1 in Amazon EC2 instances.

  • alexber.utils.importer.importer - Convert str to Python construct that target is represented.

  • alexber.utils.importer.new_instance - Convert str to Python construct that target is represented. args and kwargs will be passed in to appropriate new() / init() / init_subclass() methods.

  • alexber.utils.inspects.issetdescriptor - Return true if the object is a method descriptor with setters.

    But not if ismethod() or isclass() or isfunction() are true.

  • alexber.utils.inspects.ismethod - Return false if object is not a class and not a function. Otherwise, return true iff signature has 2 params.

  • alexber.utils.parsers.safe_eval - The purpose of this function is convert numbers from str to correct type.

    This function support convertion of built-in Python number to correct type (int, float)

    This function doesn't support decimal.Decimal or datetime.datetime or numpy types.

  • alexber.utils.parsers.is_empty - if value is None returns True.

    if value is empty iterable (for example, empty str or emptry list),returns true otherwise false.

    Note: For not iterable values, behaivour is undefined.

  • alexber.utils.parsers.parse_boolean - if value is None returns None.

    if value is boolean, it is returned as it is. if value is str and value is equals ignoring case to "True", True is returned. if value is str and value is equals ignoring case to "False", False is returned.

    For every other value, the answer is undefined.

  • alexber.utils.props.Properties - A Python replacement for java.util.Properties class

    This is modelled as closely as possible to the Java original.

    Created - Anand B Pillai abpillai@gmail.com.

    Update to Python 3 by Alex.

    Also there are some tweeks that was done by Alex.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for alex-ber-utils, version 0.6.0a0
Filename, size File type Python version Upload date Hashes
Filename, size alex_ber_utils-0.6.0a0-py3-none-any.whl (47.7 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size alex_ber_utils-0.6.0a0.tar.gz (72.2 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page