Skip to main content

Utilities for RADICAL CybertoolsProjects

Project description

[![Build Status](https://travis-ci.org/radical-cybertools/radical.utils.svg?branch=devel)](https://travis-ci.org/radical-cybertools/radical.utils)

Package radical.utils
=====================

This Python package contains shared code and tools for various
[Radical Group](http://radical.rutgers.edu) projects. Sometimes we call it the
Radical Kitchen Sink.

The radical.utils package contains the following things:

* A [URL class](https://github.com/saga-project/radical.utils/blob/master/radical/utils/url.py) (A SAGA-compliant url parser class)
* A [Plugin manager](https://github.com/saga-project/radical.utils/blob/master/radical/utils/plugin_manager.py) (A simple yet flexible plugin manager and loader)
* A [Config parser](https://github.com/saga-project/radical.utils/tree/master/radical/utils/config) (Config file reader writeer parser)
* A [Logger](https://github.com/saga-project/radical.utils/tree/master/radical/utils/logger) (Support for color log output and tracing)
* An [Object cache](https://github.com/saga-project/radical.utils/blob/master/radical/utils/object_cache.py) (Well, an object cache)
* A [Type checking module](https://github.com/saga-project/radical.utils/blob/master/radical/utils/signatures.py) (Type checking for python call signatures)
* A [Singleton metaclass](https://github.com/saga-project/radical.utils/blob/master/radical/utils/singleton.py) (Implements the Sigleton pattern)
* A [Thread wrapper](https://github.com/saga-project/radical.utils/blob/master/radical/utils/threads.py) (A thin convenience wrapper around threading.Thread)
* A [Call tracer](https://github.com/saga-project/radical.utils/blob/master/radical/utils/tracer.py) (Support for call tracing / debugging)
* A ['which' function](https://github.com/saga-project/radical.utils/blob/master/radical/utils/which.py)(Simliar to the 'which' command line tool)

The package should in general be compatible with Python 2.6.



License
-------

This software is released under the
[MIT License](http://opensource.org/licenses/MIT).

Parts of the module (radical.utils.atfork) are licensed under the [Apache-v2.0
license](http://www.apache.org/licenses/).


Installation
------------

You can install the latest radical.utils directly from [PyPi](https://pypi.python.org/pypi/radical.utils/):

pip install --upgrade radical.utils

You can also install the latest development version (which might be broken)
from GitHub directly:

pip install -e git://github.com/saga-project/radical.utils.git#egg=radical.utils

If you want to use radical.utils in your own project, include it in your
`setup.py` file:

'install_requires': ['radical.utils']


Documentation
-------------

You can generate the documentation yourself with [Sphinx](http://sphinx-doc.org/)

pip install sphinx
make docs

The resulting documentation will be in `open docs/build/html/index.html`.




- For a list of bug fixes, see
https://github.com/radical-cybertools/radical.utils/issues?q=is%3Aissue+is%3Aclosed+sort%3Aupdated-desc
- For a list of open issues and known problems, see
https://github.com/radical-cybertools/radical.utils/issues?q=is%3Aissue+is%3Aopen+



0.60.0 Release 2019-04-10
--------------------------------------------------------------------------------

- fir network interface blacklisting
- add pep8 and pylint configs
- add expand_env
- fix ReString class
- add heartbeat class
- add shell level proc watcher
- add stdio redirect for background callouts
- add zmq support, backported from v2
- add gtod support
- dict mixin cleanup
- clean out stale code
- consolidate configuration system across RCT stack
- fix config error handling and test
- fix config getter
- fix env expansion, apply in config patrser
- fix path interpretation
- fix state event duplicaation
- fix setup and test suite
- improve scheduler viz
- increase timing precision in profiler
- impove shell process mgmt
- more resilient mod loading
- move configuration tooling to util layer
- remove some debug prints
- switch to pytest
- use dict mixin for config


0.50.3 Release 2018-12-19
--------------------------------------------------------------------------------

- fix msg evaluation in process watcher (#149)
- update setup.py (#146)
- a bitarray based scheduler
- use config files for bitarray scheduler configuration
- add install info for macos
- adding git-error in version check (#145)
- fix div-by-zero on fast runs
- handle `None` messages in watcher (#141)


0.50.2 Release 2018-08-20
--------------------------------------------------------------------------------

- fix profile cleanup to work w/o state models


0.50.1 Release 2018-07-03
--------------------------------------------------------------------------------

- fix relocation ID state storage
- fix issue RS-661
- support dynamic code snippet injection
- avoid mkdir race
- expand exe excution logic from RP
- fix array test
- uniform env getter for radical namespace


0.47.5 Release 2018-06-02
--------------------------------------------------------------------------------

- fix and cleanup log, prof and rep settings


0.47.4 Release 2018-03-27
--------------------------------------------------------------------------------

- fix RA issue #65


0.47.3 Release 2018-03-20
--------------------------------------------------------------------------------

- more thorough approach to relocate ID state storage (#131)
- travis badge


0.47.2 Release 2018-02-28
--------------------------------------------------------------------------------

- relocate ID state storage (#131)


0.47.1 Release 2018-02-21
--------------------------------------------------------------------------------

- introduce name spaces for the ID generator
- iterate on some of the debug methods


0.46.2 Release 2017-11-19
--------------------------------------------------------------------------------

- backport of profile handle checks
- add tid arg to prof
- add support for legacy profiles
- clean profile handle check
- add shell callout helper
- function stack recognizes anaconda virtual envs
- reduce profile buffering
- fixes issue #120
- add comments on profil format
- fix #RA-52
- relax time limits on process / thread creation
- radical-stack-clone supports Conda Envs Now!! :)
- add -p to specify python executable to stack-clone
- add a `get_size()` debug method
- cooperative process termination
- make ID generator behave under docker
- move gettid to where it belongs
- add close() method to logger
- expand poller lock scope
- remove some debug prints
- support changes in RA
- improve runtime behavior of concurrency calculation


0.46.2 Release 2017-08-23
--------------------------------------------------------------------------------

- hotfix for RP #1415


0.46.1 Release 2017-08-23
--------------------------------------------------------------------------------

- hotfix for RP #1415


Version 0.46 release 2017-08-11
--------------------------------------------------------------------------------

- Feature/managed process (#104)
- Feature/poll (#106)
- Fix/version strings (#107)
- Revert "Feature/managed process" (#102)
- add a managed process class
- add close() method to logger
- add some doc on raise_in_thread()
- apply ru.Process syntax/semantics to ru.Thread
- be friendly to MacOS
- clean out and comment on utils thread wrapper
- cleanup on ru Process class, tests
- fix logging, leave parent termination alone
- fix process termination test case
- fix raise_on messaging and eval; work on cancellation (#101)
- fix thread locality, logging
- fuck python
- get stack info and clone in sync
- improve error reporting on child processes
- iterate and complete documentation;
- iteration on reliable process management
- make a sanity check non-fatal
- make sure that thread local data are accessible
- move gettid to where it belongs
- proper handling of null logger
- remove potential filename collission in test suite
- rename ru.Thread to ru.Future (#95)
- remove some debug prints/logs, add assert
- simplify radical-stack, make programmatic
- support for cprofile in ru.Thread
- rename ru.Thread to ru.Future


Version 0.45 release 2017-02-28
--------------------------------------------------------------------------------

- add radical-stack, radical-stack-clone tools
- setup fix
- silence a warning
- Silence empty profile warning


Version 0.44 release 2016-11-01
--------------------------------------------------------------------------------

- remote debugging!
- SIGINT is nono in threads
- comment on exception types in raise_in_thread
- add range matching alg
- add condition watcher
- add some tracing methods
- add get_thread_name and get_thread_id
- add raise_in_thread
- add netifaces dep
- hardcode the netiface version until it is fixed upstream.
- allow to overwrite profiler name
- fix double releases
- convert atfork monkeypatch exception into warning
- make singleton fork-safe
- move get_hostip and get_hostname to ru
- move profiler from RP to RU
- move raise_on functionality from rpu to ru, to support random
error triggers for stress testing
- make raise_on thread save, get limits from env


Version 0.41.1 release 2016-06-02
--------------------------------------------------------------------------------

- add fs barrier to debug helper
- add get_traceback
- clean up counter management
- fix a log handler
- fix path creation for logfile output
- Fix regression on query split.
- fix thread inspection problem
- include atfork in RU
- keep doc in sync with implementation
- log version info is now optional
- make sig checking conditional
- make sure log level is string before calling "upper()"
- fix split_module
- monkeypatch python's logging module


Version 0.40 release 2016-02-03
--------------------------------------------------------------------------------

- add an algorithm to create balanced partitions of a space
- add ssl support to mongodb_connect
- allow fragment parsing in Url class
- allow reporter and debug log to coexist if log target is set.
- fix default for get in dict mixin
- fix type conversion for int, float configurables from env vars
- implement reset_id_counters


Version 0.38 release 2015-11-06
--------------------------------------------------------------------------------

- support install on anaconda


Version 0.37 release 2015-10-15
--------------------------------------------------------------------------------

- fix reporter log level
- several changes to reporter class
- align reporter, logger, and logreporter


Version 0.36 release 2015-10-08
--------------------------------------------------------------------------------

- clean up logging (getLogger -> get_logger)
- log pid and tid on log creation
- fix type conversion on some config data types
- make uid generation independent if getpwuid
- fix log message for warning about log level
- add boolean test for url
- add SAGA_VERBOSE backport
- fix some more import shadowing
- fix layer violation: make logs look nice in RP, finally
- do not use signals in threads...
- rename and install version tool.
- fix the debug helper


Version 0.35 release 2015-08-27
--------------------------------------------------------------------------------

- small bugfix release


Version 0.29 release 2015-07-13
--------------------------------------------------------------------------------

- attempt to avoid sdist naming error during pypi installation
- add uuid support, avoid counter reset on reboot
- add two convenience methods which support range finding
- have multiple threads lease from same pool. Speed up LM test
- make sure that the pull is filled during lease manager testing
- make sure json is exported as encoded unicode
- make sure we write encoded strings for json
- add test for LeaseManager


Version 0.28 release 2015-04-16
--------------------------------------------------------------------------------

- minimize an unlock/release race
- resilience against missing git in setup.py
- cleaner version string
- sync setup.py with recent changes in RP
- export range collapsing algorithm
- fix documentation
- fix reporting of type checks when multiple types are valid and optional; fix URL init sig
- fix url tests
- install sdist, export sdist location
- tool rename
- add version tool


Version 0.8 release 2015-02-24
--------------------------------------------------------------------------------

- Preserve case for "hostnames" in GO URIs.
- Allow # in host part of Globus Online URIs.
- preserve trailing slashes on URL paths
- move sources into src/
- fix version strings. again.
- fix id counters to start from 0
- fix comment parsing fix in second json read


Version 0.7.11 release 2014-12-11
--------------------------------------------------------------------------------

- fixed botched merge from devel


Version 0.7.10 release 2014-12-11
--------------------------------------------------------------------------------

- fixed lease manager debug output


Version 0.7.9 release 2014-12-02
--------------------------------------------------------------------------------

- fix to json comment parsing
- att flexible pylint util
- support append / overwrite mode for log targets
- add daemonize class
- fix leaking logs
- reduce lease manager logging noise
- log python version on startup
- add reporter class for nice demo output etc
- add namespace for object_cache, backward compatible
- fix bson/json/timestamp mangling
- json support for json writing


Version 0.7.8 release 2014-10-29
--------------------------------------------------------------------------------

- implemented lease manager (manage finite set of resources with
finite lifetime over multiple threads)
- implemented DebugHelper class (prints stack traces for all threads
on SIGUSR1
- implement decorator for class method timings
- cache configuration settings on logger creation, which
significantly speeds up logging over different log objects
- remove deepcopy from configuration management (improves
performance)
- add wildcard expanstion on dict_merge
- make pymongo and nose dependencies optional


Version 0.7.7 release 2014-08-27
--------------------------------------------------------------------------------

- lease manager which handles resource leases (like, leases ssh connections to saga adaptors)
- fixes on deepcopy, logging and config handling


Version 0.7.5 release 2014-07-22
--------------------------------------------------------------------------------

- Some small bug fixes.


--------------------------------------------------------------------------------

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

radical.utils-0.60.0.tar.gz (176.9 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page