Skip to main content

the set of common tools

Project description

sine.utils

the set of common tools

tools

Class ConfigFileAlone

Simplify read/write on a Json config file. Based on module config_file. Main features:

  • Most used in single program to create and read from a local config file.
  • Specify your default config value in your code and then it will be add to the file.
  • Don't worry about whether the file exists, and ONLY when config changed the file will be saved.
try:
    config = ConfigFileAlone('json_file_path')
    save_path = config.getAlone('prop1', 'default')
    api_path = config.getAlone('prop2', 2)
    # ... your work
    config.saveAlone()
except Exception:
    print("can not load config file")

Class PredictableProcessBar

Process bar just wrap module tqdm and provide new members to make the progress change more smoothly when the unit size is not equivalent and keep unknown in some time and need to predict. Using simple algorithm. For example, to download 100 picture from the website, you create 10 threads to do this, you set the origin total to 100 and never know the true size of the picture you have not read in.

import requests
from sine.utils import *
bar=PredictableProcessBar(total=100)

def download_pic(url, path, bar):
    with closing(requests.get(url, stream=True)) as resp:
        size = int(resp.headers['Content-Length'])
        # you now know the true 'size' of one of origin total size in the bar, call this for adjust the true process and help to prediction
        bar.load_in(size, 1)
        with open(path, 'wb') as code:
            for chunk in resp.iter_content(chunk_size=1024*1024): # buffer size: 1MB
                code.write(chunk)
                # send the true size just for log, and the origin size for update, and optional the core cost time for log
                bar.update_out(len(chunk), len(chunk)/size)
                bar.set_postfix_str(byte2readable(bar.loaded_avg * bar.n / (time.time() - bar.start_t)) + '/s')

Class EventManager

provide event sending and listening. use a hashable key to identify an event.

def f(key, data):
    print(data)

manager = EventManager()
manager.start() # start listen

manager.addListener(key='evnet_key', listener=f)

manager.sendEvent(key='evnet_key', 'print hello')


# other methods
manager.removeListener(key='evnet_key', f)
manager.stop() # stop listen
manager.clear() # clear event

Class Path

convenient to join file path in a chain manner:

s = Path('.')
s = s.join('a', 'b').join('..')
# s == 'a'

it uses os.path.join and always normalizes the path with os.path.normpath
because os.path.join join the 'a', '..' to 'a/..'

Module properties

read/write .properties file in line-oriented format
key=value per line through function.

the code refers to java.util.Properties in Java 1.6.

Common Usage

  • file I/O:

    # input
    properties = load(file)
    # or update existing dictionary
    load(file, properties)
    
    # output
    store(file, properties)
    
  • custom input:
    class LineReader read in one key-value data. It skips all comment lines,
    blank lines, leading whitespace, and processes multi-line data.
    loadSingle(string) read each piece of data given above to key-value.

    for line in LineReader(file): # each key-value line has no line seperator
        key, value = loadSingle(line)
        # do something
    
  • custom output:
    storeComments(writable, comments, linesep=os.linesep))
    write comment (accept multi-line), can specify the line terminator.
    storeSingle(writable, key, value, sep='=', linesep=os.linesep)
    write one key-value, can specify the seperator and the line terminator.

    storeComments(file, 'this is a comment')
    storeSingle(file, 'key', 'value')
    

File Format

normally each line is comment line or a key-value pair.

main features:

  • seperate key and value by one of =, :, , \t
  • ignore whitespaces leading in a line or around = or :
  • comment line begin with # or !
  • escape unicode by \uxxxx
  • escape special characters by adding \

others:

  • data line ends with \ discard the line break

differences with Java:

  • store method will not write datetime comment

Thread classes

def func(a, b, stop_event):
    while 1:
        if stop_event.is_set():
            break
        # do your work

thread = StoppableThread(target=func, args=('a', 'b'))
thread.start()
# ...
thread.stop(-1) # stop and join forever
# thread.stopped() == True


thread = ReStartableThread(target=func, args=('a', 'b'), event_name='stop_event') # can specify the parameter's name
thread.start()
# ...
thread.stop(1) # stop and join for 1 second
# ...
thread.start()
# ...
thread.stop()
thread.join()
# ...

sine.storage

Brief

provide simple reliable persistence for string data base on a string key.

data changes will append to the file like logging while update. data are store in csv format. you can compress the data anytime reliably.

Examples

# common use
storage = getStorage('./data.csv')
author = storage.setdefault('author', 'sine')
storage['author'] = 'Sine'
del storage['author']

for k in storage.keys():
    print(k, storage[k])

# compress data
storage.compress()

Change Log

v0.2.1, 2021-12-26

  • [fix] using Thread.is_alive instead of Thread.isAlive (not found)
  • [fix] bugs in PredictableProcessBar

v0.2.0, 2021-12-26

  • [new] class PredictableProcessBar
  • [new] class ConfigFileAlone
  • [new] functional: byte2readable, mkdir
  • [fix] using collections.abc.Iterable

v0.1.1, 2019-12-14

  • EventManager's process thread defaults to be daemon, can be set.

v0.1.0, 2019-7-21

collect from exist package:

  • sine.path-v0.1.3
  • sine.threads-v0.1.7
  • sine.event-v0.0.2
  • sine.properties-v0.1.1

and the new 'storage'.

news:

  • EventManager: change arguments passing, include the key (just like calling the sendEvent)
  • threads: fix about args appending

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sine.utils-0.2.1.tar.gz (26.6 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page