ppss-pyramidutils 1.6.1.3

Simple utils to handle data from ini files in Pyramid for python 2.7 & 3

Overview

A collection of simple utils for common operations This includes configuration item from .ini file, a filemanger for hadling file uploads, some utilities for SQLAlchemy models and a CSV reader/writer unicode-safe for python 2.7

Configuration from .ini

The class ppss_pyramidutils.Utils offers convenience options to get data from the main ini file, allowing default values. The easiest way to use it is to subclass it with your conf class and override the myconf property:

from ppss_pyramidutils import Utils as IniUtils

class MyClass(IniUtils):
  myconf = ['param1','param2']

Then you can read ini file calling the class method config:

MyClass.config(settings)

This method accetps other optional parameters: prefix and defaultval.

If prefix is not passed, lowered case class name is used instead (i.e.: myclass). Ths config method use all values in myconf property and read in the ini file the parameter: prefix.value

In this example it reads myclass.param1 and myclass.param2. If a key in missing in the ini file, the corresponding class property will assume the value defaultval.

myconf can be a tuple/list instead of a string. In this case the first value is the string, the second value is the default value only for that key.

A full example can work as follow:

#ini file
myprefix.param1 = 12
myprefix.param2 = A simple String

from ppss_pyramidutils import Utils as IniUtils

class MyClass(IniUtils):
  myconf = ['param1','param2',('param3',"a missing value with a default"),"param4"]

MyClass.config(settings,"myprefix",defaultval="default")

MyClass.param1
# '12'
MyClass.param2
# 'A simple String'
MyClass.param3
# 'a missing value with a default'
MyClass.param4
# 'param4'

Because they are just class values, you can simply include your class anywhere in your code and access the class property required. This will allow to read and set default for each value in a centralized place and to access it whenever and wherever needed.

filemanger (for upload and other purposes)

ppss_pyramidutils.FileManager extends ppss_pyramidutils.Uitls setting myconf attribute myconf = ['savepath','tmppath'] You can include the class in the ini of the pyramid app and call the config method against it to properly config the class.

FileManager offer the following class methods:

• saveToTmp(cls,requestfile) -> file_path: takes as argument the file object from the form, replace some chars in the filename and save it to a temp location (specified through ini file). This method returns the file path of the temporary file.
• moveToDestination(cls,source,filename,subfolder="") -> file_path: this method takes a file path (tipically the return value of pthe revious saveToTmp call) and moves it in target folder (as of ini file), with the given filename. It can put it in a subfolder if subfolder is specified. This will create the folder if required. Returns the complete path of the file.
• deleteFile(cls,file) -> None : delete a file with the path file

It also has two commodity class methods:

• _sanitizeFilename(cls,filename, whitelist=valid_filename_chars, replace=' ') -> sanitizedfilename: replaces all occurency of each char of replace string with a "_", that removes all not allowed char (allowed chat by default are: -, _, ., (, ), string.ascii_letters and string.digits ). The method returns the sanitized file name. It is called automatically by saveToTmp to prevent attempts of injections in file system.
• slugify(cls,filename) -> sluggifiedfilename: This method tries to convert an arbitrary file name in a sluggied string.

CSV reader/writer for python 2.7

Python 2.7 CSV reader and writer fail to address many unicode problems. You can simply use ppss_pyramidutils.UnicodeReader and ppss_pyramidutils.UnicodeWriter for this purpose.

Both UnicodeReader and UnicodeWriter .__init____ methods accept this parameters:

• f: the already opened file to read.
• dialect=csv.excel: the csv dialect to use.
• encoding="utf-8-sig": encoding to use.

All other keyword arguments are passed on to the CSV reader (check the module in standard Python 2.7 documentation)

For conveninece and use as importer/export of CSV formatted data, two more classes are defined.

Importer class can be initialized with this parameters:

• fn: File name of the source. It will be opened and passed to
• mapping=None: allow to remap column names. If present the mapping param must be a dictionary-like object, containing all CSV column names as key, and the mapped names as values
• delimiter=",": the delimiter for the CSV input
• headertransform=None: a callable that receive each column name and may transform it. Usually used to do some sanitization on the names (ie: lower and remove/substitute forbiden chars)

Creating the Importer object actually triggers the file opening and reading. To get the resulting rows as a list of dictionaries, you can use the method getRows.

Exporter class takes this parameters when initialized:

• evcollection : list-like object, where each item is a dictionary-like (see getter param) object.
• titles : ordered list-like object, containing column names
• parserows = True: if set, allow pre-processing of the input (evcollection), using getter and datetimeformat paramesters.
• delimiter = ',': the delimiter for the CSV output
• getter = None: a callable to override the dictionary like way to get values. If set the getter will be called for each column with the item and name of the column (ie: val = getter(ev,k)). Only used if parserows is True.
• datetimeformat = "%Y-%m-%d" : formatter for datetime.datetime and datetime.date objects. Only used if parserows is True.

This method set the property retfile to a tempfile.NamedTemporaryFile (set to writemode in Python 2)

the method writeAll accept only a delimiter param and actualy writes all rows in the retfile property (by default a tempfile.NamedTemporaryFile instance)

Modelbase utilities

ppss_pyramidutils.ModelCommonParent attach some commodity methods to derived SQLAlchemy classes, as class methods, like:

• all(cls,DBSession): returns all elements of the given class
• byId(cls,id,DBSession): returns the object with id = id. COC: the class must have a id column.
• byField(cls,field,value,DBSession): like byId, on an arbitraty field
• byFields(cls,fields,DBSession): accept a list of filters as tuples of (columnname, value) and returns all items that matches
• delete(cls,element,DBSession): delete the element
• deleteAll(cls,elements,DBSession): delete all elements in elements list-like parameter.

__v1.6.1.3 fix "quote all" in exporter

__v1.6.1 added skip rows to jsonwalker

__v1.6.0 added cryptoutils and jsonutils

__v1.5.4.0 added custom header row to importer __v1.5.3.2 imported names in base init __v1.5.3.1 fix conf reader

v1.5.2

• New background thread in loop interface
• ini reader updated and refactored v1.5
• added/refactored functions to have dbsessions for off-request activities v1.5
• added utility for background jobs v1.4.4.5
• fixed encoding of file in csv importer (now forced to utf-8)
• create tmp dir if not exists v1.4.4.4
• addedd order by and order direction to modelbase commodity methods v1.4.4.3
• fix in utf8csv change six unicode method to str method for py3 compatibility v1.4.4.2
• fix in modelbase str method for py3 compatibility 1.4.3.1
• added check on int/float before encoding for UTF-8 writer