The ObjDict class has many uses including: as a tool for processing and generating json information, for ad-hoc classes and mutable named tuples, or just as dictionaries that allow dot notation access.
Project description
Uses
Why an ‘ObjDict’? The reasons include:
Background
Instructions
Uses
The ad-hoc structure/object ‘swiss army knife’ class
As described in this ‘uses’ section, the ObjDict class has many uses, and can
be used in place of namedtuples, namedlists, OrderedDict objects, as
well as for processing JSON data. One single import gives this flexibility.
The one trade-off for this flexibility, compared to using the individual specialised classes is performance. If you have performance critical code that is used in massively iterative loops then, for example, namedtuples are far better, as long as namedtuples provide all the functionality you require. But every last nanosecond is not of the essence and flexibility to adapt and simply code is desired, then ‘ObjDict’ can be a replacement for several other classes, plus provide best tools for working with JSON data.
Support for JSON message encoding and decoding
Where an application has the need to build JSON data to save or transmit, or to decode and process JSON data loaded or received, the ObjDict structure provides all the tools to achieve this, with clear object oriented code. This usage has different requirements than JSON serialisation (as discussed below), as it is necessary to be able to produce not just a JSON representation of an object, but create objects that can describe any required possible arbitrary JSON data to produce or decode specific messages. For example, the order of fields may be significant in a JSON message, although field order may not be significant for object serialisation. The ObjDict class has the tools to produce exactly the JSON data required by any application, and to decode any possible incoming JSON messages for processing. It was for this usage that ObjDict was initially developed.
ObjDict in place of dictionaries as convenient ad-hoc data structures
See the text below on ‘multiple uses of dictionaries’ for background. There is a significant amount of code where dictionaries have been used for ad-hoc structures. The use case often arises where it can become useful if these data structure can have elements accessed in the simpler are many.
Mutable equivalent to nametuple (or namedlist)
There are occasions where a ‘namedtuple’ cannot be used due to the need for mutable objects. The ObjDict also fulfills this need and can be initialised from list data. There are many other classes that also fill this need, but the ObjDict combines this functionality with JSON processing, with dictionary access to data and other functions.
Adding JSON serialization to classes
Applications that have a need to serialise objects in order to restore those objects either within the same application, or in an application connected through a data-link, may desire JSON as the format for object storage or object message format. The ObjDict class and module provides the tools for this, serialising the state of an object in order for that state to be later loaded, either by an identical class, or a different class which has use for some or all of the same ‘object state’ information.
OrderedDict alternative
OrderedDicts do everything dictionaries can, and in some applications it can be useful to simply move to OrderedDict classes for all dictionaries. ‘ObjDict’ is another alternative, with a shorter name, even more flexibility and power, and a much more readable ‘str’ representation that can also be used for clearer initialisation. See instructions for details on ‘str’ and initialisation flexibility.
Background
History and acknowledgements
The project emerged from a need for code to generate and decode JSON messages. Originally the package JsonWeb was selected for the task, but it became clear the use case differed. ‘JsonWeb’ is ideal for representing classes as JSON, and reloading classes from that JSON and provides validation and tests and schema that are not reproduced in ObjDict. However ObjDict provides specifically for classes created to generate or process JSON as data, as opposed to JSON as a representation of the class, and now the ObjDict class with a wider range of uses. The whole issue of JSON data which ambiguously may correspond to either a dictionary collection, or an object, arises from general processing of JSON data and gives rise to the ObjDict. The ObjDict project started out to add more control over JSON as a fork of JsonWeb, but evolved over time to the different use cases.
JsonWeb alternative to ObjDict JSON processing
The project ‘JsonWeb’ overlaps is use cases with this project. The focus of ‘JsonWeb’ is to provide for serializing python object structures and instancing python objects from the serialized form. ObjDict can be used for this role also, but currently lacks the validation logic used by ‘JsonWeb’ to ensure JSON data matches exactly the required format.
In fact, rather than an emphasis on validation, the original primary use case of ObjDict is to allow maximum flexibility for the JSON data representing an object. The ObjDict object itself is a generic object to enable working with JSON data without having a matching object definition. Beyond the ObjDict class, the entire ObjDict-JSON processing philosophy is to provide for information sent between computer systems with flexible, adaptable message handling. Where, for example, the message specification may evolve from version to version. This requires flexible interpretation of data, and the ability to easily ignore additional data that may have been added in later versions, providing easy backward compatibility.
The structure for JSON dump and load is a very flexible framework, and any feature including more rigid validation could easily be added.
Multiple uses of dictionaries
In python, dictionaries are designed as ‘collections’ but are often used as ad-hoc structures or objects. In a true collection, the key for an entry does not indicate properties of the value associated with the key. For example, a collection of people, keyed by names would not normally infer the significance or type of data for each entry (or in this case person) by the key. The data has the same implications regardless of whether the key is ‘bob’ or ‘jane’. The data associated with ‘bob’ or ‘jane’ is of the same type and is interpreted the same way. For an ‘ad-hoc’ structure the keys do signal both the nature of the data and even the type of data. Consider for each entry for a person we have a full name and age. A dictionary could be used to hold this information, but this time it is an ad-hoc structure. As a dictionary we always expect the same two keys, and each is specific to the information and different keys even have different types of data. This is not a dictionary as a collection, but as an ad-hoc structure. These are two very different uses of a dictionary, the collection the dictionary was designed for, and the ad-hoc structure or ad-hoc object as a second use.
Introducing the ObjDict
An ObjDict is a subclass of dictionary designed to support this second ‘ad-hoc object’ mode of use. An ObjDict supports all normal dict operations, but adds support for accessing and setting entries as attributes.
So:
bob['full_name'] = 'Robert Roberts'
is equivalent to:
bob.full_name = 'Robert Roberts'
Either form can be used. ObjDicts also have further uses.
Multiple modes of dictionary use and JSON
The standard JSON dump and load map JSON ‘objects’ to python dictionaries. JSON objects even look like python dictionaries (using {} braces and a ‘:’). In JavaScript, objects can also be treated similarly to dictionaries in python. The reality is some JSON objects are best represented in python as objects, yet others are best represented as dictionaries.
Consider:
{ "name": {"first": "fred", "last": "blogs" }
"colour_codes": {"red": 100, "green": 010, "yellow": 110, "white": 111 }
}
In this data, the ‘name’ is really an object but ‘color_codes’ is a true dictionary. Name is not a true dictionary because it is not a collection of similar objects, but rather something with two specific properties. Iterating through name does not really make sense, however iterating through our colours does make sense. Adding to the collection of colours and their being a variable number of colours in the collection is all consistent. Treating ‘name’ is not ideal as the ‘keys’ rather than being entries in a collections each have specific meaning. Keys should not really have meaning, and these keys are really ‘attributes’ of name, and name better represented as an object.
So two types of information are represented in the same way in JSON.
Another limitation of working with python dictionaries and JSON is that in messages, order can be significant but dictionaries are not ordered.
The solution provided here is to map JSON ‘objects’ to a new python ObjDict (Object Dictionaries). These act like OrderedDictionaries, but can also be treated as python objects.
So ‘dump’ or ‘__JSON__()’ or ‘str()’ / ‘__str__()’ of the ‘names’ and ‘colour_codes’ example above produces an outer ObjDict containing two inner ‘ObjDict’s, ‘name’ and ‘colour_codes’. Assume the outer ObjDict is assigned to a variable called ‘data’. Each ObjDict can be treated as either an object or a dictionary, so all the code below is valid:
data = ObjDict(string_from_above) name = data['name'] # works, but as 'data' is not a real 'dict' not ideal name = data.name # better first_name = data.name.first first_name = data["name"]["first"] # works but again not ideal red_code = data.colour_codes["red"] # as colour codes is a true collection it will be unlikely to set # members to individual variables, but the code is valid
ObjDict items also ‘str’ or ‘dump’ back to the original JSON as above. However if the original string was changed to:
{ "name": {"first": "fred", "last": "blogs", "__type__": "Name" }
"colour_codes":{"red": 100, "green": 010, "yellow": 110, "white": 111 }
}
The JSON ‘load’ used to load or initialise ObjDict uses an ‘object_pairs_hook’ that checks a table of registered class names and corresponding classes.
If there is an entry in the table, then that class will be used for embedded objects. Entries with no ‘__type__’ result in ObjDict objects, and if the ‘DefaultType` is set then a class derived from the default type, with the name from the value of ‘__type__’ will be returned. If ‘DefaultType’ is None, then an exception will be generated.
See the instructions section for further information.
ObjDict JSON general
The tools provided allow for dumping any class to JSON, and loading any class from JSON data. There is no requirement for the basing classes on the ObjDict class. The main use of ObjDict is to decode JSON data which is NOT already identified as matching a class within the application. The ObjDict provides the catchall.
The main challenge is not the specific class being loaded or dumped, but the objects within that class.
Consider loading an object properties from JSON. A simple loop to use each JSON field to set each attribute, and the class to be set is simply one class. However, what if some of those fields are themselves objects, and possibly fields within those again objects? Within the single ‘top-level’ object, there may be many embedded objects and identifying and processing these embedded objects is the actual challenge.
In general, handling embedded objects is achieved through the ‘__from_JSON__’ class method within each class for the ‘JSON.load’, or the ‘__JSON__’ method within each object for the ‘JSON.dump’.
Standard routines to perform these methods are available, together with the tools to easily decorate classes and other utilities.
ObjDict JSON load tools
The three main tools for loading JSON objects are an ‘object_pairs_hook’ method to be passed to the standard ‘JSON.load’ function, the ‘__from_JSON__’ class method that can be added to any class to control instancing the class from JSON and the ‘from_JSON’ decorator.
The philosophy is the use of simple, flexible building blocks.
object_pairs_hook
A class within the objdict module, ‘ObjPairHook’, is a wrapper tool to provide a function for the standard library JSON.load() function. Simply instance an ObjPairHook and pass the ‘from_JSON’ method to JSON_load(). eg:
hook=ObjPairHook().from_JSON
JSON.load(object_pairs_hook=hook)
class ObjPairsHook()
def __init__(classes_list=[],BaseHook=None,BaseType=None):
The ‘from_JSON’ method will check all JSON objects for a ‘__type__’ entry, or use ‘default’ processing. For objects with a ‘__type__’, both the entries in the ‘classes_list’ parameter and the default_classes_list maintained within the objdict module and added to through the ‘from_JSON’ decorator, can be instanced if there is a name match.
For objects with ‘__type__’ entries but no name match with either source of classes then the a dynamic class based on ‘BaseClass’ is generated and selected as the ‘class’.
For objects with no ‘__type__’ entry, then the ‘BaseHook’ is selected as the ‘class’ (although in practice is it also possible to use a method rather than a class).
Once a class is selected, then if this class has a ‘__from_JSON__’ attribute, then this class method is called to instance an object, otherwise the normal init method for the class is called.
__from_JSON__ class method
Providing a ‘__from_JSON__’ class method is called to instance an the object by the ‘object_pairs_hook’ if an attribute of this name is present.
from_JSON decorator
The from_JSON decorator, when used to decorate a class, adds the class to default_class list used by the object_pairs_hook.
ObjDict JSON dump tools
The ‘__JSON__’ method, JSONEncoder class, the @to_JSON decorator and the
JSON_registry of to_JSON converters are the main
tools for encoding JSON. Whereas JsonWeb takes an approach of decorating classes
with configuration information to allow the encoder class to produce the JSON
output, ObjDict uses a JSONEncoder that delegates the encoding to ‘__JSON__’
method within each object, or from a table of class/converter pairs.
JSONEncoder class
The JSON_encoder class does the actual encoding, and for each object it first checks for a ‘__JSON__’ method and class that method if present. For objects defined outside of scope e.g. Decimal(), the encoder checks the encoder_table for a matching entry and if present calls that encoder.
to_JSON decorator
This decorator checks if the class has a ‘__JSON__’ method, and if not, decorates the class with a default ‘__JSON__’ method. The ‘__JSON__’ method itself is then decorated with any configuration data.
__JSON__ method
For any object this is either a function or a bound method to be called with the object to be encoded as a parameter. The method should return either a string or a dictionary to be included included in the JSON output.
JSON_registry
This is an object which can be imported from the objdict module to access the
‘add_to’ method (JSON_registry.add_to(<class>,<method/function>). By default, the
table contains entries for Decimal, datetime.datetime and datetime.time.
Any entry can be overwritten by simply adding new values for the same class.
Instructions
General notes and restrictions
Since valid keys for an ObjDict may not necessarily be valid attribute names (for example an integer can be a dictionary key but not an attribute name, and dictionary keys can contain spaces), not all key entries can be accessed as attributes. Similarly, there are attributes which are not considered to be key data, and these attributes have an underscore preceding the name. Some attributes are part of the scaffolding of the ObjDict class and these all have a leading underscore, as well as a trailing underscore. It is recommended to use a leading underscore for all class ‘scaffolding’ added as extensions to the ObjDict class or to derived classes, where this scaffolding is not to be included as also dictionary data.
Initialisation and JSON load
ObjDict can be initialised from lists, from JSON strings, from dictionaries, from parameter lists or from keyword parameter lists.
Examples:
a = ObjDict('{"a": 1, "b": 2}')
class XYZ(ObjDict):
__keys__ = 'x y z'
xyz = XYZ(10,20,30)
xyz.y == 20
Initialisation from lists or parameter lists
Initialisation from a list of key value pairs, as with OrderedDict class is supported. Beyond key value pairs, there is also support for direct initialisation from lists. The ‘_keys’ parameter must be included for initialisation from lists. Also, Classes derived from ObjDict can have ‘_keys’ as a class attribute, providing a similar use pattern to the ‘namedtuple’. ‘_keys’ can be either a list of strings, or a string with space or comma separated values. When initialising from a list or parameter list, the list size must match the number of keys created through ‘_keys’, however other items can be added after initialisation.
So this code produces True:
class XY(ObjDict):
__keys__ = 'x y'
sample = XY(1, 3)
sample.x, sample.y == 1, 3
Alternatively the form to produce a similar result but with the SubClass would be:
sample = ObjDict(1, 3, __keys__ = 'x y') sample = ObjDict([1, 3] ,__keys__ = 'x y')
Initialisation from JSON strings
For more complex initialisation, JSON strings can provide an ideal solution. This allows for complex structures with nested/embedded ‘ObjDict’ or other objects.
Note that initialising from either dictionaries or keyword parameters will result in the order being lost.
For example:
>>> ObjDict(a=1, b=2, c=3)
{"c": 3, "b": 2, "a": 1}
>>> ObjDict({"a": 1, "b": 2, "c": 3})
{"a": 1, "b": 2, "c": 3}
So initialisation from a JSON string is useful if key order is important.
Initialisation from dict, OrderedDict, or key word arguments
As discussed already, initialisation from dict or key word arguments will not maintain order of keys, but if order is not important, such as when the data has already been inserted into a dictionary.
‘str’ and JSON dumps
A limitation with OrderDict objects is that ‘str’ representation can be clumsy when the structure is nested.
The ‘__str__’ method of ObjDict class calls the ‘__JSON__’ method. ‘__str__’ can be overridden without disturbing the ‘__JSON__’ method.
Custom classes and JSON
Custom classes allow for JSON data to result in instantiating objects other
than ObjDict from JSON data. These custom classes can be sub-classed from ObjDict
or built simply using the @to_JSON() and/or @from_JSON() decorators.
Sub-classing ObjDict
The from/to decorators are not required if sub-classing from ObjDict.
JSON.dumps from decorators
The alternative to subclassing ObjDict avoids inheriting other properties of
ObjDict which may not be relevant to the application. The @to_JSON decorator
decorates a class with a ‘__JSON__’ method, and if JSON.dumps() is called as follows:
from objdict import JSONEncoder import JSON JSON.dumps(my_object, cls = JSONEncoder)
Alternate method using objdict.dumps:
import objdict objdict.dumps(my_object)
Then all decorated classes will be encoded using their ‘__JSON__’ method, in addition to any classes in the JSON_registry.
JSONEncoder and JSON_registry
The JSONEncoder encodes all classes added to the JSON_registry, as well as any class with a ‘__JSON__’ method. Classes such as datetime.date or decimal.Decimal are standard library classes and it may not be convenient to sub-class these to have a ‘__JSON__’ method. For these cases, calling the add_to method of the JSON_registry allows adding these objects to be encoded.
For example:
from objdict import JSON_registry JSON_registry.add_to(datetime.date, str)
This will ensure JSONEncoder will use the ‘str’ function to encode dates.
JSON.loads from decorators
The @from_JSON() decorator adds the class to the class register internal to the
objdict module, to then be used by the ‘object_hook_pair’ function provided
as a parameter to the JSON.loads function.
ObjPairHook().decode()
To call JSON.loads, instance an ObjPairHook object and then pass the decode method of that object to JSON.loads.
The decode method will, for all classes in the load_class_register, check if the class has a ‘__from_JSON__’ class method, and if present, call the ‘__from_JSON__’ class method will be called to instance an object from the set of key, value pairs.
For example, if you have:
{ "name":{
"first": "joe",
"last": "foo"
}
}
# now code
@objdict.from_JSON()
class Name:
def __init__(self, first=None, last=None, **kwargs):
self.first = first
self.last = last
Read with:
loads(string)
then convert the name dictionary into an object and put that object back in the original tree:
tree = combiParse(string) tree['name'] = Name(**tree['name']) # kwargs!!! i.e. "**" required :-)
The result would be ‘unParsed’
{ "name":{
__type__: "Name"
"first": "joe",
"last": "foo"
}
}
Decoding automatically to objects can then be added at a later time.
Maintaining order with custom classes and defaults
ObjDict classes and automatically created classes currently maintain key order, but of course cannot provide for default values for attributes.
Custom classes can specify default values for attributes, but currently custom classes do not automatically maintain order, even if based on ObjDict classes.
Maintaining order and supporting default values are available with an ‘__init__’ method. Note, the order attributes are set will be their order in a message. Classes sub-classed from ObjDict will have ‘__type__’ at the end of JSON output.
If a custom class is decorated with @decode.from_object(JSONSimpleHandler),
then all fields in the raw JSON will be sent in a single dict. Of course, as
a dict order is lost and also there are no default values.
The recommended code for the init is something like this:
@objdict.from_JSON()
class Custom(ObjDict):
def __init__(self, *args, **kwargs):
super(Custom,self).__init__()
if args:
arg0 = args[0]
assert len(args) == 0, "unexpected argument"
self.arg1 = arg0.pop('arg1', default)
self.arg2 = arg0.pop('arg2', default)
........
self.update(arg0)
self.update(**kwargs)
Life is much simpler with @decode.from_object(), but at the expense of ignoring
any unexpected arguments. Currently **kwargs will always be empty in this case
but a future update will likely address this.
Example:
@decode.from_object()
class Custom(ObjDict):
def __init__(self,arg1=None, arg2=None ...., **kwargs):
super(Custom,self).__init__()
self.arg1 = arg1
self.arg2 = arg1
........
self.update(**kwargs) # currently kwargs is empty
All that is needed as imports is above.
This system supports both ‘ObjDict’ and custom classes. In JSON representation a ‘__type__’ field is used to indicate actual type. For your own classes use:
@encode.to_object()
@decode.from_object()
class Sample:
def __init(self, p1, p2, ...):
self.p1 = p1
self.p2 = p2
....
to map between:
{ "p1": 1, "p2": 2, "__type__": "Sample"}
and:
Sample(1,2)
However simple examples such as this could also use the default ‘ObjDict’ objects.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for objdict-0.2.2-py2.py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 37b9319231e595fd627a336b9c2e15890ae5863f9323b77f518b20dc62aeff3a |
|
| MD5 | bbda1aac2a3c4ffc95480e77497728f7 |
|
| BLAKE2b-256 | 67173806d229e9f649c49c624b4c50dd66bc32e0f591189931213e5abd5615ef |
