Simple method invocation pickling.
MethodPickle (methodpickle) is a quick library that allows simple pickling and unpickling of function and method invocation. Function & method module loading is handled automatically, and methods can be specified by name as well.
The ability to pickle a method invocation allows for queueing and delayed execution of arbitrary code. This is useful for parallelization, logging, queueing, etc.
Steve Lacy <email@example.com> Twitter: @sklacy http://slacy.com/blog
Features & Usage
Please see the unit tests in test.py for some more verbose examples, but I’ll go through a quick example here.:
from methodpickle.defer import defer # These are the functions that we're going to defer def some_function(x, y): return x*x + y*y # methodpickle supports deferring execution of classmethods as well, so # here's a simple class with a method: def some_class(object): def __init__(self, x): self._x = x def calc(self, y): return (self._x * self._x + y * y) if __name__ == '__main__': # the defer function takes a method and it's arguments, and turns it # into a pickleable object. storable_func = defer(some_function, 5, 4) # So, we pickle that guy into a string. method_str = pickle.dumps(storable_func) # You can now take method_str and do whatever you like with it. Write # it to a database, send it to another process, put it in your logs, # whatever. # Then, you can unpickle the stored method invocation, and run it, # like this: recovered_func = pickle.loads() assert(recovered_func.run() == 5*5 + 4*4) # methodpickle also supports pickling of classmethods. Note that your # class must support pickling and the methods should have no side # effects. i = some_class(2) storable_classmethod = defer(i, 3) classmethod_str = storable_method.dumps() recovered_classmethod = pickle.loads(classmethod_str) assert(recovered_classmethod.run() == 2*2 + 3*3)
For convenience, there’s also a decorator form of the defer function, called deferred. Again, see the implementation or test.py for more details.
- All arguments to functions must themselves be pickle-able. This
- includes ‘self’ for class method invocations
- Functions and classes must be at the module level. Inner classes and
- inner functions don’t have an easy-to-discover import path, so all the deferred functions should be at the top level of your module. I’d suggest putting them all in the same file (say, tasks.py)
- All method arguments are deepcopied at the time of the deferral. Thus,
- if you pass a very large datastructure to the deferral methods, it may have a performance impact. In addition, if you pass a mutable datastructur (dict, list, etc.) then subsequent modifications will have no effect.
- Watch out for double invocation of functions & methods. This is both
- a feature and a caveat. Once you pickle a function call, that value could be unpickled and run more than once. Watch out for anything that has unexpected side effects!