A python package to offload a function call to an http server automatically using a decorator.
Project description
auto-function-serving
A python package to offload a function call to an http server running on localhost automatically using a decorator. Should work with multiprocessing, pickle, flask and others.
Why
Imagine a case of a multi threaded or multiprocessing application where 1 or few functions are heavily resource (cpu or memory) intensive, but the other functions can run in parallel.
Example - an api call followed by tokenization and classification using a large DL model followed by further API calls.
In such a case, it would make sense to create a server (using torchserve or tfserving) to serve requests.
ServerHandler does this automatically during runtime
Reccomended Usage
from somemodule import someheavyfunction
from auto_function_serving.ServerHandler import ServerHandler
someheavyfunction = ServerHandler.decorator(someheavyfunction)
Any calls to this new someheavyfunction will make requests to 1 instance of a process running a http.server.HTTPServer which runs the function within it. Even calls made from different processes, threads, multiprocessing or servers like flask.
It can also be used like a traditional decorator for functions with no dependencies specified outside the function.
from auto_function_serving.ServerHandler import ServerHandler
@ServerHandler.decorator
def someheavyfunctionwithnodependency(args,**kwargs):
for i in range(big_number)
someexpensivecomputation
Installation
Use the package manager pip to install auto_function_serving
pip install auto_function_serving
More Usage
In general :
some independent code with a callable
can be replaced with
from auto_function_serving.ServerHandler import ServerHandler
callable_name = ServerHandler("""
some independent code with a callable
""", "callable_name")
Example :
import module1
import module2
def functionname(someinput):
a = module1.function1(someinput)
return module2.function2(a)
can be replaced with
from auto_function_serving.ServerHandler import ServerHandler
functionname = ServerHandler("""
import module1
import module2
def functionname(someinput):
a = module1.function1(someinput)
return module2.function2(a)
""", "functionname")
How does this work?
Code for the server is stored in ServerHandler.base_code, inspect.cleandoc and some string formatting is used to fill in the blanks.
A port from 50000 to 60000 is chosen by hashing the input text to make it independent of the source of a function. Collisions are possible, but unlikely. The port can be specified if needed.
from somemodule import someheavyfunction
from auto_function_serving.ServerHandler import ServerHandler
someheavyfunction = ServerHandler.decorator(someheavyfunction, port = 4321)
ServerHandler.ip_address is set as "127.0.0.1".
The server process is started with Popen (or multiprocessing if specified), and the first thing it does is import socket and bind the port - if it's not available the code stops after an exception. Therefore only 1 instance of the server runs at a time on a machine.
We know the function is ready after we can recieve a valid get request from the server.
Inputs and outputs are sent as bytes, converted to and from objects using pickle.
Performance (On my machine)
using urllib.request seemed to give the best performance.
~2ms overhead for small input and output (few bytes).
~10ms overhead for 0.5 mb input and output (1mb total transfer).
~60ms overhead for 5 mb input and output (10 mb total transfer).
Other things to look into
Libraries : Celery, Tfserving, Torchserve, Flask
Sending globals and locals to exec
ast trees
Contributing
Pull requests are welcome.
License
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for auto_function_serving-0.0.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 242acd9105d042393e0d39293ba4589d1b1539b1b5f7c8c2775fe4d73bb250a5 |
|
MD5 | 032ffd999b692a7b9df1b171e36e4939 |
|
BLAKE2b-256 | 4ff03d87103157851d14cf24676ca3414a3e94287a2c6b6afd75d2fbc762ef21 |
Hashes for auto_function_serving-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f3ca8ba2debcf1550548e9c5e0919116fa48d9e5436c8c92bff3aacfb386cd6b |
|
MD5 | ebc35685d256842bd6c5bb336b4cbfa6 |
|
BLAKE2b-256 | 90384b854776335488008c296a960d7f59befa1da2fcbd4cfb2d0156b6d0008e |