lxmlrpc - XMLRPClib patch
This module monkeypatch python’s xmlrpclib to use lxml based parser to reduce memory consumption on big xmlrpc requests / responses (100+ Mb)
NOTE
This module is useful only for *python2.7*.
Use it only if you suffer from high memory consumption of xmlrpclib
Install
This module is hosted on PyPI so it could bu easily installed via pip:
pip install lxmlrpc_monkeyUsage
To use this module just do following
import xmlrpclib
from lxmlrpc_monkey import patch_xmlrpclib
# This line will monkey-patch xmlrpclib to use lxml for parser
patch_xmlrpclib()Benchmarks (how to run)
To run benchmarks:
install memory_profiler
prepare data with python generate_data.py --path <demo data path> --size 50000000
run benchmarks with python benchmark.py --path <demo data path>
Benchmark results (50 Mb (real 65 Mb) data file)
look at `p.feed(data)` call in `loads` function of xmlrpclib
*Running unpatched loads*
Filename: /usr/lib/python2.7/xmlrpclib.py
Line # |
Mem usage |
Increment |
Line Contents |
|---|---|---|---|
1134 |
104.7 MiB |
0.0 MiB |
def loads(data, use_datetime=0): |
1135 |
“””data -> unmarshalled data, method name |
||
1136 |
|||
1137 |
Convert an XML-RPC packet to unmarshalled data plus a method |
||
1138 |
name (None if not present). |
||
1139 |
|||
1140 |
If the XML-RPC packet represents a fault condition, this function |
||
1141 |
raises a Fault exception. |
||
1142 |
“”” |
||
1143 |
104.7 MiB |
0.0 MiB |
p, u = getparser(use_datetime=use_datetime) |
>1144 |
622.4 MiB |
517.7 MiB |
p.feed(data) |
1145 |
558.0 MiB |
-64.4 MiB |
p.close() |
1146 |
558.0 MiB |
0.0 MiB |
return u.close(), u.getmethodname() |
*Running patched loads*
Filename: /usr/lib/python2.7/xmlrpclib.py
Line # |
Mem usage |
Increment |
Line Contents |
|---|---|---|---|
1134 |
106.9 MiB |
0.0 MiB |
def loads(data, use_datetime=0): |
1135 |
“””data -> unmarshalled data, method name |
||
1136 |
|||
1137 |
Convert an XML-RPC packet to unmarshalled data plus a method |
||
1138 |
name (None if not present). |
||
1139 |
|||
1140 |
If the XML-RPC packet represents a fault condition, this function |
||
1141 |
raises a Fault exception. |
||
1142 |
“”” |
||
1143 |
106.9 MiB |
0.0 MiB |
p, u = getparser(use_datetime=use_datetime) |
>1144 |
235.9 MiB |
129.0 MiB |
p.feed(data) |
1145 |
171.5 MiB |
-64.4 MiB |
p.close() |
1146 |
171.5 MiB |
0.0 MiB |
return u.close(), u.getmethodname() |
Filename: bechmark.py
Line # |
Mem usage |
Increment |
Line Contents |
|---|---|---|---|
13 |
104.7 MiB |
0.0 MiB |
@profile |
14 |
def bench_load(xmldata): |
||
15 |
104.7 MiB |
0.0 MiB |
print (“Running unpatched loads”) |
16 |
106.9 MiB |
2.2 MiB |
loads(xmldata) |
17 |
|||
18 |
106.9 MiB |
0.0 MiB |
lxmlrpc.patch_xmlrpclib() |
19 |
|||
20 |
106.9 MiB |
0.0 MiB |
print (“Running patched loads”) |
21 |
107.1 MiB |
0.2 MiB |
loads(xmldata) |
Benchmark results (100 Mb (real 129 Mb) data file)
*Running unpatched loads*
—
Filename: /usr/lib/python2.7/xmlrpclib.py
Line # |
Mem usage |
Increment |
Line Contents |
|---|---|---|---|
1134 |
169.2 MiB |
0.0 MiB |
def loads(data, use_datetime=0): |
1135 |
“””data -> unmarshalled data, method name |
||
1136 |
|||
1137 |
Convert an XML-RPC packet to unmarshalled data plus a method |
||
1138 |
name (None if not present). |
||
1139 |
|||
1140 |
If the XML-RPC packet represents a fault condition, this function |
||
1141 |
raises a Fault exception. |
||
1142 |
“”” |
||
1143 |
169.2 MiB |
0.0 MiB |
p, u = getparser(use_datetime=use_datetime) |
>1144 |
1203.0 MiB |
1033.8 MiB |
p.feed(data) |
1145 |
1074.2 MiB |
-128.8 MiB |
p.close() |
1146 |
1074.2 MiB |
0.0 MiB |
return u.close(), u.getmethodname() |
*Running patched loads*
—
Filename: /usr/lib/python2.7/xmlrpclib.py
Line # |
Mem usage |
Increment |
Line Contents |
|---|---|---|---|
1134 |
171.6 MiB |
0.0 MiB |
def loads(data, use_datetime=0): |
1135 |
“””data -> unmarshalled data, method name |
||
1136 |
|||
1137 |
Convert an XML-RPC packet to unmarshalled data plus a method |
||
1138 |
name (None if not present). |
||
1139 |
|||
1140 |
If the XML-RPC packet represents a fault condition, this function |
||
1141 |
raises a Fault exception. |
||
1142 |
“”” |
||
1143 |
171.6 MiB |
0.0 MiB |
p, u = getparser(use_datetime=use_datetime) |
>1144 |
429.4 MiB |
257.8 MiB |
p.feed(data) |
1145 |
300.6 MiB |
-128.8 MiB |
p.close() |
1146 |
300.6 MiB |
0.0 MiB |
return u.close(), u.getmethodname() |
Filename: bechmark.py
Line # |
Mem usage |
Increment |
Line Contents |
|---|---|---|---|
13 |
169.2 MiB |
0.0 MiB |
@profile |
14 |
def bench_load(xmldata): |
||
15 |
169.2 MiB |
0.0 MiB |
print (“Running unpatched loads”) |
16 |
171.6 MiB |
2.4 MiB |
loads(xmldata) |
17 |
|||
18 |
171.6 MiB |
0.0 MiB |
lxmlrpc.patch_xmlrpclib() |
19 |
|||
20 |
171.6 MiB |
0.0 MiB |
print (“Running patched loads”) |
21 |
171.8 MiB |
0.2 MiB |
loads(xmldata) |
1 maintainer
Metadata
1 maintainer
Release files for lxmlrpc_monkey 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| lxmlrpc_monkey-1.0.0.tar.gz | 4.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| lxmlrpc_monkey-1.0.0-py2-none-any.whl | Python 2 | none | any | Details |
Total release size: 10.2 kB
Release files / lxmlrpc_monkey-1.0.0.tar.gz
| Download URL | lxmlrpc_monkey-1.0.0.tar.gz |
|---|---|
| Size | 4.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
999373a2726d4d1d359728e81b7cc1871af48cf28ff18be45b48d852d1e3a682
|
|
BLAKE2b-256 checksum How to use checksums |
cb16177b997fa7506bed52a75eb8311d510dac48a21d8b3260ea6441de969869
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / lxmlrpc_monkey-1.0.0-py2-none-any.whl
| Download URL | lxmlrpc_monkey-1.0.0-py2-none-any.whl |
|---|---|
| Size | 5.9 kB |
| Tags | Python 2 |
|
SHA-256 checksum How to use checksums |
80f11dfdcde8c9584160d044d4b373ca808c975350aeb6d0742b5e712784f34c
|
|
BLAKE2b-256 checksum How to use checksums |
6f10d0c86cff6edb38e673b2c125fcab0a2e0b3447fd944e885f3849dc6ec5ad
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
