Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

xmlrpclib patch to reduce its memory consumption

Project description

lxmlrpc - XMLRPClib patch

This module monkeypatch python’s xmlrpclib to use lxml based parser to reduce memory consumption on big xmlrpc requests / responses (100+ Mb)

NOTE

This module is useful only for *python2.7*.

Use it only if you suffer from high memory consumption of xmlrpclib

Install

This module is hosted on PyPI so it could bu easily installed via pip:

pip install lxmlrpc_monkey

Usage

To use this module just do following

import xmlrpclib
from lxmlrpc_monkey import patch_xmlrpclib
# This line will monkey-patch xmlrpclib to use lxml for parser
patch_xmlrpclib()

Benchmarks (how to run)

To run benchmarks:

  1. install memory_profiler
  2. prepare data with python generate_data.py --path <demo data path> --size 50000000
  3. run benchmarks with python benchmark.py --path <demo data path>

Benchmark results (50 Mb (real 65 Mb) data file)

look at `p.feed(data)` call in `loads` function of xmlrpclib

*Running unpatched loads*

Filename: /usr/lib/python2.7/xmlrpclib.py

Line # Mem usage Increment Line Contents
1134 104.7 MiB 0.0 MiB def loads(data, use_datetime=0):
1135     “”“data -> unmarshalled data, method name
1136      
1137     Convert an XML-RPC packet to unmarshalled data plus a method
1138     name (None if not present).
1139      
1140     If the XML-RPC packet represents a fault condition, this function
1141     raises a Fault exception.
1142     “”“
1143 104.7 MiB 0.0 MiB p, u = getparser(use_datetime=use_datetime)
>1144 622.4 MiB 517.7 MiB p.feed(data)
1145 558.0 MiB -64.4 MiB p.close()
1146 558.0 MiB 0.0 MiB return u.close(), u.getmethodname()

*Running patched loads*

Filename: /usr/lib/python2.7/xmlrpclib.py

Line # Mem usage Increment Line Contents
1134 106.9 MiB 0.0 MiB def loads(data, use_datetime=0):
1135     “”“data -> unmarshalled data, method name
1136      
1137     Convert an XML-RPC packet to unmarshalled data plus a method
1138     name (None if not present).
1139      
1140     If the XML-RPC packet represents a fault condition, this function
1141     raises a Fault exception.
1142     “”“
1143 106.9 MiB 0.0 MiB p, u = getparser(use_datetime=use_datetime)
>1144 235.9 MiB 129.0 MiB p.feed(data)
1145 171.5 MiB -64.4 MiB p.close()
1146 171.5 MiB 0.0 MiB return u.close(), u.getmethodname()

Filename: bechmark.py

Line # Mem usage Increment Line Contents
13 104.7 MiB 0.0 MiB @profile
14     def bench_load(xmldata):
15 104.7 MiB 0.0 MiB print (“Running unpatched loads”)
16 106.9 MiB 2.2 MiB loads(xmldata)
17      
18 106.9 MiB 0.0 MiB lxmlrpc.patch_xmlrpclib()
19      
20 106.9 MiB 0.0 MiB print (“Running patched loads”)
21 107.1 MiB 0.2 MiB loads(xmldata)

Benchmark results (100 Mb (real 129 Mb) data file)

*Running unpatched loads*

Filename: /usr/lib/python2.7/xmlrpclib.py

Line # Mem usage Increment Line Contents
1134 169.2 MiB 0.0 MiB def loads(data, use_datetime=0):
1135     “”“data -> unmarshalled data, method name
1136      
1137     Convert an XML-RPC packet to unmarshalled data plus a method
1138     name (None if not present).
1139      
1140     If the XML-RPC packet represents a fault condition, this function
1141     raises a Fault exception.
1142     “”“
1143 169.2 MiB 0.0 MiB p, u = getparser(use_datetime=use_datetime)
>1144 1203.0 MiB 1033.8 MiB p.feed(data)
1145 1074.2 MiB -128.8 MiB p.close()
1146 1074.2 MiB 0.0 MiB return u.close(), u.getmethodname()

*Running patched loads*

Filename: /usr/lib/python2.7/xmlrpclib.py

Line # Mem usage Increment Line Contents
1134 171.6 MiB 0.0 MiB def loads(data, use_datetime=0):
1135     “”“data -> unmarshalled data, method name
1136      
1137     Convert an XML-RPC packet to unmarshalled data plus a method
1138     name (None if not present).
1139      
1140     If the XML-RPC packet represents a fault condition, this function
1141     raises a Fault exception.
1142     “”“
1143 171.6 MiB 0.0 MiB p, u = getparser(use_datetime=use_datetime)
>1144 429.4 MiB 257.8 MiB p.feed(data)
1145 300.6 MiB -128.8 MiB p.close()
1146 300.6 MiB 0.0 MiB return u.close(), u.getmethodname()

Filename: bechmark.py

Line # Mem usage Increment Line Contents
13 169.2 MiB 0.0 MiB @profile
14     def bench_load(xmldata):
15 169.2 MiB 0.0 MiB print (“Running unpatched loads”)
16 171.6 MiB 2.4 MiB loads(xmldata)
17      
18 171.6 MiB 0.0 MiB lxmlrpc.patch_xmlrpclib()
19      
20 171.6 MiB 0.0 MiB print (“Running patched loads”)
21 171.8 MiB 0.2 MiB loads(xmldata)

Project details


Release history Release notifications

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for lxmlrpc_monkey, version 1.0.0
Filename, size File type Python version Upload date Hashes
Filename, size lxmlrpc_monkey-1.0.0-py2-none-any.whl (5.9 kB) File type Wheel Python version 2.7 Upload date Hashes View hashes
Filename, size lxmlrpc_monkey-1.0.0.tar.gz (4.2 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page