Skip to main content

faster zipfile imports for frozen python apps

Project description

This package aims to speed up imports from zipfiles for frozen python apps (and other scenarios where the zipfile is assumed not to change) by taking several shortcuts that aren’t available to the standard zipimport module.

It exports a single useful name, “zipimporter”, which is a drop-in replacement for the standard zipimporter class. To replace the builtin zipimport mechanism with zipimportx, do the following:

import zipimportx
zipimportx.zipimporter.install()

With no additional work you may already find a small speedup when importing from a zipfile. Since zipimportx assumes that the zipfile will not change or go missing, it does fewer stat() calls and integrity checks than the standard zipimport implementation.

To further speed up the loading of a zipfile, you can pre-compute the zipimport “directory information” dictionary and store it in a separate index file. This will reduce the time spent parsing information out of the zipfile. Create an index file like this:

from zipimportx import zipimporter
zipimporter("mylib.zip").write_index()

This will create the file “mylib.zip.idx” containing the pre-parsed zipfile directory information. Specifically, it will contain a marshalled dictionary object with the same structure as those in zipimport._zip_directory_cache.

In my tests, use of these indexes speeds up the initial loading of a zipfile by about a factor of 3 on Linux, and a factor of 5 on Windows.

To further speed up the loading of a collection of modules, you can “preload” the actual module data by including it directly in the index. This allows the data for several modules to be loaded in a single sequential read rather than requiring a separate read for each module. Preload module data like this:

from zipimportx import zipimporter
zipimporter("mylib.zip").write_index(preload=["mymod*","mypkg*"])

Each entry in the “preload” list is a filename pattern. Files from the zipfile that match any of these patterns will be preloaded when the zipfile is first accessed for import. You may want to remove them from the actual zipfile in order to save space.

Finally, it’s possible to convert a zipfile into inline python code and include that code directly in your frozen application. This can simulate the effect of having that zipfile on sys.path, while avoiding any fie IO during the import process. To get the necessary sourcecode, do the following:

from zipimportx import zipimporter
code = zipimporter("mylib.zip").get_inline_code()

Finally, it’s worth re-iterating the big assumption made by this module: the zipfile must never change or go missing. If the data in the index does not reflect the actual contents of the zipfile, imports will break in unspecified and probably disasterous ways.

Note also that this package uses nothing but builtin modules. To bootstrap zipfile imports for a frozen application, you can inline this module’s code directly into your application’s startup script. Simply do something like this in your build process:

import zipimportx
import inspect

SCRIPT = '''
%s
zipimporter.install()
import myapp
myapp.main()
''' % (inspect.getsource(zipimportx),)

freeze_this_script_somehow(SCRIPT)
zipimportx.zipimporter("path/to/frozen/library.zip").write_index()

Project details


Release history Release notifications

This version
History Node

0.3.2

History Node

0.3.1

History Node

0.3.0

History Node

0.2.0

History Node

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
zipimportx-0.3.2.tar.gz (10.4 kB) Copy SHA256 hash SHA256 Source None Jan 1, 2011

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page