Skip to main content

A pure Python implementation of a sliding window memory map manager

Project description

Motivation

When reading from many possibly large files in a fashion similar to random access, it is usually the fastest and most efficient to use memory maps.

Although memory maps have many advantages, they represent a very limited system resource as every map uses one file descriptor, whose amount is limited per process. On 32 bit systems, the amount of memory you can have mapped at a time is naturally limited to theoretical 4GB of memory, which may not be enough for some applications.

Limitations

  • System resources (file-handles) are likely to be leaked! This is due to the library authors reliance on a deterministic __del__() destructor.
  • The memory access is read-only by design.

Overview

Python package

Smmap wraps an interface around mmap and tracks the mapped files as well as the amount of clients who use it. If the system runs out of resources, or if a memory limit is reached, it will automatically unload unused maps to allow continued operation.

To allow processing large files even on 32 bit systems, it allows only portions of the file to be mapped. Once the user reads beyond the mapped region, smmap will automatically map the next required region, unloading unused regions using a LRU algorithm.

The interface also works around the missing offset parameter in python implementations up to python 2.5.

Although the library can be used most efficiently with its native interface, a Buffer implementation is provided to hide these details behind a simple string-like interface.

For performance critical 64 bit applications, a simplified version of memory mapping is provided which always maps the whole file, but still provides the benefit of unloading unused mappings on demand.

Prerequisites

  • Python 2.7 or 3.4+
  • OSX, Windows or Linux

The package was tested on all of the previously mentioned configurations.

Installing smmap

Documentation Status

Its easiest to install smmap using the pip program:

$ pip install smmap

As the command will install smmap in your respective python distribution, you will most likely need root permissions to authorize the required changes.

If you have downloaded the source archive, the package can be installed by running the setup.py script:

$ python setup.py install

It is advised to have a look at the Usage Guide for a brief introduction on the different database implementations.

Homepage and Links

The project is home on github at https://github.com/gitpython-developers/smmap .

The latest source can be cloned from github as well:

  • git://github.com/gitpython-developers/smmap.git

For support, please use the git-python mailing list:

Issues can be filed on github:

A link to the pypi page related to this repository:

License Information

smmap is licensed under the New BSD License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

smmap-3.0.5.tar.gz (23.2 kB view details)

Uploaded Source

Built Distribution

smmap-3.0.5-py2.py3-none-any.whl (25.1 kB view details)

Uploaded Python 2Python 3

File details

Details for the file smmap-3.0.5.tar.gz.

File metadata

  • Download URL: smmap-3.0.5.tar.gz
  • Upload date:
  • Size: 23.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.55.1 CPython/3.8.2

File hashes

Hashes for smmap-3.0.5.tar.gz
Algorithm Hash digest
SHA256 84c2751ef3072d4f6b2785ec7ee40244c6f45eb934d9e543e2c51f1bd3d54c50
MD5 84b4f0ed6522cc1d12a9d8413e8f7bd7
BLAKE2b-256 2b6fd48bbed5aa971943759f4ede3f12dca40aa7faa44f22bad483de86780508

See more details on using hashes here.

File details

Details for the file smmap-3.0.5-py2.py3-none-any.whl.

File metadata

  • Download URL: smmap-3.0.5-py2.py3-none-any.whl
  • Upload date:
  • Size: 25.1 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.55.1 CPython/3.8.2

File hashes

Hashes for smmap-3.0.5-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 7bfcf367828031dc893530a29cb35eb8c8f2d7c8f2d0989354d75d24c8573714
MD5 ad0d21e1306193b8cb1f735d64a6375b
BLAKE2b-256 d51e6130925131f639b2acde0f7f18b73e33ce082ff2d90783c436b52040af5a

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page