Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Package provides classes: FileList, FileDeque that behaves like bulltins but keeps items at disk.

Project Description

Module contains class with extended python list that stores items at disk. By default items before save are pickled and compressed. Use that list as usual list!

In addition, there is implemented extended python deque with disk storage and same behaviour as collections.deque.

Intend of package was to create generic iterables that stores really big collection of items that does not fit in memory and to avoid usage of external cache and local database storages.

>>> from diskcollections.iterables import FileList
>>> flist = FileList()
>>> flist.extend([1, 2, 3])
>>> flist.append(4)
>>> flist
[1, 2, 3, 4]
>>> flist[2]
3
>>> flist2 = flist[:]  # copy makes new FileList
>>> my_list = list(flist)  # now its simple list
>>> from diskcollections.iterables import FileQueue
>>> fdeque = FileQueue()
>>> fdeque.extend([1, 2, 3])
>>> fdeque.append(4)
>>> fdeque
FileDeque([1, 2, 3, 4])
>>> fdeque.pop()
4
>>> fdeque.appendleft(0)
>>> fdeque.popleft()
0

There are available more ways to serialize items.

>>> from diskcollections.iterables import FileList, FileDeque
>>> from diskcollections.handlers import (
    PickleHandler,  # pickle items
    PickleZLibHandler,  # pickle + compress items
    JsonHandler, # convert to json items
    JsonZLibHandler  # convert to json + compress items
)
>>> from functools import partial
>>> JsonFileList = partial(FileList, handler_class=JsonHandler)
>>> flist = JsonFileList()
>>> flist.append({'a': 1, 'b': 2, 'c': 3})
>>> flist[0]
{u'a': 1, u'b': 2, u'c': 3}

Installation

To install package type

$ pip install python-disk-collections

How it works

In order to implement your serializer create class with methods: dumps and loads or import interface.

>>> from diskcollections.interfaces import IHandler

class IHandler:

@staticmethod
def dumps(obj):
    """Converts object to string.

    :param obj: any python object
    :return: dumped string
    """
    raise NotImplementedError

@staticmethod
def loads(obj):
    """Restored dumped string into python object.

    :param obj: Object stored as string
    :return: python object restored from dump
    """
    raise NotImplementedError

All handlers from example above implements interface IHandler.

Under the hood, FileList for storage items uses tempfile.mktemp (in python2) or tempfile.TemporaryDirectory (in python3). It means, that every list has own unique directory, placed likely in /tmp/. When list is removed by garbage collector, all items that was stored are lost.

For FileDeque stores items in the same way as FileList. Difference between them is that FileList implements: insert, slicing, indexing. Because of overlaping indexes of FileList while using insert, FileList stores own alphabet to index new and inserted items. FileDeque doesn’t have indexing so it doesn’t take any memory.

Contribute

  1. Fork repository on GitHub to start making your changes to the master branch (or branch off of it).
  2. Write tests that prove that bug or future works as expected
  3. Check your code and tests with tox
  4. Send a pull request!

License

Python-Disk-Collection is under MIT license, see LICENSE for more details.

Release History

This version
History Node

0.0.2

History Node

0.0.1

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Hash SHA256 Hash Help Version File Type Upload Date
python_disk_collections-0.0.2-py2.py3-none-any.whl
(14.9 kB) Copy SHA256 Hash SHA256
py2.py3 Wheel Nov 18, 2017
python-disk-collections-0.0.2.tar.gz
(9.8 kB) Copy SHA256 Hash SHA256
Source Nov 18, 2017

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting