Skip to main content

Like sorted but using external sorting so that large data sets can be sorted.

Project description

https://travis-ci.org/moagstar/xsorted.svg?branch=master https://coveralls.io/repos/github/moagstar/xsorted/badge.svg?branch=master

xsorted

Like sorted but using external sorting so that large data sets can be sorted, for example:

>>> from random import random
>>> from six.moves import xrange
>>> from xsorted import xsorted
>>> nums = (random() for _ in xrange(pow(10, 7)))
>>> for x in xsorted(nums): pass

The only restriction is that the items must be pickleable (or you can provide your own serializer for externalizing partitions of items).

Motivation

It is sometimes necessary to sort a dataset without having to load the entire set into memory. For example, if you want to group a very large csv file by one of it’s columns. There are several ways in which this can be achieved, a common solution is to use the unix command sort. However unix sort does not offer the flexibility of the python csv module. xsorted attempts to generalize external sorting of any python iterable in a similar way in which sorted generalises the sorting of any iterable.

Installation

$ pip install xsorted

Usage

Just like sorted

>>> from xsorted import xsorted
>>> ''.join(xsorted('qwertyuiopasdfghjklzxcvbnm'))
'abcdefghijklmnopqrstuvwxyz'

With reverse

>>> ''.join(xsorted('qwertyuiopasdfghjklzxcvbnm', reverse=True))
'zyxwvutsrqponmlkjihgfedcba'

And a custom key

>>> list(xsorted(('qwerty', 'uiop', 'asdfg', 'hjkl', 'zxcv', 'bnm'), key=lambda x: x[1]))
['uiop', 'hjkl', 'bnm', 'asdfg', 'qwerty', 'zxcv']

The implementation details of xsorted can be customized using the factory xsorter (in order to provide the same interface as sorted the partition_size is treated as an implementation detail):

>>> from xsorted import xsorter
>>> xsorted_custom = xsorter(partition_size=4)
>>> ''.join(xsorted_custom('qwertyuiopasdfghjklzxcvbnm'))
'abcdefghijklmnopqrstuvwxyz'

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for xsorted, version 0.0.4
Filename, size File type Python version Upload date Hashes
Filename, size xsorted-0.0.4.tar.gz (119.9 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page