Skip to main content

External sort algorithm implementation.

Project description

Build status License Supported Python versions

External sort algorithm implementation. External sorting is a class of sorting algorithms that can handle massive amounts of data. External sorting is required when the data being sorted do not fit into the main memory (RAM) of a computer and instead must be resided in slower external memory, usually a hard disk drive. Sorting is achieved in two passes. During the first pass it sorts chunks of data that each fit in RAM, during the second pass it merges the sorted chunks together. For more information see https://en.wikipedia.org/wiki/External_sorting.

Compatibility

ext-sort requires 3.6+.

Installation

You can install ext-sort with pip:

$ pip install ext-sort

Quick start

Quick start.

import csv
import io
import logging

import ext_sort as es


class CSVSerializer(es.Serializer):

    def __init__(self, writer):
        super().__init__(csv.writer(io.TextIOWrapper(writer, write_through=True)))

    def write(self, item):
        return self._writer.writerow(item)


class CSVDeserializer(es.Deserializer):

    def __init__(self, reader):
        super().__init__(csv.reader(io.TextIOWrapper(reader)))

    def read(self):
        return next(self._reader)


logging.basicConfig(
    level=logging.DEBUG,
    format='[%(levelname)-8s] %(asctime)-15s (%(name)s): %(message)s',
)

with open('/home/user/data.csv', 'rb') as unsorted_file, open('/home/user/data.sorted.csv', 'wb') as sorted_file:
    # save the csv header
    sorted_file.write(unsorted_file.readline())

    es.sort(
        reader=unsorted_file,
        writer=sorted_file,
        chunk_size=10_000_000,
        Serializer=CSVSerializer,
        Deserializer=CSVDeserializer,
        workers_cnt=4,
    )

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for ext-sort, version 0.2.0
Filename, size File type Python version Upload date Hashes
Filename, size ext-sort-0.2.0.tar.gz (5.3 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page