spillsort
Sort more items than fit in memory: sorted runs spill to disk and merge as they are read.
Install with pip or uv:
pip install spillsort
Or add the Rust crate:
cargo add spillsort
Introduction
This library sorts like sorted(items, key=...), but in a fixed amount of memory.
Each item is keyed and encoded into bytes once, as it is added.
When the memory fills, a background thread sorts it and writes it to a temporary file while you keep adding.
Sorting merges those files with what is still in memory, so items come out in key order, and items with equal keys come out in the order they were added.
The engine is Rust, and it is also a Rust crate that needs no Python.
Quick Start
Give a key function and sort items, once:
from spillsort import Sorter
reads = [("chr2", 500, "read1"), ("chr1", 900, "read2"), ("chr1", 100, "read3"), ("chr1", 900, "read4")]
with Sorter(lambda read: (read[0], read[1])) as sorter:
ordered = list(sorter.sort(reads))
assert ordered == [("chr1", 100, "read3"), ("chr1", 900, "read2"), ("chr1", 900, "read4"), ("chr2", 500, "read1")]
Add items one at a time as they arrive, bound the memory with max_bytes, sort on more threads with threads, and choose where to spill with tmp_dir:
import random
from tempfile import TemporaryDirectory
numbers = [random.random() for _ in range(100_000)]
with TemporaryDirectory() as tmp_dir, Sorter(max_bytes=2**20, threads=4, tmp_dir=tmp_dir) as sorter:
for number in numbers:
sorter.add(number)
ordered = list(sorter.sort())
assert ordered == sorted(numbers)
Items are pickled while they wait unless a codec turns them into bytes and back:
class Line:
def encode(self, item: str, /) -> bytes:
return item.encode()
def decode(self, data: bytes, /) -> str:
return data.decode()
with Sorter(lambda line: int(line.split("\t")[1]), codec=Line()) as sorter:
lines = list(sorter.sort(["chr1\t300\n", "chr1\t20\n", "chr1\t1000\n"]))
assert lines == ["chr1\t20\n", "chr1\t300\n", "chr1\t1000\n"]
If you already have the bytes, add them with their key, and no Python runs per item. Write the sorted bytes to any binary file in large chunks, merged on another thread while the file writes:
import io
with Sorter() as sorter:
sorter.add_encoded(("chr2", 5), b"chr2\t5\t9\n")
sorter.add_encoded(("chr1", 70), b"chr1\t70\t80\n")
sorter.add_encoded(("chr1", 8), b"chr1\t8\t9\n")
out = io.BytesIO()
sorter.write_to(out)
assert out.getvalue() == b"chr1\t8\t9\nchr1\t70\t80\nchr2\t5\t9\n"
From Rust, push keyed byte records and read them back merged:
use spillsort::{Config, Sorter, key};
fn main() -> Result<(), spillsort::Error> {
let mut sorter = Sorter::new(Config::default().max_bytes(64 << 20))?;
for (refname, start) in [("chr2", 5), ("chr1", 70), ("chr1", 8)] {
let mut sort_key = Vec::new();
key::push_str(&mut sort_key, refname);
key::push_int(&mut sort_key, start);
sorter.push(&sort_key, format!("{refname}\t{start}\n").as_bytes())?;
}
let mut out = Vec::new();
sorter.finish()?.write_values(&mut out)?;
assert_eq!(out, b"chr1\t8\nchr1\t70\nchr2\t5\n");
Ok(())
}
See the API documentation for the crate.
Keys
Keys compare exactly as Python compares them, even ints and floats of any size, and NaN sorts last:
with Sorter() as sorter:
ordered = list(sorter.sort([2**64, float("nan"), 1.5, True, -0.5, 3]))
assert ordered[:-1] == [-0.5, True, 1.5, 3, 2**64]
Development and Testing
See the contributing guide for more information.
1 maintainer
Metadata
1 maintainer
Release files for spillsort 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| spillsort-0.1.0.tar.gz | 47.8 kB | Details |
Built distributions (wheels)
Total release size: 6.1 MB
Release files / spillsort-0.1.0.tar.gz
| Download URL | spillsort-0.1.0.tar.gz |
|---|---|
| Size | 47.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4feebe1750158d006d3bfd3594e43e6dfdcbede4037b66e6a61ca4f4fb8c04c9
|
|
BLAKE2b-256 checksum How to use checksums |
cedaefdf2a3f7fb330559d751b8db09ba5eceb73ec886d3c25bbcd75cbf7554d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / spillsort-0.1.0-cp314-cp314t-win_amd64.whl
| Download URL | spillsort-0.1.0-cp314-cp314t-win_amd64.whl |
|---|---|
| Size | 277.8 kB |
| Tags | CPython 3.14 CPython 3.14 free-threading Windows x86-64 |
|
SHA-256 checksum How to use checksums |
a166d2db4db9528ec7be3022b2ea5172136767d40257203864ba89bec9443ed0
|
|
BLAKE2b-256 checksum How to use checksums |
39b33cda44922687da88d9edc5f08a170316cc96d54e4db7937f87d8bc001e1a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / spillsort-0.1.0-cp314-cp314t-musllinux_1_2_x86_64.whl
| Download URL | spillsort-0.1.0-cp314-cp314t-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 618.7 kB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ x86-64 |
|
SHA-256 checksum How to use checksums |
96163d020a840612fbc19b18968850b7c6a048931246f75c8712647191c50b02
|
|
BLAKE2b-256 checksum How to use checksums |
cf3aa4680c55e9c5e886b8472cd9f60e7f41b07809b60f059b9acca93c8639eb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / spillsort-0.1.0-cp314-cp314t-musllinux_1_2_aarch64.whl
| Download URL | spillsort-0.1.0-cp314-cp314t-musllinux_1_2_aarch64.whl |
|---|---|
| Size | 582.2 kB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ ARM64 |
|
SHA-256 checksum How to use checksums |
25613a596567e02c58dbeafa090ccc35b7250f59ca56c0e789e66070da917ab7
|
|
BLAKE2b-256 checksum How to use checksums |
3e406f9c2f884aebc2e95dffddf5eb34c5647ed058bab3a8a400e7b923d47aed
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / spillsort-0.1.0-cp314-cp314t-manylinux_2_28_aarch64.whl
| Download URL | spillsort-0.1.0-cp314-cp314t-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 403.0 kB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux glibc 2.28+ ARM64 |
|
SHA-256 checksum How to use checksums |
fbb6e6714e1b181c8334c6d86a6496f4786cc2f245b921699b6651fdd60fe8c2
|
|
BLAKE2b-256 checksum How to use checksums |
768563796b6e7737532f569411ae736fe5456412f2d00db13b91957f3dcd6caf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / spillsort-0.1.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | spillsort-0.1.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 406.0 kB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
57fcc231cc46bf42ac36b1c8f96e8be3e5111ea5f3fda27bc653c2b75b9d2c2f
|
|
BLAKE2b-256 checksum How to use checksums |
17c8e595ac564e6a956ad236e3432466a16cbc58fb8128870ac583be81b434ba
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / spillsort-0.1.0-cp314-cp314t-macosx_11_0_arm64.whl
| Download URL | spillsort-0.1.0-cp314-cp314t-macosx_11_0_arm64.whl |
|---|---|
| Size | 359.0 kB |
| Tags | CPython 3.14 CPython 3.14 free-threading macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
4801018f27c5b64dedd6cc167da0f3c2b29da832f1b3cd3abaf3e90db4973d58
|
|
BLAKE2b-256 checksum How to use checksums |
8f7693b8f076a6d18a025e8f9abdac9f2e5328a2c3b3692846989cdcb81d375c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / spillsort-0.1.0-cp314-cp314t-macosx_10_12_x86_64.whl
| Download URL | spillsort-0.1.0-cp314-cp314t-macosx_10_12_x86_64.whl |
|---|---|
| Size | 371.1 kB |
| Tags | CPython 3.14 CPython 3.14 free-threading macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
556fccecbbd16c3476837c31f23edd8fe36953d30082db1141cbe8033f6b4886
|
|
BLAKE2b-256 checksum How to use checksums |
c74aae8c5479f629a69381dda5208c637ddf677fbdbd29cdeb85f47dfb5ac9b2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / spillsort-0.1.0-cp311-abi3-win_amd64.whl
| Download URL | spillsort-0.1.0-cp311-abi3-win_amd64.whl |
|---|---|
| Size | 281.1 kB |
| Tags | CPython 3.11 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
a8872dd45e46534571dd95324f3e12ba6a05c7f1c97f21089f55073197d1be75
|
|
BLAKE2b-256 checksum How to use checksums |
ad10bf6a2a83477355a9b4404b4cdbbac5d549c5d94944aa0438d36556630457
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / spillsort-0.1.0-cp311-abi3-musllinux_1_2_x86_64.whl
| Download URL | spillsort-0.1.0-cp311-abi3-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 620.6 kB |
| Tags | CPython 3.11 Linux musl 1.2+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
72dd092ca59b522faf6256b9ee425979c3cd24102cc247a87230aad1bb97290c
|
|
BLAKE2b-256 checksum How to use checksums |
f4c6567ca83f84825bdad5998f05a366bde88f23e7a40d44c8a1e0f38611a30b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / spillsort-0.1.0-cp311-abi3-musllinux_1_2_aarch64.whl
| Download URL | spillsort-0.1.0-cp311-abi3-musllinux_1_2_aarch64.whl |
|---|---|
| Size | 583.0 kB |
| Tags | CPython 3.11 Linux musl 1.2+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
ca9f837a2fc0a4fa9caf4ad287fb8c72a8810e770d5b6e6105e705b0de4abbc1
|
|
BLAKE2b-256 checksum How to use checksums |
44ba4e76da7eecf450b97edd67b66ba53396e9108a12f26deecdac425666b2e1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / spillsort-0.1.0-cp311-abi3-manylinux_2_28_aarch64.whl
| Download URL | spillsort-0.1.0-cp311-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 404.5 kB |
| Tags | CPython 3.11 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
5d7b495e527333e8f5a565236fc482f2661ebd6417e3eb0099e1530bb73034c7
|
|
BLAKE2b-256 checksum How to use checksums |
c2767c075dacefd82de466d9e5f713266848911693b6f78da9a604c10ac841c5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / spillsort-0.1.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | spillsort-0.1.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 408.0 kB |
| Tags | CPython 3.11 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
d3c5d7586f7f9674d53b1e91cd148a643543cd1a7d9c6f5fc65c14fa372f0529
|
|
BLAKE2b-256 checksum How to use checksums |
45fe4ea8aaf49e686506eb40829e3c5579e25955e5fa596b6e8e9163acfa6912
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / spillsort-0.1.0-cp311-abi3-macosx_11_0_arm64.whl
| Download URL | spillsort-0.1.0-cp311-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 362.2 kB |
| Tags | CPython 3.11 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
0bfe67514e18bafff861f4a20e0fb372839233f94f78053e711cc24f1657ecc6
|
|
BLAKE2b-256 checksum How to use checksums |
6a1054f63cb6f60c3d5a0319f8bfe950b760fada9e199845a4c9e5fe90d7bbe0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / spillsort-0.1.0-cp311-abi3-macosx_10_12_x86_64.whl
| Download URL | spillsort-0.1.0-cp311-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 370.4 kB |
| Tags | CPython 3.11 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
b31b0d6b2010420d8f471eda3c6ca288c7f315764198a49a773e4bd2cce1f57f
|
|
BLAKE2b-256 checksum How to use checksums |
97d6a371f70eb0b481f791e7524c5995095ec04abb64577f1075bf85a046e90b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency log
