paged-list 0.1.2

A disk-backed list implementation for handling large datasets efficiently

Paged List

A Python package that provides a disk-backed list implementation for handling large datasets efficiently. When your data gets too large for memory, paged-list automatically chunks it into pickle files on disk, only loading relevant chunks when needed.

Links

• PyPI Package: https://pypi.org/project/paged-list/
• Documentation: https://paged-list.readthedocs.io/
• Source Code: https://github.com/christensendaniel/paged-list
• Issues: https://github.com/christensendaniel/paged-list/issues
• Changelog: CHANGELOG.md

Features

• Memory Efficient: Only keeps a small portion of data in memory
• Automatic Chunking: Transparently splits large datasets into manageable chunks
• List-like Interface: Supports indexing, slicing, and iteration like regular Python lists
• Parallel Processing: Built-in map and serialization functions with multi-threading support
• Type Safety: Designed for dictionaries with comprehensive type hints
• Context Manager: Automatic cleanup of temporary files

Requirements

• Python 3.9 or higher
• No external dependencies for core functionality

Installation

Install from PyPI:

pip install paged-list

PyPI Package: https://pypi.org/project/paged-list/

Note: Python 3.9+ is required. If you're using an older Python version, please upgrade before installing.

Python Version Compatibility

paged-list supports Python 3.9 and later versions:

• ✅ Python 3.9+ (recommended)
• ✅ Python 3.10+
• ✅ Python 3.11+
• ✅ Python 3.12+

The package is tested across multiple Python versions and operating systems (Linux, Windows, macOS) to ensure compatibility.

Testing Compatibility

To test compatibility on your system:

# Install with development dependencies
pip install paged-list[dev]

# Run compatibility tests
python -m pytest tests/test_python_compatibility.py -v

# Or use the standalone compatibility script
python scripts/test_compatibility.py

Multi-Version Testing

For developers, you can test across multiple Python versions using tox:

# Install tox
pip install tox

# Test on available Python versions
tox

# Test on specific Python version
tox -e py39

# Run linting and formatting checks
tox -e flake8,black,mypy

Note: The tox command will automatically skip Python versions that aren't installed on your system.

Install from source:

git clone https://github.com/christensendaniel/paged-list.git
cd paged-list
pip install -e .

Quick Start

from paged_list import PagedList

# Create a disk-backed list
cl = PagedList(chunk_size=50000, disk_path="data")

# Add data - will automatically chunk to disk when needed
for i in range(100000):
    cl.append({"id": i, "value": f"item_{i}", "score": i * 1.5})

# Access data like a regular list
print(cl[0])  # First item
print(cl[-1])  # Last item
print(cl[1000:1010])  # Slice of 10 items

# Update items
cl[5] = {"id": 5, "value": "updated", "score": 99.9}


# Apply transformations to all data (uses threading)
def double_score(record):
    record["score"] *= 2
    return record


cl.map(double_score)

# Serialize complex data types to JSON strings
cl.serialize()

# Clean up when done
cl.cleanup_chunks()

Use Cases

• Large Dataset Processing: Handle datasets that don't fit in memory
• Data Pipelines: Process streaming data with automatic disk overflow
• ETL Operations: Transform large datasets chunk by chunk
• Data Analysis: Analyze large datasets without memory constraints
• Caching: Implement persistent, memory-efficient caches

Advanced Usage

Context Manager (Recommended)

from paged_list import PagedList

with PagedList(chunk_size=10000) as cl:
    # Add lots of data
    for i in range(1000000):
        cl.append({"data": f"item_{i}"})

    # Process data
    result = cl[500000:500010]

    # Automatic cleanup on exit

Custom Serialization

# Serialize complex Python objects to JSON strings
cl.append(
    {
        "id": 1,
        "metadata": {"tags": ["python", "data"], "active": True},
        "scores": [1.2, 3.4, 5.6],
    }
)

cl.serialize()  # Converts lists, dicts, and bools to JSON strings

Parallel Processing

# Process data in parallel across chunks
def process_record(record):
    record["processed"] = True
    record["timestamp"] = "2024-01-01"
    return record


cl.map(process_record, max_workers=4)  # Use 4 threads

Performance

PagedList is designed for scenarios where:

• Your dataset is too large for memory
• You need random access to data
• You want to process data in chunks
• Memory usage is more important than raw speed

Typical performance characteristics:

• Memory usage: O(chunk_size) instead of O(total_items)
• Access time: O(1) for sequential access, O(log chunks) for random access
• Disk usage: Temporary pickle files (cleaned up automatically)

Development

This project uses Hatch for development environment management and packaging.

Development Quick Start

# Install Hatch
pip install hatch

# Run tests
hatch run test

# Run tests with coverage
hatch run test-cov

# Format and lint code
hatch run format
hatch run lint

# Run everything (format, lint, test with coverage)
hatch run all

Hatch Environments

• default: Main development environment with all tools
• test: Testing across Python 3.9-3.13
• docs: Documentation building

For detailed development setup, see CONTRIBUTING.md.

Legacy Commands

# Also works (legacy pytest)
pytest

# Run examples
python -m paged_list demo      # Small demonstration
python -m paged_list example   # Full example with 1M items

About the Author

paged-list was created by Christensen Daniel, a passionate data engineer who specializes in building tools that make working with large datasets more efficient and enjoyable.

Connect

• LinkedIn: dbchristensen - For data engineering insights and project updates
• GitHub: christensendaniel - Explore more projects and contributions
• Email: [e-mail]

License

This project is licensed under the MIT License - see the LICENSE file for details.