Skip to main content

Converter matrix and type determination for a range of array formats, focusing on sparse arrays

Project description


Format detection, identifiers and converter matrix for a range of numerical array formats (backends) in Python, focusing on sparse arrays.


Basic usage:

import numpy as np
import sparseconverter as spc

a1 = np.array([
    (1, 0, 3),
    (0, 0, 6)

# array conversion
a2 = spc.for_backend(a1, spc.SPARSE_GCXS)

# format determination
print("a1 is", spc.get_backend(a1), "and a2 is", spc.get_backend(a2))
a1 is numpy and a2 is sparse.GCXS

See examples/ directory for more!


This library can help to implement algorithms that support a wide range of array formats as input, output or for internal calculations. All dense and sparse array libraries already do support format detection, creation and export from and to various formats, but with different APIs, different sets of formats and different sets of supported features -- dtypes, shapes, device classes etc.

This project creates an unified API for all conversions between the supported formats and takes care of details such as reshaping, dtype conversion, and using an efficient intermediate format for multi-step conversions.


  • Supports Python 3.7 - (at least) 3.10
  • Defines constants for format identifiers
  • Various sets to group formats into categories:
    • Dense vs sparse
    • CPU vs CuPy-based
    • nD vs 2D backends
  • Efficiently detect format of arrays, including support for subclasses
  • Get converter function for a pair of formats
  • Convert to a target format
  • Find most efficient conversion pair for a range of possible inputs and/or outputs

That way it can help to implement format-specific optimized versions of an algorithm, to specify which formats are supported by a specific routine, to adapt to availability of CuPy on a target machine, and to perform efficient conversion to supported formats as needed.

Supported array formats

Still TODO

  • cupyx.sparse formats with dtype bool
  • PyTorch arrays
  • SciPy sparse arrays as opposed to SciPy sparse matrices.
  • More detailed cost metric based on more real-world use cases and parameters.

Known issues

  • conda install -c conda-forge cupy on Python 3.7 and Windows 11 may install cudatoolkit 10.1 and cupy 8.3, which have sporadically produced invalid data structures for cupyx.sparse.csc_matrix for unknown reasons. This doesn't happen with current versions. Running the benchmark function benchmark_conversions() can help to debug such issues since it performs all pairwise conversions and checks for correctness.


This project is developed primarily for sparse data support in LiberTEM. For that reason it includes the backend CUDA, which indicates a NumPy array, but targeting execution on a CUDA device.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sparseconverter-0.1.0rc2.tar.gz (15.6 kB view hashes)

Uploaded Source

Built Distribution

sparseconverter-0.1.0rc2-py3-none-any.whl (13.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page