CSTL : The C++ Standard Template Library (STL) for Python

In this cstl tool, we wrap several C++ STL containers to use in Python. The containers use native C++ implementation and does not have the Copy-on-Write issue like native list, and dict in Python (Someone refers to it as memory leakage in multiprocessing which always happens in all Python native objects with ref count). Though it is designed to solve the CoW issue, it can also be used in scenarios where a standard C++ container is needed.

Install

Install from pip:

pip install cstl

Build from source:

conda install swig # You should first install swig
git clone https://github.com/fuzihaofzh/cstl.git
cd cstl
./build.sh
python setup.py install --single-version-externally-managed --record files.txt

Usage

import cstl

# Directly covert containers from python
v = cstl.frompy({"1":[1,2,3], "2":[4,5,6]})
v["1"][2] = 10
pv = cstl.topy(v)
print(pv)

# You can also explictly specify the type
vec = cstl.VecInt([1,2,3,4,5,6,7])
print(vec[2])      #3

vec[2] = 1
print(vec[2])      #1

vec.append(10)
print(vec[-1])     #10

# User should explictly convert std::vector into list as follows:
print(list(vec))   #[1, 2, 1, 4, 5, 6, 7, 10] 

vmif = cstl.VecMapIntFloat([{1:3.4},{4:5.5}])
print(vmif[0][1])  #3.4000000953674316

Please refer to more usage in test/test/py.

Supported Datatype

We support the following data types as the elements in the containers

Supported Containers

The supported containers are listed as follows

We also support nested container, namely, structure like std::unordered_map< std::string,std::vector< bool > > is supported. Currently, at most 2 nested layers are supported. If you want to support more layers, simply uncomment the line full.update(third) in generate_swig.py and compile from the source. But please note the the generated files could be very large.

Please be noted that if you want to pass a Python set to set in cstl, you should first convert it into a list. This is a known issue for swig, let's see whether it will be resolved soon.

Speed Comparison

It can be concluded from the table that cstl is slower than python native list and numpy in basic tasks. However, it is faster in some specific task. Most importantly, it sovles the CoW issue and provide more data structures than numpy.

Copy-on-Write Issue in Python

Copy-on-write is a feature and not a bug for Python. Usually, in python multiprocessing, if we have a large data shared by each process, we will see as the program runs, the engaged memory grows gradually and finally occupy all the machine's memory and raise a Memory Error. Someone refers to it as memory leaky in the multiprocess. However, this is caused by a feature of Python. It is because, in multi-processing programs, the shared object will be copied to each process if they access the data. However, if the data is large and we use several processes, the memory cannot hold a separate copy for each process. This cannot be solved in Python as all Python's native structures with ref count have such problems (feature?). A more detailed discussion can be found at https://github.com/pytorch/pytorch/issues/13246 . Many other containers like pytorch, numpy. However, they do not support data structure like nested map.

Supported STL Containers List

We support the following nested containers. If you need more than 2 layers of nested containers please refer to Supported Containers