concurrent-collections 2.0.3

A brief description of concurrent-collections

Python Concurrent (thread-safe) collections

tl;dr

Despite what many people think, Python's built-in list, dict, and deque are NOT thread-safe.
They may be thread safe for some operations, but not all.
This created a lot of confusion in the Python community.
Google style-guide recommends to not rely on atomicity of built-in collections.

concurrent_collections provides thread-safe alternatives by using locks internally to ensure safe concurrent access and mutation from multiple threads.

Inspired from the amazing C#'s concurrent collections.

Why use these collections?

There is a lot of confusion on whether Python collections are thread-safe or not^{1, 2, 3}.

The bottom line is that Python's built-in collections are not fully thread-safe for all operations.
While some simple operations (like list.append() or dict[key] = value) are thread-safe due to the Global Interpreter Lock (GIL), compound operations and iteration with mutation are not. This can lead to subtle bugs, race conditions, or even crashes in multi-threaded programs.

See the Python FAQ: "What kinds of global value mutation are thread-safe?" for details. The FAQ explains that only some (if common) operations are guaranteed to be atomic and thread-safe, but for anything more complex, you must use your own locking.
The docs even go as far as to say:

When in doubt, use a mutex!

Which is telling.

Even Google recommends to not rely on atomicity of built-in collections.

This concurrent_collections library provides drop-in replacements that handle locking for you.
Suggestions and feedbacks are welcome.

1. Are lists thread-safe?

2. Google style guide advises against relying on Python's assignment atomicity

3. What kind of "thread safe" are deque's actually?

Installation

Pip:

pip install concurrent_collections

My recommendation is to always use uv instead of pip – I personally think it's the best package and environment manager for Python.

uv add concurrent_collections

Collections

ConcurrentBag

A thread-safe, list-like collection.

from concurrent_collections import ConcurrentBag

bag = ConcurrentBag([1, 2, 3])
bag.append(4)
print(list(bag))  # [1, 2, 3, 4]

ConcurrentDictionary

A thread-safe dictionary. It has a few notable methods:

• assign_atomic()
• get_locked()
• update_atomic()

ConcurrentDictionary's assign_atomic()

Assigns a dictionary value under a key in a thread-safe way. While dict["somekey"] = value is allowed, it's best to use assign_atomic() for clarity of intent. Using normal assignment will work but raise a UserWarning.

ConcurrentDictionary's get_locked()

When working with ConcurrentDictionary, you should use the get_locked method to safely read or update the value for a specific key in a multi-threaded environment. This ensures that only one thread can access or modify the value for a given key at a time, preventing race conditions.

from concurrent_collections import ConcurrentDictionary

d = ConcurrentDictionary({'x': "some value" })

# Safely read and update the value for 'x'
with d.get_locked('x') as value:
    # value is locked for this thread
    d['x'] = "new value"

ConcurrentDictionary's update_atomic()

Performs a thread-safe, in-place update to an existing value under a key.

d = ConcurrentDictionary({'x': 1 })
d.update_atomic("x", lambda v: v + 1) # d now contains 2 under the 'x' key.

ConcurrentQueue

A thread-safe double-ended queue.

from concurrent_collections import ConcurrentQueue

q = ConcurrentQueue()
q.append(1)
q.appendleft(0)
print(q.pop())      # 1
print(q.popleft())  # 0

License

MIT License