mmap-ninja-dataframe 0.8.0

mmap_ninja_dataframe: Memory mapped data structures

mmap_ninja_dataframe

Memory-mapped dataframe abstraction based on mmap_ninja

Run tests:

uvx --with-editable . --with joblib --with zstandard --with dnn_cool_synthetic_dataset --with opencv-contrib-python --with transformers pytest

TextPropertiesMmap

TextPropertiesMmap stores unique texts and their computed properties (e.g. embeddings, token counts) as memory-mapped arrays. Texts are deduplicated by content hash. Properties can be added incrementally — a property does not need to be fully computed before the store is used.

Create from a list of texts

from mmap_ninja_dataframe import TextPropertiesMmap

store = TextPropertiesMmap.from_texts(
    out_dir="my_store",
    texts=["Hello, world!", "Memory maps are fast.", "Another sentence."],
)

Add a text and get its index

idx = store.add("A new sentence.")
# Returns the index. If the text already exists, returns its existing index without duplicating.

Add multiple texts

indices = store.update(["First.", "Second.", "Hello, world!"])
# "Hello, world!" already exists — its existing index is returned, store does not grow.

Add a computed property

import numpy as np

embeddings = [np.random.rand(768).astype(np.float32) for _ in range(len(store))]
store.add_property("embedding", embeddings)

Properties can be added partially — the array does not need to cover all texts yet:

store.add_property("token_count", [np.array([5]), np.array([4])])  # only first two texts

Query property status

result = store.get_property("embedding")
print(result.unprocessed_count)    # number of texts without this property computed
print(result.staging_files_count)  # number of pending staged result files
print(len(result.mmap))            # number of computed results

Get unprocessed indices

indices = store.get_unprocessed_indices_for_property("embedding")
texts_to_process = store.text[indices]

Stage and flush results

Use set_results_for_property to stage results (e.g. from a batch inference job). Results are flushed automatically when they can be applied in order:

batch_texts = ["First.", "Second."]
batch_embeddings = [np.random.rand(768).astype(np.float32) for _ in batch_texts]
store.set_results_for_property("embedding", batch_texts, batch_embeddings)

# Explicitly flush any remaining staged results:
store.flush_results_for_property_if_possible("embedding")

Look up properties by text

props = store.get_text_properties("Hello, world!")
# {"embedding": array([...]), "token_count": array([5])}
# If a property is not yet computed for this text, it appears in props["unprocessed"]

Fetch properties for multiple texts

result = store.get_properties_for_texts(["Hello, world!", "Memory maps are fast."])
# {"text": [...], "content_hash": [...], "idx": [0, 1], "embedding": [...]}
# Raises KeyError for unknown texts, ValueError if any property is not fully computed.

Check overall status

status = store.get_status()
# Returns a list of PropertyResult for properties that are not yet fully computed.
for r in status:
    print(r)

Delete a property

store.delete_property("embedding")
# Removes the mmap directory and any staged files for that property.

List computed properties

store.get_properties()  # ["embedding", "token_count"]
# Does not include "text" or "content_hash" — use store.text and store.content_hash directly.