QMEM

Memory library for seamless data ingestion, storage, and retrieval with customizable embedding models.

QMem is a toolkit for vector search. It provides a command-line interface (CLI) and a Python library for interacting with a Qdrant database. It is designed for directness and utility, offering a guided CLI for interactive tasks and a minimal Python API for programmatic control..

Features

• Store documents in a vector database (Qdrant(cloud) or Chroma(local))
• Search text by semantic meaning, not just exact words
• Apply filters to narrow search results
• Choose embeddings from OpenAI, Gemini, Voyage, or MiniLM
• MongoDB integration for analytics

Installation

pip install qmem

Getting Started

Step 1: Setup

Run once to configure qmem:

qmem init

You will be asked to select:

• Database backend: Qdrant (cloud) or Chroma (local)
• Embedding provider: OpenAI, Gemini, Voyage, or MiniLM
• API keys and embedding model

Configuration is saved in .qmem/config.toml.

Step 2: Prepare Your Data

qmem expects a .jsonl file (one JSON object per line).

Example:

{"query": "What is the plot of Inception?", "response": "A thief enters dreams to steal secrets.", "summary": "Dream heist story.", "title": "Inception", "description": "A skilled thief enters people's dreams to steal corporate secrets.", "genre": "Sci-Fi", "year": 2010}
{"query": "Who is the main character in The Godfather?", "response": "Michael Corleone, son of Vito.", "summary": "Mafia family saga.", "title": "The Godfather", "description": "The patriarch of a crime dynasty hands control to his reluctant son.", "genre": "Crime", "year": 1972}
{"query": "What challenge does Batman face in The Dark Knight?", "response": "He must stop the Joker.", "summary": "Hero vs chaos.", "title": "The Dark Knight", "description": "Batman faces the Joker, a criminal mastermind terrorizing Gotham.", "genre": "Action", "year": 2008}
{"query": "What is Parasite about?", "response": "A poor family cons its way into a rich household.", "summary": "Class divide thriller.", "title": "Parasite", "description": "A poor family infiltrates a wealthy household with dire consequences.", "genre": "Thriller", "year": 2019}
{"query": "What is Interstellar about?", "response": "Explorers travel through a wormhole to save humanity.", "summary": "Space odyssey.", "title": "Interstellar", "description": "A group of explorers journey through a wormhole to find a new home.", "genre": "Adventure", "year": 2014}

• query: main text to search on
• response:response to the query
• others: extra information

Step 3: Add Data

import qmem as qm

# Create a collection
qm.create(collection_name="testing", dim=1024, distance_metric="cosine")

# Ingest data from file
qm.ingest(file="data.jsonl", embed_field="query",payload_field="x,y,z") #what to put in payload , not added then everything goes in payload

Step 4: Search

results = qm.retrieve(
    query="who is batman",
    top_k=5,
    show=["description", "title"]
)
print(results)

Results are returned in a table showing similarity scores and payload fields.

Step 5: Use Filters

Build filters interactively (CLI):

qmem filter

Filters are saved under .qmem/filters/.

Apply a filter in Python:

filtered = qm.retrieve_filter(
    query="who is batman",
    filter_json=".qmem/filters/latest.json",
    top_k=5,
    show=["description", "title"]
)
print(filtered)

Step 6: Mirror Data to MongoDB

qm.mongo(
    collection_name="testing",
    mongo_db="final_test_db",
    mongo_collection="final_test",
    fields=["description", "title"]
)

This copies your Qdrant collection into MongoDB for downstream use.

CLI Commands

qmem init      # configure qmem
qmem filter    # create filters interactively
qmem index     # (only for Qdrant) build indices for faster filtering

Python API

• qm.create() → Create a new collection
• qm.ingest() → Add documents from a file
• qm.retrieve() → Search documents by meaning
• qm.retrieve_filter() → Search with filters
• qm.mongo() → Copy a collection into MongoDB

Why use qmem?

• Simple to set up and use
• Works with both local (Chroma) and cloud (Qdrant) databases
• AI embeddings understand meaning, not just exact words
• Filters allow more precise retrieval
• MongoDB integration enables analytics pipelines

Example Workflow

1. qmem init → configure backend and embeddings
2. qm.create(...) → create a collection
3. qm.ingest(...) → add data from JSONL
4. qm.retrieve(...) → search data by meaning
5. qmem filter + qm.retrieve_filter(...) → apply filters
6. qm.mongo(...) → mirror to MongoDB if needed

qmem makes it easy to:

• Store your documents
• Search them intelligently
• Filter results by properties
• Extend usage with MongoDB integration

Searching for “car” may also return results for “automobile” or “vehicle.”