waveflowdb-client 1.0.3

VectorLake SDK — Deterministic backend engine powering agent workflows

WaveflowDB SDK — VectorLake Python Client v2.0.0

A Python SDK for interacting with WaveflowDB and performing WaveQL (VQL) brace-based semantic retrieval.

This SDK provides:

• Full Config management via constructor args, environment variables, or .env file
• Document ingestion — direct payload mode or batched filesystem mode
• Intelligent document sync with MD5-based change detection
• Resumable batch uploads with start_from_batch / end_batch range control
• Semantic search and top-matching-doc retrieval with hybrid filtering
• Namespace and document metadata queries
• Structured CSV logging for performance, errors, and skipped files
• Simple {stem}_part{num}.txt chunk naming for idempotent re-runs

---

📌 Overview

Vector Lake is an agentic backend for enterprises to build AI products, enabling:

• Natural-language structured filtering through WaveQL (VQL)
• Hybrid ranking (Filter + Semantic)
• Zero-schema ingestion (no JSON schemas required)
• SQL-like logical joins on raw text
• Automatic semantic fallback when filters are absent

---

🚀 Getting Started

1. Install Dependencies

pip install waveflowdb_client

Optional extras for richer file support:

pip install PyPDF2 python-docx python-dotenv tqdm

2. Configure API Credentials

The SDK resolves settings in this priority order: constructor args → environment variables → .env file → hard-coded defaults.

Option A — .env file

VECTOR_LAKE_API_KEY=your_api_key_here
VECTOR_LAKE_HOST=https://waveflow-analytics.com
USER_ID=[e-mail]
NAMESPACE=your_namespace

Option B — Constructor arguments

from waveflowdb_client import Config, VectorLakeClient

cfg = Config(
    api_key="your_api_key_here",
    host="https://waveflow-analytics.com",
    vector_lake_path="/path/to/documents",
)
client = VectorLakeClient(cfg)

---

🗂️ Supported File Types

The allowed extension list is the single source of truth defined in Config (or VECTOR_LAKE_ALLOWED_EXTENSIONS). The default set is:

Extension	Processing strategy
txt	Plain text, paragraph-chunked
py	Plain text read, paragraph-chunked
ipynb	Code cells extracted, paragraph-chunked
pdf	Text extracted via PyPDF2, paragraph-chunked
docx	Text extracted via python-docx, paragraph-chunked
csv	Row-safe split; header preserved in every chunk
jsonl / ndjson	Record-safe split; each line validated as JSON
json	Kept atomic — never split

To add or remove extensions without changing code, update VECTOR_LAKE_ALLOWED_EXTENSIONS in your .env.

---

📦 Chunk Naming

All chunks written to disk follow this simple convention:

{stem}_part{num}.txt

Examples — a PDF that splits into 3 parts:

electrolytes-test-report_part1.txt
electrolytes-test-report_part2.txt
electrolytes-test-report_part3.txt

All formats (pdf, docx, py, ipynb, txt, etc.) produce .txt chunk files. The VersionedChunkName model provides build() and parse() helpers and an is_versioned() static method. Chunks already on disk are reused on re-run — the pipeline is idempotent and safe to restart after a crash.

---

🔄 Document Sync — How Uploads Work

  YOUR FOLDER (vector_lake_path/)
  ┌─────────────────────────────────────────────┐
  │  report.pdf   notes.txt   data.csv  ...     │
  └─────────────────────────┬───────────────────┘
                             │
                    sync_documents()
                    classifies each file
                             │
          ┌──────────────────┼───────────────────┐
          ▼                  ▼                   ▼
     No chunk          Chunk exists,        Chunk exists,
      on disk           MD5 matches          MD5 differs
          │                  │                   │
        NEW              UNCHANGED             CHANGED
          │                  │                   │
      add_docs()           skip             refresh_docs()
          │                                      │
          └──────────────────┬───────────────────┘
                             ▼
                    Upload in batches
                    (resumable via start_from_batch)

sync_documents() is the recommended upload method for most use cases. Use add_documents() or refresh_documents() directly only when you are certain all files are brand-new or all already exist in the namespace.

Resumable Batch Ranges

All upload methods support start_from_batch and end_batch. If a long run is interrupted, set START_FROM to the last successful batch + 1 and re-run:

  Batch 1 ✓  Batch 2 ✓  Batch 3 ✓  [CRASH]  Batch 4 ...  Batch 39
  ──────────────────────────────────────────────────────────────────
                                     ↑
                              start_from_batch=4
                              Re-run resumes here

---

Search Modes — flat vs flat_filter

The two search_type values represent fundamentally different retrieval strategies. Both require hybrid_filter=True.

search_type="flat" — Full Corpus Fusion

Both semantic search and hybrid filter run independently across ALL documents in the namespace. Their results are then combined by the proprietary fusion algorithm, which re-ranks everything into a single fused list.

  Your Query
      │
      ├─────────────────────────────────┐
      ▼                                 ▼
  Semantic Search               Hybrid Filter
  (ALL documents                (keyword scan
   in namespace)                 ALL documents)
      │                                 │
      │   ┌─────────────────────────────┘
      ▼   ▼
  Proprietary Fusion Algorithm
  (combines + re-ranks both result sets)
      │
      ▼
  ┌─────────────────────────────────────────────────┐
  │  Tier 1: matched BOTH  →  highest fused score   │
  │  Tier 2: filter match only                      │
  │  Tier 3: semantic match only (fallback)         │
  └─────────────────────────────────────────────────┘

Use flat for: exploratory search, maximum recall, when you want every document to have a chance to surface.

---

search_type="flat_filter" — Filtered Semantic Search

The hybrid filter runs first as a hard gate — only documents that pass are eligible for semantic search. Semantic ranking then runs only over that filtered candidate set.

  Your Query
      │
      ▼
  Hybrid Filter
  (keyword scan ALL documents)
      │
      ▼
  Filtered candidate set
  (only docs that passed the filter)
      │
      ▼
  Semantic Search
  (only over filtered candidates)
      │
      ▼
  Final ranked results
  (higher precision, smaller result set)

Use flat_filter for: targeted retrieval, when filter criteria are strong, when you want tighter and more focused results.

---

Quick Comparison

  ┌─────────────────┬────────────────────────────┬────────────────────────────┐
  │                 │           flat             │        flat_filter         │
  ├─────────────────┼────────────────────────────┼────────────────────────────┤
  │ Semantic scope  │ ALL documents              │ Filter-passing docs only   │
  │ Filter role     │ Scoring signal (boosts)    │ Hard gate (excludes)       │
  │ Result style    │ Fused, broad               │ Precise, narrower          │
  │ Best for        │ Exploratory search         │ Targeted retrieval         │
  │ hybrid_filter   │ Required (True)            │ Required (True)            │
  └─────────────────┴────────────────────────────┴────────────────────────────┘

---

run.py — Local Admin Starter Script

run.py is a ready-to-use local launcher for admin and testing purposes. Set your credentials via .env, drop files into your upload folder, uncomment exactly one function in __main__, and run:

python run.py

Setup

# run.py — top of file
API_KEY          = os.getenv("VECTOR_LAKE_API_KEY")
HOST             = os.getenv("VECTOR_LAKE_HOST", "https://waveflow-analytics.com")
VECTOR_LAKE_PATH = os.getenv("VECTOR_LAKE_PATH", "upload")
USER_ID          = os.getenv("USER_ID")
NAMESPACE        = os.getenv("NAMESPACE")

START_FROM = 1    # ← set to (last completed batch + 1) to resume

Function Map

  run.py
  │
  ├── UPLOAD ──────────────────────────────────────────────────────────────
  │   ├── run_add_direct()          Add new files via string content
  │   ├── run_add_path()            Add new files from disk (batched)
  │   ├── run_refresh_direct()      Update existing files via string content
  │   ├── run_refresh_path()        Update existing files from disk (batched)
  │   ├── run_sync()                Auto-classify & route (recommended)
  │   └── run_sync_dry_run()        Preview sync plan, no upload
  │
  ├── QUERY ───────────────────────────────────────────────────────────────
  │   ├── run_match_static(q)             flat search, static index
  │   ├── run_match_filtered(q)           flat_filter search, static index
  │   ├── run_match_dynamic(q)            flat search, temp docs
  │   └── run_match_dynamic_filtered(q)   flat_filter search, temp docs
  │
  └── INFO ────────────────────────────────────────────────────────────────
      ├── run_health()              Ping server
      ├── run_namespace_details()   Storage & quota info
      └── run_docs_info()           List all indexed documents

Full Script

"""
run.py — Simple launcher for VectorLake SDK v2.0.0

QUICK START
───────────
1. Copy .env.example → .env and fill in your credentials.
2. Drop your files into VECTOR_LAKE_PATH (default: ./upload).
3. Uncomment exactly ONE action at the bottom and run:  python run.py

RESUMING A BATCH UPLOAD
───────────────────────
Set START_FROM to (last successful batch + 1) and re-run.
"""

import json
import os
from dotenv import load_dotenv

load_dotenv()

from waveflowdb_client import Config, VectorLakeClient

# ── Configuration ─────────────────────────────────────────────────────────────
API_KEY          = os.getenv("VECTOR_LAKE_API_KEY")
HOST             = os.getenv("VECTOR_LAKE_HOST", "https://waveflow-analytics.com")
VECTOR_LAKE_PATH = os.getenv("VECTOR_LAKE_PATH", "upload")
USER_ID          = os.getenv("USER_ID")
NAMESPACE        = os.getenv("NAMESPACE")

START_FROM = 1    # ← set to (last completed batch + 1) to resume

# ── Client ────────────────────────────────────────────────────────────────────
def get_client() -> VectorLakeClient:
    cfg = Config(
        api_key=API_KEY,
        host=HOST,
        vector_lake_path=VECTOR_LAKE_PATH,
    )
    return VectorLakeClient(cfg)

client = get_client()

# ── Print helper ──────────────────────────────────────────────────────────────
def _print(label: str, result) -> None:
    print(f"\n{'─'*70}")
    print(f"  {label}")
    print(f"{'─'*70}")
    if isinstance(result, list) and result and isinstance(result[0], dict):
        keys = list(result[0].keys())
        col  = max(len(k) for k in keys) + 2
        print("  " + "".join(k.ljust(col) for k in keys))
        print("  " + "─" * (col * len(keys)))
        for row in result:
            print("  " + "".join(str(row.get(k, "")).ljust(col) for k in keys))
    else:
        print(json.dumps(result, indent=2, default=str))

# ─────────────────────────────────────────────────────────────────────────────
#  UPLOAD / REFRESH / SYNC
# ─────────────────────────────────────────────────────────────────────────────

def run_health():
    _print("HEALTH CHECK", client.health_check(USER_ID, NAMESPACE))

def run_add_direct():
    """Direct mode: add brand-new files by passing content as strings."""
    result = client.add_documents(
        user_id=USER_ID,
        vector_lake_description=NAMESPACE,
        files_name=["test1.txt", "test2.txt"],
        files_data=["hello world", "this is test doc 2"],
    )
    _print("ADD DOCUMENTS — direct", result)

def run_add_path():
    """
    Batch mode: add files from VECTOR_LAKE_PATH.
    Use only when ALL files are brand-new to the namespace.
    Use run_sync() for a mixed folder.
    """
    result = client.add_documents(
        user_id=USER_ID,
        vector_lake_description=NAMESPACE,
        start_from_batch=START_FROM
    )
    _print(f"ADD DOCUMENTS — path (start={START_FROM})", result)

def run_refresh_direct():
    """Direct mode: refresh files that already exist in the namespace."""
    result = client.refresh_documents(
        user_id=USER_ID,
        vector_lake_description=NAMESPACE,
        files_name=["test1.txt"],
        files_data=["UPDATED CONTENT FOR TEST1"],
    )
    _print("REFRESH DOCUMENTS — direct", result)

def run_refresh_path():
    """
    Batch mode: refresh files from VECTOR_LAKE_PATH.
    Use only when ALL files already exist in the namespace.
    Use run_sync() for a mixed folder.
    """
    result = client.refresh_documents(
        user_id=USER_ID,
        vector_lake_description=NAMESPACE,
        start_from_batch=START_FROM,
    )
    _print(f"REFRESH DOCUMENTS — path (start={START_FROM})", result)

def run_sync():
    """
    *** RECOMMENDED FOR MOST USE CASES ***

    Classifies every file in VECTOR_LAKE_PATH and routes it automatically:
      No chunk on disk      → add_docs     (new file)
      Chunk exists, same    → skipped      (unchanged)
      Chunk exists, differs → refresh_docs (updated file)
    """
    result = client.sync_documents(
        user_id=USER_ID,
        vector_lake_description=NAMESPACE,
        start_from_batch=START_FROM,
    )
    _print("SYNC DOCUMENTS", result)

def run_sync_dry_run():
    """Preview sync classification without touching the server."""
    result = client.sync_documents(
        user_id=USER_ID,
        vector_lake_description=NAMESPACE,
        dry_run=True,
    )
    _print("SYNC DRY RUN — classification plan", result)

# ─────────────────────────────────────────────────────────────────────────────
#  QUERY
# ─────────────────────────────────────────────────────────────────────────────

def run_match_static(query: str):
    """Semantic search across ALL docs (search_type='flat')."""
    _print("MATCHING DOCS — flat (all docs)", client.get_matching_docs(
        query=query, user_id=USER_ID, vector_lake_description=NAMESPACE,
        pattern="static", hybrid_filter=True, search_type="flat",
        top_docs=5, threshold=0.1, with_data=True,
    ))

def run_match_filtered(query: str):
    """Semantic search ONLY over hybrid-filter-passing docs (search_type='flat_filter')."""
    _print("MATCHING DOCS — flat_filter (filtered docs only)", client.get_matching_docs(
        query=query, user_id=USER_ID, vector_lake_description=NAMESPACE,
        pattern="static", hybrid_filter=True, search_type="flat_filter",
        top_docs=5, threshold=0.1, with_data=True,
    ))

def run_match_dynamic(query: str):
    _print("MATCHING DOCS — dynamic / flat", client.get_matching_docs(
        query=query, user_id=USER_ID, vector_lake_description=NAMESPACE,
        pattern="dynamic", hybrid_filter=True, search_type="flat",
        files_name=["temp.txt"], files_data=["Sample dynamic content"],
    ))

def run_match_dynamic_filtered(query: str):
    _print("MATCHING DOCS — dynamic / flat_filter", client.get_matching_docs(
        query=query, user_id=USER_ID, vector_lake_description=NAMESPACE,
        pattern="dynamic", hybrid_filter=True, search_type="flat_filter",
        files_name=["temp.txt"], files_data=["Sample dynamic content"],
    ))

# ─────────────────────────────────────────────────────────────────────────────
#  INFO
# ─────────────────────────────────────────────────────────────────────────────

def run_namespace_details():
    _print("NAMESPACE DETAILS", client.get_namespace_details(USER_ID, NAMESPACE))

def run_docs_info():
    _print("DOCS INFORMATION", client.get_docs_information(USER_ID, NAMESPACE))

# ─────────────────────────────────────────────────────────────────────────────
#  MAIN — uncomment exactly ONE line
# ─────────────────────────────────────────────────────────────────────────────
if __name__ == "__main__":

    QUERY = """select top 2 where query is "what is cbc absolut count of yash patel"
     contains {yash patel} """

    # ── UPLOAD (choose based on server state) ────────────────────────────────
    # run_add_path()
    # run_add_direct()
    # run_refresh_path()
    # run_refresh_direct()
    # run_sync()           # ← recommended when folder has mix of new + existing files
    # run_sync_dry_run()   # ← preview sync plan without uploading

    # ── QUERY ────────────────────────────────────────────────────────────────
    # run_health()
    run_match_static(QUERY)           # flat        — all docs
    # run_match_filtered(QUERY)         # flat_filter — filtered docs only
    # run_match_dynamic(QUERY)
    # run_match_dynamic_filtered(QUERY)

    # ── INFO ─────────────────────────────────────────────────────────────────
    # run_namespace_details()
    # run_docs_info()

---

Sample Outputs

run_health()

Pings the backend to confirm connectivity and that your namespace is reachable.

──────────────────────────────────────────────────────────────────────
  HEALTH CHECK
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "content": "Processing time at server 0.00 ms",
    "docs": []
  },
  "message": "Health check successful",
  "status_code": 200
}

---

run_add_direct()

Uploads brand-new documents by passing file names and content directly as strings — no files on disk needed.

────────────────────────────────────────────────────────────
  ▶  ADD_DOCUMENTS  (direct mode)
────────────────────────────────────────────────────────────
making request
   ✓  Done
────────────────────────────────────────────────────────────
──────────────────────────────────────────────────────────────────────
  ADD DOCUMENTS — direct
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "content": "2 file(s) uploaded successfully",
    "docs": {
      "new_files": ["test2", "test1"],
      "updated_files": [],
      "failed_files": []
    }
  },
  "message": "2 file(s) uploaded successfully",
  "status_code": 200
}

---

run_add_path()

Reads all supported files from VECTOR_LAKE_PATH, chunks them into {stem}_part{num}.txt files, and uploads in batches with a live progress bar. Use only when all files are brand-new to the namespace.

────────────────────────────────────────────────────────────────
  ▶  ADD_DOCS
────────────────────────────────────────────────────────────────
   Source files  : 152
   Chunks        : 164
   This run      : 39 batches
   Overall total : 39 batches
   Batch range   : #1 → #39
────────────────────────────────────────────────────────────────
add_docs:   3%|████                    | 1/39 [00:22<14:00, 22.1s/batch]
...
────────────────────────────────────────────────────────────────
  ■  ADD_DOCS COMPLETE
────────────────────────────────────────────────────────────────
   ✓  All batches succeeded
   Succeeded : 39/39
   Failed    : 0
────────────────────────────────────────────────────────────────

Resuming after a crash — set START_FROM to last successful batch + 1:

START_FROM = 15   # batches 1–14 already done

---

run_refresh_direct()

Updates files that already exist in the namespace, passing updated content directly as strings.

────────────────────────────────────────────────────────────
  ▶  REFRESH_DOCUMENTS  (direct mode)
────────────────────────────────────────────────────────────
making request
   ✓  Done
────────────────────────────────────────────────────────────
{
  "reply": {
    "content": "1 file(s) updated successfully",
    "docs": {
      "new_files": [],
      "updated_files": ["test1"],
      "failed_files": []
    }
  },
  "message": "1 file(s) updated successfully",
  "status_code": 200
}

---

run_sync() — Recommended

Classifies every file in the upload folder and automatically routes it.

────────────────────────────────────────────────────────────────
  🔍  CLASSIFYING FILES
────────────────────────────────────────────────────────────────
classify: 100%|████████████████████| 152/152 [00:02<00:00, file/s]

   NEW=12  CHANGED=5  UNCHANGED=135

────────────────────────────────────────────────────────────────
  ▶  ADD_DOCS          (12 new files)
────────────────────────────────────────────────────────────────
...
────────────────────────────────────────────────────────────────
  ▶  REFRESH_DOCS      (5 changed files)
────────────────────────────────────────────────────────────────
...
────────────────────────────────────────────────────────────────
  ■  SYNC COMPLETE
────────────────────────────────────────────────────────────────
   Added     : 12
   Refreshed : 5
   Skipped   : 135
   Failed    : 0
────────────────────────────────────────────────────────────────

---

run_sync_dry_run()

Previews the sync plan without touching the server.

────────────────────────────────────────────────────────────────
  ■  DRY RUN COMPLETE — no files uploaded
────────────────────────────────────────────────────────────────
   NEW=12  REFRESH=5  SKIP=135
────────────────────────────────────────────────────────────────
{
  "mode": "dry_run",
  "plan": { "to_add": 12, "to_refresh": 5, "to_skip": 135, "total": 152 },
  "classifications": [
    { "filename": "report.pdf",  "status": "new",       "endpoint": "add_docs",     "reason": "No chunks on disk — first upload" },
    { "filename": "notes.txt",   "status": "changed",   "endpoint": "refresh_docs", "reason": "Content differs from existing chunks" },
    { "filename": "data.csv",    "status": "unchanged", "endpoint": "skip",         "reason": "Content matches existing chunks — skipping" }
  ]
}

---

run_match_static(query) — search_type="flat"

Full-corpus fusion search across ALL documents.

──────────────────────────────────────────────────────────────────────
  MATCHING DOCS — flat (all docs)
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "content": [
      {
        "file_name": "electrolytes-test-report-f_part1",
        "doc_score": 199.95,
        "category": "Very Good Match",
        "chunks": {
          "chunk": ["Yash M. Patel Age: 21 Years ... Sodium 110.00 Low ..."],
          "similarities": [2.0]
        }
      }
    ]
  },
  "message": "Data fetched successfully",
  "status_code": 200
}

---

run_match_filtered(query) — search_type="flat_filter"

Precision search — semantic ranking only runs over filter-passing documents.

Sample output: Identical structure to flat above. The difference is in the candidate pool — results are narrower and more precise.

---

run_match_dynamic(query) / run_match_dynamic_filtered(query)

Same as static search but over temporary files provided at query time — nothing needs to be pre-indexed.

---

run_namespace_details()

──────────────────────────────────────────────────────────────────────
  NAMESPACE DETAILS
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "content": [
      {
        "vector_lake_description": "customdataset_",
        "faiss_disk_size_mb": "14.43 mb",
        "source_store_files": 166,
        "source_store_used_mb": "30.9062 mb",
        "source_store_quota_remaining_mb": "12257.09 mb"
      }
    ]
  },
  "message": "Information retrieved successfully",
  "status_code": 200
}

---

run_docs_info()

──────────────────────────────────────────────────────────────────────
  DOCS INFORMATION
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "docs": [
      "001-hide-and-seek-free-chi",
      "002-ginger-the-giraffe-fre",
      "2._a_tale_of_two_cities_au",
      ...
    ]
  },
  "message": "Information retrieved successfully",
  "status_code": 200
}

---

📚 VectorLake Client API Reference

All public methods return a plain dict and never raise — errors are surfaced as {"success": False, "error": "...", "message": "..."}.

1. add_documents

Uploads new documents to the index. The server rejects files already present in the namespace — use sync_documents() for mixed folders.

Parameter	Type	Default	Description
user_id	str	required	User identifier (registration email).
vector_lake_description	str	required	Target namespace.
files_name + files_data	List[str]	—	Direct mode: names and string contents. Both required together, equal length.
files	List[str]	None	Batch mode: filenames to read from vector_lake_path. Defaults to all supported files.
start_from_batch	int	1	Resume point for batch mode.
end_batch	int	None	Upper batch bound (inclusive). None = process all.
intelligent_segmentation	bool	True	Enables server-side segmentation before embedding.
session_id	str	None	Optional session identifier.

---

2. refresh_documents

Updates existing documents in the index. Same parameters and modes as add_documents. The server rejects files that do not already exist. Use sync_documents() for mixed folders.

---

3. sync_documents

Intelligently syncs a folder by routing each file to the correct endpoint based on MD5 comparison against existing chunks on disk.

Parameter	Type	Default	Description
user_id	str	required	User identifier.
vector_lake_description	str	required	Target namespace.
files	List[str]	None	Files to sync. Defaults to all supported files in vector_lake_path.
dry_run	bool	False	If True, returns classification plan without uploading.
start_from_batch	int	1	Resume point.
end_batch	int	None	Upper batch bound.
intelligent_segmentation	bool	True	Server-side segmentation.
session_id	str	None	Optional session identifier.

---

4. get_matching_docs

Retrieves top-matching document chunks using semantic search with optional hybrid filtering.

Parameter	Type	Default	Description
query	str	required	Natural language or WaveQL search query.
user_id	str	required	User identifier.
vector_lake_description	str	required	Target namespace.
pattern	str	"static"	"static" for indexed docs; "dynamic" for temporary files.
hybrid_filter	bool	False	Enables keyword hybrid filter. Required when search_type is set.
search_type	str	None	"flat" — full corpus fusion. "flat_filter" — semantic search over filter-passing docs only. None — server default.
top_docs	int	10	Maximum chunks to return.
threshold	float	0.2	Similarity score cutoff.
with_data	bool	False	If True, includes raw chunk text in response.
files_name + files_data	List[str]	None	Dynamic mode: temporary files to search over.
session_id	str	None	Optional session identifier.

Validation rules:

• search_type must be "flat" or "flat_filter"; anything else returns InvalidSearchTypeError.
• hybrid_filter=True is required whenever search_type is specified.

---

5. health_check

Parameter	Type	Default	Description
user_id	str	required	User identifier.
vector_lake_description	str	required	Namespace to check.
session_id	str	None	Optional session identifier.

---

6. get_namespace_details

Returns storage and quota metadata for one or all namespaces belonging to the user.

Parameter	Type	Default	Description
user_id	str	required	User identifier.
vector_lake_description	str	None	Scopes response to that namespace if supplied.
session_id	str	None	Optional session identifier.

---

7. get_docs_information

Returns document-level metadata within a namespace, optionally filtered by keyword.

Parameter	Type	Default	Description
user_id	str	required	User identifier.
vector_lake_description	str	required	Namespace to query.
keyword	str	None	Optional filter keyword.
threshold	int	70	Keyword-match threshold.
session_id	str	None	Optional session identifier.

---

8. full_corpus_search

Full-text keyword search across all documents in a namespace.

Parameter	Type	Default	Description
user_id	str	required	User identifier.
vector_lake_description	str	required	Namespace to search.
keyword	str	required	Search term.
top_docs	int	10	Maximum results to return.
session_id	str	None	Optional session identifier.

---

Error Handling

All exceptions inherit from VectorLakeError and expose .to_response(). Since all public methods catch and convert exceptions internally, you only ever need to check the "error" key in the returned dict.

Exception	When raised
ConfigError	API key missing or config invalid at init time.
ValidationError	Mismatched files_name/files_data lengths or other pre-flight failures.
InvalidSearchTypeError	search_type is not "flat" or "flat_filter".
UnsupportedFileTypeError	File extension not in allowed_extensions.
FileProcessingError	File I/O, encoding, or parsing failure.
APIError	HTTP 4xx/5xx from server. Carries .status_code and .response_text.
ThrottleError	HTTP 429 rate limit. Carries .retry_after.

---

Structured Logging

The Logger class writes three CSV files to log_dir (default: logs/):

File	Contents
performance.csv	Per-request latency, payload/response KB, HTTP status
api_errors.csv	Operation, batch number, error message
skipped_files.csv	Filename and reason when a file is skipped

---

Retry and Backoff

  Request attempt 1
      │
      ├── HTTP 429     → wait Retry-After header (or 2^attempt s) → retry
      ├── Timeout      → wait 2^attempt s → retry
      ├── ConnError    → wait 2^attempt s → retry
      ├── HTTP 4xx/5xx → return error dict immediately (no retry)
      └── Success      → return response dict

Max retries: 2 (configurable via VECTOR_LAKE_MAX_RETRIES).

---

Using WaveQL (VQL) Queries

WaveQL enables natural language filtering using brace-based logical groups.

Syntax at a Glance

  ┌──────────────────────────────────────────────────────────────────┐
  │  {(clinical trials) or (observational studies)} {diabetes}       │
  │   └──────────────── group 1 ─────────────────┘  └── group 2 ──┘ │
  │                                                                  │
  │  Groups combine with implicit AND:                               │
  │  → (clinical trials OR observational studies) AND diabetes       │
  └──────────────────────────────────────────────────────────────────┘

  Within a group:
  ┌──────────────────────────────────────────────────────────┐
  │  {A and B}       → both A and B must match               │
  │  {A or B}        → either A or B                         │
  │  {(A B) or C}    → phrase "A B", or term C               │
  │  {A B}           → implicit AND: A and B                 │
  └──────────────────────────────────────────────────────────┘

  ✅ Correct                        ❌ Wrong
  ──────────────────────────────    ─────────────────────────────
  {(machine learning) or            {machine learning or
   (deep learning)}                  deep learning}
                                      → "machine" treated as
                                        a separate term

  {(product manager) or             {product manager or Delhi}
   (data scientist)}                  → ambiguous parse

Three-Tier Result Ranking

  ┌─────────────────────────────────────────────────────────────┐
  │             RESULT TIERS (highest → lowest)                 │
  ├──────────┬──────────────────────────────────────────────────┤
  │  Tier 1  │  ✓ Filter match  +  ✓ Semantic match             │
  │          │  Highest confidence — structure & meaning align  │
  ├──────────┼──────────────────────────────────────────────────┤
  │  Tier 2  │  ✓ Filter match  +  ✗ Semantic match             │
  │          │  Structured match, lower semantic relevance      │
  ├──────────┼──────────────────────────────────────────────────┤
  │  Tier 3  │  ✗ Filter match  +  ✓ Semantic match             │
  │          │  Meaning-based fallback, no filter alignment     │
  └──────────┴──────────────────────────────────────────────────┘

Query Examples by Domain

Healthcare:

Select Top 10 
Where QUERY IS "Need detail about disease state progression of patient id 555"
Contains {diabetes} {(clinical trial)} {PID 555}

Recruitment:

Select Top 30 
Where QUERY IS "Need list of resources who have good experience in python and machine learning"
Contains {Python} {(machine learning)} {Delhi}

Filter Design Best Practices

DO: Use 1–2 keywords per brace · Wrap multi-word phrases in parentheses with operators · Keep filters domain-consistent · Trust semantic fallback for edge cases

DON'T: Use 5+ word phrases · Mix unrelated domains ({resume} {clinical trials}) · Forget parentheses around multi-word phrases with OR/AND · Over-specify filters

---

🎯 No Schema Required

  Traditional approach                  WaveflowDB approach
  ────────────────────                  ───────────────────
  1. Define JSON schema           vs.   1. Upload raw files
  2. Extract & map every field             (PDF, txt, docx, csv…)
  3. Maintain schema consistency
  4. Update schema for new fields   →   2. Query immediately
  5. Re-index on schema change             with WaveQL

Feature	Traditional	WaveflowDB
Data Ingestion	Extract, map, validate	Direct upload
Schema Definition	Required upfront	Not required
Query Capability	Exact field matching	Semantic + logical filtering
New Document Types	Requires schema update	Works immediately
Maintenance	High	Low

---

📧 Support

For API or platform support, visit: https://db.agentanalytics.ai

---

📄 License

Copyright DIBR tech private ltd.