d-vecDB Server Python Package

Zero-configuration vector database server with embedded binaries - Production-ready with WAL corruption protection, GPU acceleration (10-50x speedup), and SIMD optimization (2-3x speedup).

A complete Python package that provides the d-vecDB server with embedded pre-built binaries for all major platforms. Run a high-performance vector database server directly from Python with a single pip install command - no Rust compiler needed!

🚀 Key Features

✅ Zero Configuration - Pre-built binaries included for Linux, macOS (Intel/ARM), Windows ✅ One-Line Install - pip install d-vecdb-server and you're ready to go ✅ Production Ready - WAL corruption protection with CRC32 checksumming ✅ Hardware Accelerated - GPU support (10-50x speedup) with automatic CPU fallback ✅ SIMD Optimized - AVX2/SSE2 for 2-3x faster distance calculations ✅ Auto Platform Detection - Automatically uses the correct binary for your OS ✅ Python Integration - Start/stop server from Python code or command line ✅ Complete Package - Includes d-vecdb Python client as a dependency

Prerequisites

• Python: 3.8 or higher
• Operating System: Linux (x86_64), macOS (Intel/Apple Silicon), or Windows (x86_64)
• Memory: Minimum 512MB RAM available
• Disk Space: At least 100MB for binaries and data storage
• Network: Available ports for REST API (default: 8080) and gRPC (default: 9090)

Installation Options

Option 1: Standard Installation (Recommended)

# Install the server package
pip install d-vecdb-server

# Verify installation
d-vecdb-server version

Note: The package includes embedded binaries and is ready to use immediately after installation.

Option 2: Development Installation

# Clone the repository
git clone https://github.com/rdmurugan/d-vecDB.git
cd d-vecDB/d-vecdb-server-python

# Install in development mode
pip install -e .

# Verify installation
d-vecdb-server version

Option 3: Virtual Environment Installation (Recommended for Development)

# Create virtual environment
python -m venv d-vecdb-env

# Activate virtual environment
# On Linux/macOS:
source d-vecdb-env/bin/activate
# On Windows:
d-vecdb-env\Scripts\activate

# Install package
pip install d-vecdb-server

# Verify installation
d-vecdb-server version

Option 4: Install with Python Client

# Install server with Python client for complete functionality  
pip install 'd-vecdb-server[client]'

# Or install separately
pip install d-vecdb-server
pip install d-vecdb  # Python client library

Quick Start

Command Line Usage

# Show version (includes binary status)
d-vecdb-server version

# Start the server (foreground)
d-vecdb-server start

# Start in background
d-vecdb-server start --daemon

# Start with custom settings
d-vecdb-server start --host 0.0.0.0 --port 8081 --data-dir ./my-data

# Stop the server
d-vecdb-server stop

# Check server status
d-vecdb-server status

Python API

from d_vecdb_server import DVecDBServer

# Create and start server
server = DVecDBServer(
    host="127.0.0.1",
    port=8080,
    data_dir="./vector-data"
)

# Start the server
server.start()

print(f"Server running: {server.is_running()}")
print(f"REST API: http://{server.host}:{server.port}")
print(f"gRPC API: {server.host}:{server.grpc_port}")

# Stop the server
server.stop()

Context Manager

from d_vecdb_server import DVecDBServer

# Automatically start and stop server
with DVecDBServer(port=8080) as server:
    print(f"Server is running on port {server.port}")
    # Server will be automatically stopped when exiting the context

Configuration

Step-by-Step Server Configuration

Step 1: Choose Your Configuration Method

You can configure the server in three ways:

1. Command line arguments (quick setup)
2. Python API parameters (programmatic setup)
3. Configuration file (persistent setup)

Step 2: Basic Configuration

Default Configuration:

• Host: 127.0.0.1 (localhost only)
• REST Port: 8080
• gRPC Port: 9090
• Data Directory: Temporary directory (auto-generated)
• Log Level: info

Step 3: Create Data Directory (Optional)

# Create persistent data directory
mkdir -p /path/to/your/vector-data
chmod 755 /path/to/your/vector-data

Step 4: Configuration Options

Option A: Command Line Configuration

# Basic configuration
d-vecdb-server start --host 0.0.0.0 --port 8081 --data-dir ./data

# Advanced configuration
d-vecdb-server start \
  --host 0.0.0.0 \
  --port 8081 \
  --grpc-port 9091 \
  --data-dir /path/to/data \
  --log-level debug \
  --daemon

Option B: Python API Configuration

from d_vecdb_server import DVecDBServer

# Basic configuration
server = DVecDBServer(
    host="0.0.0.0",           # Listen on all interfaces
    port=8081,                # Custom REST port
    grpc_port=9091,           # Custom gRPC port
    data_dir="/path/to/data", # Persistent data directory
    log_level="debug"         # Verbose logging
)

server.start()

Option C: Configuration File Setup

1. Create a configuration file:

# Create config directory
mkdir -p ~/.config/d-vecdb

# Create configuration file
cat > ~/.config/d-vecdb/server.toml << EOF
[server]
host = "0.0.0.0"
port = 8080
grpc_port = 9090
workers = 8

[storage]
data_dir = "/path/to/your/data"
wal_sync_interval = "1s"
memory_map_size = "1GB"

[index]
hnsw_max_connections = 32
hnsw_ef_construction = 400
hnsw_max_layer = 16

[monitoring]
enable_metrics = true
prometheus_port = 9091
log_level = "info"
EOF

2. Use the configuration file:

# Start with configuration file
d-vecdb-server start --config ~/.config/d-vecdb/server.toml

Step 5: Verify Configuration

# Check server status
d-vecdb-server status

# Test REST API endpoint
curl http://localhost:8080/health

# Check metrics (if enabled)
curl http://localhost:9091/metrics

Step 6: Production Configuration Tips

# Production-ready configuration example
[server]
host = "0.0.0.0"          # Accept connections from any IP
port = 8080               # Standard HTTP port
grpc_port = 9090          # Standard gRPC port
workers = 16              # Match CPU cores

[storage]
data_dir = "/var/lib/d-vecdb"  # Persistent storage location
wal_sync_interval = "5s"       # Longer interval for better performance
memory_map_size = "8GB"        # More memory for larger datasets

[index]
hnsw_max_connections = 64      # Higher connections for better recall
hnsw_ef_construction = 800     # Higher construction for better quality
hnsw_max_layer = 16           # Default is usually fine

[monitoring]
enable_metrics = true
prometheus_port = 9091
log_level = "warn"            # Less verbose for production

API Reference

DVecDBServer Class

Constructor

DVecDBServer(
    host: str = "127.0.0.1",
    port: int = 8080,
    grpc_port: int = 9090,
    data_dir: Optional[str] = None,
    log_level: str = "info",
    config_file: Optional[str] = None
)

Methods

• start(background: bool = True, timeout: int = 30) -> bool

  • Start the server process
  • Returns True if successful

• stop(timeout: int = 10) -> bool

  • Stop the server process
  • Returns True if successful

• restart(timeout: int = 30) -> bool

  • Restart the server
  • Returns True if successful

• is_running() -> bool

  • Check if server is running
  • Returns True if running

• get_status() -> Dict[str, Any]

  • Get detailed server status
  • Returns status dictionary

Command Line Interface

d-vecdb-server [OPTIONS] COMMAND

Commands:
  start    Start the server
  stop     Stop the server  
  status   Check server status
  version  Show version information

Options:
  --host HOST           Server host (default: 127.0.0.1)
  --port PORT           REST API port (default: 8080)
  --grpc-port PORT      gRPC port (default: 9090)
  --data-dir DIR        Data directory
  --config FILE         Configuration file
  --log-level LEVEL     Log level (debug/info/warn/error)

Start Options:
  --daemon              Run in background

📊 Performance

Production Benchmarks (October 2025)

Tested on DigitalOcean 2 vCPU, 2GB RAM

Batch Size	d-vecDB	Qdrant	Status
Single (1)	315 vec/s	275 vec/s	✅ 15% FASTER
Small (10)	1,293 vec/s	1,628 vec/s	1.26x slower
Medium (100)	2,027 vec/s	3,720 vec/s	1.84x slower
Large (500)	2,262 vec/s	4,244 vec/s	1.88x slower

Key Achievement: d-vecDB beats Qdrant on single insert throughput! 🏆

Performance Features

• Distance Calculations: 35M+ operations/second
• Vector Search: 13K+ queries/second
• Sub-microsecond latency for vector operations
• 6.7x total improvement from baseline (CPU SIMD + GPU + Lock-Free HNSW)

Hardware Acceleration

• GPU Acceleration: 10-50x speedup with CUDA/Metal/Vulkan
• SIMD Optimization: 2-3x speedup with AVX2/SSE2
• Automatic Detection: Falls back to CPU if GPU unavailable
• Lock-Free HNSW: Optimized index for concurrent operations

Production Features

• WAL Corruption Protection: CRC32 checksumming for all entries
• Magic Number Boundaries: Detect corruption and partial writes
• Graceful Recovery: Automatic crash recovery
• Production Durability: Enterprise-grade data protection

Platform Support

• Linux: x86_64 (with musl for better compatibility)
• macOS: Intel (x86_64) and Apple Silicon (ARM64)
• Windows: x86_64

All platform binaries include full feature support (GPU, SIMD, WAL protection).

Advanced Setup

Environment Variables

You can also configure the server using environment variables:

# Set environment variables
export DVECDB_HOST="0.0.0.0"
export DVECDB_PORT="8080"
export DVECDB_GRPC_PORT="9090"
export DVECDB_DATA_DIR="/var/lib/d-vecdb"
export DVECDB_LOG_LEVEL="info"
export RUST_LOG="info"

# Start server (will use environment variables)
d-vecdb-server start

Docker Setup (Alternative)

If you prefer Docker deployment:

# Pull the Docker image
docker pull rdmurugan/d-vecdb:latest

# Run with custom configuration
docker run -d \
  --name d-vecdb-server \
  -p 8080:8080 \
  -p 9090:9090 \
  -v /path/to/data:/data \
  rdmurugan/d-vecdb:latest

Service Configuration (Linux)

Create a systemd service for automatic startup:

# Create service file
sudo cat > /etc/systemd/system/d-vecdb.service << EOF
[Unit]
Description=d-vecDB Vector Database Server
After=network.target

[Service]
Type=simple
User=d-vecdb
Group=d-vecdb
WorkingDirectory=/var/lib/d-vecdb
ExecStart=/usr/local/bin/d-vecdb-server start --config /etc/d-vecdb/server.toml
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

# Enable and start service
sudo systemctl enable d-vecdb
sudo systemctl start d-vecdb
sudo systemctl status d-vecdb

Troubleshooting

Installation Issues

Binary Not Found (should not happen with pip install):

# Check if binary is available
d-vecdb-server version

# If you still get binary not found errors:
# 1. Reinstall the package
pip uninstall d-vecdb-server
pip install d-vecdb-server

# 2. For development installations:
pip install -e . --force-reinstall

Permission Denied:

# Install in user directory
pip install --user d-vecdb-server

# Or use virtual environment
python -m venv venv && source venv/bin/activate
pip install d-vecdb-server

Runtime Issues

Port Already in Use:

# Check what's using the port
lsof -i :8080  # On Linux/macOS
netstat -ano | findstr :8080  # On Windows

# Use different port
d-vecdb-server start --port 8081

Server Won't Start:

# Check server logs
d-vecdb-server start  # Run in foreground to see errors

# Check disk space
df -h  # On Linux/macOS

# Check permissions on data directory
ls -la /path/to/data/directory

Connection Refused:

# Verify server is running
d-vecdb-server status

# Check if ports are accessible
telnet localhost 8080

# For remote connections, ensure host is set to 0.0.0.0
d-vecdb-server start --host 0.0.0.0

Performance Issues

Slow Performance:

# Optimize configuration for better performance
[storage]
memory_map_size = "4GB"  # Increase based on available RAM

[index]
hnsw_max_connections = 64
hnsw_ef_construction = 800

[server]
workers = 16  # Match your CPU cores

High Memory Usage:

# Reduce memory usage
[storage]
memory_map_size = "512MB"  # Reduce if needed

[index]
hnsw_max_connections = 16
hnsw_ef_construction = 200

License

This project is licensed under the MIT License - see the LICENSE file for details.

Using with Python Client

After installing the server, you can use it with the Python client:

# Install the Python client
pip install d-vecdb

from d_vecdb_server import DVecDBServer
from d_vecdb import VectorDBClient
import numpy as np

# Start the server
with DVecDBServer(port=8080) as server:
    # Connect client
    client = VectorDBClient(host=server.host, port=server.port)
    
    # Create collection
    client.create_collection_simple("documents", 128, "cosine")
    
    # Insert vectors
    vector = np.random.random(128)
    client.insert_simple("documents", "doc1", vector)
    
    # Search
    query = np.random.random(128)
    results = client.search_simple("documents", query, limit=5)
    print(f"Found {len(results)} similar vectors")

Next Steps

After installation and configuration:

1. Start using the REST API: Visit http://localhost:8080/docs for API documentation
2. Use Python client: See example above for Python integration
3. Check examples: See the main repository for usage examples
4. Join community: Report issues and get support

🔗 Links

• PyPI Package (Server): https://pypi.org/project/d-vecdb-server/
• PyPI Package (Client): https://pypi.org/project/d-vecdb/
• GitHub Repository: https://github.com/rdmurugan/d-vecDB
• Binary Releases: https://github.com/rdmurugan/d-vecDB/releases
• Docker Hub: https://hub.docker.com/r/rdmurugan/d-vecdb
• Documentation: https://github.com/rdmurugan/d-vecDB#readme

🆘 Support

• GitHub Issues: Report bugs or request features
• GitHub Discussions: Community support and Q&A
• Email: durai@infinidatum.com
• Commercial Support: Available for enterprise customers

🤝 Related Packages

• d-vecdb-server - Complete server package with embedded binaries (this package)
• d-vecdb - Python client library (included as dependency)

📈 Version History

See CHANGELOG for version history and release notes.

Current Version: 0.1.7

• ✅ Published on PyPI
• ✅ Pre-built binaries for all platforms
• ✅ Zero configuration required
• ✅ Production-ready with WAL protection
• ✅ GPU acceleration and SIMD optimization
• ✅ Comprehensive documentation

---

Built with ❤️ by the d-vecDB team

Star us on GitHub ⭐