meshpop 4.8.0

MeshPOP Network Operations CLI - Multi-server mesh monitoring and management

mpop - MeshPOP Network Operations CLI

Version: 4.10.0 Last Updated: 2026-03-09

Swiss Army knife for server management + AI

A single CLI tool for managing distributed server mesh networks. Integrates VPN, security audits, secret management, AI analysis, auto-healing, project memory, and server agent monitoring.

---

Quick Start

mpop                          # Dashboard (all servers)
mpop advise --ai              # AI infrastructure recommendations
mpop deploy mpop              # Update all servers (2 sec)
mpop setup g5 -x              # One-click new server install
mpop memory ask mpop "query"  # AI answers without hallucination
mpop heal -x                  # Auto problem fix

---

Pros & Cons

Pros

Single CLI	Manage 10+ nodes with one command
vssh	Ultra-fast VPN internal comms (no encryption overhead)
AI Built-in	Problem prediction, auto-diagnosis, natural language
MCP	Server management from Claude Desktop
Auto Deploy	mpop deploy mpop updates all servers
Secrets	Level 3/4 encryption, machine-bound
No Hallucination	Project memory for evidence-based AI answers

Cons

Dependency	Requires VPN (wire/tailscale)
Single Admin	No multi-user/RBAC
CLI Only	No web UI

---

Replaces These Tools

┌─────────────────────────────────────────────────┐
│                    mpop                         │
├─────────────────────────────────────────────────┤
│  n8n        → workflow, webhook                 │
│  LangChain  → chain, ask                        │
│  Ansible    → exec, deploy, setup               │
│  Vault      → secret (L3/L4)                    │
│  Prometheus → watch, trend                      │
│  Grafana    → ai, report                        │
│  RAG        → rag, memory                       │
│  LangMem    → memory (2-Agent, no hallucination)│
├─────────────────────────────────────────────────┤
│  Single file 650KB · Python 3.8+ · Zero deps    │
└─────────────────────────────────────────────────┘

Tool	mpop Replacement
n8n	mpop workflow, mpop webhook - Workflow automation
LangChain	mpop chain, mpop ask - AI chaining, natural language
Ansible	mpop exec, mpop deploy, mpop setup - Remote execution
HashiCorp Vault	mpop secret - L3/L4 encryption, machine-bound
Prometheus	mpop watch, mpop trend - Real-time monitoring
Grafana	mpop ai, mpop report, mpop export - AI analysis
RAG Server	mpop rag - Document-based Q&A
LangChain Memory	mpop memory - Project memory, 2-Agent system

Why mpop memory is powerful:

• Standard RAG: Document embedding → similarity search → answer (may hallucinate)
• mpop memory: Code parsing → function/file/TODO extraction → 2-Agent (Historian + Advisor) → Evidence-based answers only

---

Table of Contents

1. Installation
2. Quick Start
3. All Commands
4. Features
   • Monitoring
   • Security
   • Secret Management
   • AI Features
   • Project Memory
   • Deploy/Setup
   • Network
5. MCP Integration
6. Server List
7. Architecture
8. What's New

---

Installation

# Local install
sudo cp mpop /usr/local/bin/mpop
sudo chmod +x /usr/local/bin/mpop

# Deploy to all servers
mpop deploy mpop

# One-click new server setup (mpop + vssh + agent)
mpop setup <server> -x

# Shell completion (optional)
mpop completion zsh --install

---

Quick Start

# 1. Check all servers
mpop

# 2. Auto-fix problems
mpop heal -x

# 3. Security audit
mpop audit

# 4. AI report
mpop full

---

All Commands (70+)

Monitoring

mpop                  # Dashboard (all servers)
mpop full             # Full report (9 tables + AI)
mpop watch            # Real-time monitoring
mpop watch 3 --alert  # 3-second interval, anomaly detection
mpop matrix           # Server connectivity matrix
mpop services         # Service status
mpop temp             # CPU/GPU temperature
mpop gpu              # GPU usage
mpop info <node>      # Server details (agent API: GPU, logs, security, processes)
mpop info <node> --raw # Raw JSON output (for debugging)
mpop logs <node>      # Log streaming
mpop trend            # 24-hour trend
mpop map              # Service topology

Security

mpop security         # Security overview
mpop audit            # Deep security audit
mpop audit <node>     # Audit specific server
mpop ssh-attacks      # SSH attack analysis
mpop dragon           # Auto-ban analysis
mpop dragon -x        # Execute auto-ban

Secrets (Level 3/4)

mpop secret list              # List secrets
mpop secret set KEY           # Store secret
mpop secret get KEY           # Retrieve (masked)
mpop secret get KEY --clip    # Copy to clipboard (auto-clear 30s)
mpop secret inject KEY        # Inject as environment variable
mpop secret unlock 'pass'     # Unlock Level 4
mpop secret machines          # List registered machines
mpop secret export            # Encrypted backup

AI Features

mpop report           # AI status report
mpop report --gpt4    # Use GPT-4
mpop ask "query"      # Natural language → command (dry-run)
mpop ask "query" -x   # Execute after conversion
mpop ask "query" -x -y # Execute without confirmation
mpop predict          # Problem prediction
mpop heal             # Auto-diagnosis (dry-run)
mpop heal -x          # Auto-fix
mpop logai            # AI-powered log analysis
mpop chain analyze    # Multi-step AI analysis
mpop ai               # AI trend analysis

mpop ask - Pattern Matching (instant, no AI):

mpop ask "memory" -x              # Pattern → exec 'free -m'
mpop ask "docker restart nginx"   # Pattern → exec 'docker restart nginx'
mpop ask "service restart nginx"  # Pattern → exec 'systemctl restart nginx'

Patterns defined in ~/.mpop/patterns.yaml:

_aliases:           # Korean → English (16 aliases)
  메모리: memory
  재시작: restart

memory: exec 'free -m'              # Simple pattern
docker restart: exec 'docker restart {1}'  # With argument
service restart: exec 'systemctl restart {1}'

mpop ask - AI fallback (for complex queries):

mpop ask "gpu1 memory usage" -x       # → AI generates: mpop exec -t gpu1 'free -m'
mpop ask "all servers disk"           # → mpop exec 'df -h /'
mpop ask "web1,web2,web3 uptime" -x   # → Multi-server execution
mpop ask "restart nginx on web1" -x   # → mpop restart nginx -t web1

AI Agent Management

# Agent CRUD
mpop agent                              # List all agents
mpop agent make <n> -d "desc" -b "behavior"  # Create with auto-validation
mpop agent show <name>                  # Show agent details + code
mpop agent run <name>                   # Run locally
mpop agent deploy <name> web1 web2      # Deploy to servers via SCP
mpop agent delete <name>                # Delete agent

# Workflows (sequential execution)
mpop agent workflow                     # List saved workflows
mpop agent workflow create <n> <a1> <a2>...  # Create workflow
mpop agent workflow run <name>          # Run workflow
mpop agent workflow delete <name>       # Delete workflow

# Chains (output -> input via JSON)
mpop agent chain                        # List saved chains
mpop agent chain <a1> <a2>              # Run chain directly
mpop agent chain save <n> <a1> <a2>...  # Save chain
mpop agent chain run <name>             # Run saved chain
mpop agent chain delete <name>          # Delete chain

Agent Creation Flow:

1. mpop agent make <name> -d "desc" -b "behavior"
2. AI generates Python code
3. Auto syntax check + test run
4. If error → AI retries with feedback (max 2x)
5. Save only if validation passes

Built-in Agent Functions:

SERVERS = ['web1', 'gpu1', ...]     # Server list from config
mpop_exec(server, cmd) -> str       # Run command on server
run_cmd(cmd) -> str                 # Run local command
get_input() -> dict                 # JSON from previous agent (chain)
output(data) -> None                # JSON output for next agent

Infrastructure Advisor

mpop advise                   # Full analysis (agent cache - fast)
mpop advise --ai              # AI recommendations (local Ollama)
mpop advise -t web1           # Single server analysis
mpop advise --live            # Real-time monitoring
mpop advise --fresh           # Direct query (slow but latest)
mpop advise --hardware        # Hardware analysis
mpop advise --software        # Software versions
mpop advise --server          # Server status
mpop advise --json            # JSON for AI/MCP
mpop hw                       # Local machine hardware

Features:

• Default: uses agent cache (instant response)
• --ai: local LLM analyzes data and recommends
• --json: for MCP/external AI integration

Project Memory (No Hallucination)

mpop memory                       # Help
mpop memory scan <path>           # Scan project
mpop memory list                  # List stored memories
mpop memory ask <proj> "query"    # 2-Agent query
mpop memory status <proj>         # Check status
mpop memory delete <proj>         # Delete memory

2-Agent System:

• Historian: Past/retrospective - answers only with facts from code
• Advisor: Future/design - suggests refactoring, improvements

RAG (Document-based Q&A)

# Indexing
mpop rag index ~/docs --embed           # Index local documents
mpop rag index gpu1:/home/user/docs     # Index remote server docs
mpop rag index <path> --replace         # Replace existing index
mpop rag update                         # Reindex changed docs only

# Watch & Sync
mpop rag watch ~/MeshPOP/docs           # Watch for file changes (auto-reindex)
mpop rag sync                           # Sync remote server 24h changes

# Search & Q&A
mpop rag search "nginx config"          # Search documents
mpop rag search "function" --server=gpu1  # Filter by server
mpop rag search "config" --type=code    # Filter by type (docs/config/code)
mpop rag ask "How to configure GPU?"    # Document-based Q&A (120b LLM)

# Management
mpop rag status                         # Index status (server/type stats)
mpop rag models                         # Available embedding models
mpop rag delete <pattern>               # Delete matching documents
mpop rag clear                          # Clear index

Features:

• Remote indexing - server:/path format for remote server documents
• Code chunking - Python AST for function/class-level splitting
• Metadata tagging - Auto-detect server, project, type filtering
• Real-time indexing - watch command for auto-reindex on file change
• Multilingual embeddings - multilingual-e5-large-instruct (100+ languages)
• LLM failover - Auto-detect multiple servers (gpt-oss:120b preferred)

Deploy/Setup

mpop deploy mpop              # Deploy mpop to all servers
mpop deploy <file> <nodes>    # Deploy file
mpop setup <srv> -x           # Full setup (mpop+vssh+agent)
mpop setup all -x             # Setup all servers
mpop setup vssh <srv> -x      # Install vssh only
mpop setup agent <srv> -x     # Install agent only
mpop self-update              # Update local mpop

Network

mpop vpn              # VPN status
mpop peers            # VPN peer list
mpop network          # SSH connection info
mpop dns              # MagicDNS status
mpop acl              # Access control
mpop tunnel           # Expose internal services externally
mpop tunnel add web1:8080 --domain api.example.com --ssl

Execution/Management

mpop exec <cmd>              # Execute on all servers
mpop exec -t web1 <cmd>      # Execute on specific server
mpop restart <svc>           # Restart service
mpop diff web1 web2          # Compare servers
mpop backup create           # Create backup
mpop export                  # HTML report

AI-OS & News

mpop brain            # Shared AI context
mpop project          # Project management
mpop scan             # Full scan
mpop doc              # Document lookup
mpop write            # AI writing
mpop voice            # Server status audio briefing (Edge TTS + local LLM)
mpop voice web1       # Detailed report for specific server
mpop voice me         # Desktop/laptop status report
mpop voice --ko       # Korean briefing
mpop news             # RSS news briefing (tech)
mpop news korea       # Korea news
mpop news world       # World news
mpop news dev         # Developer news

New Commands (v4.10)

mpop service                # Cluster service management
mpop service status myapp   # Check service status on all nodes
mpop service start myapp    # Start service cluster-wide
mpop service logs myapp     # Aggregate logs from all nodes

mpop retry                  # Show recent failed commands
mpop retry last             # Retry last failed command
mpop retry 5                # Retry command #5

mpop logai                  # AI-powered log analysis
mpop logai web1             # Analyze specific server logs
mpop logai --errors         # Focus on errors only
mpop logai --fix            # Suggest fixes with AI

---

Features

Monitoring

# Main dashboard - all server status at a glance
mpop

# Real-time monitoring (3-second interval, anomaly detection)
mpop watch 3 --alert

# Server connectivity matrix
mpop matrix

Server Agent (v3.3)

Real-time server monitoring agent. Reports to dashboard every 30 seconds.

# Full setup (vssh + agent) - recommended
mpop setup full all -x

# Individual setup
mpop setup agent web1 -x
mpop setup vssh gpu1 -x

# Dashboard access (VPN internal)
http://<dashboard-server>:8800/

Collected Metrics:

Category	Items
System	CPU, Memory, Disk, Swap, Load, Uptime
Network	Connections (ESTAB/LISTEN/TIME_WAIT), Traffic, Errors
Processes	Top 10 (CPU/MEM), Zombie processes
Services	Running services (auto-detect)
Web	nginx, node, python, gunicorn, etc.
Containers	Docker status + health check
Log Anomalies	errors, warnings, OOM, segfault, auth failures
Security	SSH sessions, failed logins, attacker IPs, fail2ban
DB	PostgreSQL/Redis/MySQL connections
Mail	System mail, cron errors
Updates	Security patches needed
GPU	nvidia-smi (auto-detected)

Security

# Full security audit
mpop audit

# Audit items:
# - SSH brute-force attacks (24h)
# - Top attacker IPs
# - fail2ban status
# - Open port risk analysis
# - Firewall status
# - AI recommendations

Secret Management

VPN + machine-bound encryption for secure secret storage.

# Basic usage
mpop secret set openai          # Interactive input
mpop secret get openai --clip   # Clipboard (auto-clear 30s)

# For scripts
export OPENAI_API_KEY=$(mpop secret get openai --raw --force)

# Level 4 (master password)
mpop secret level4 setup 'MyPass'
mpop secret unlock 'MyPass'     # Unlock for 5 minutes

AI Features

# Natural language server management
mpop ask "check web1 disk usage"
mpop ask "restart nginx" -x

# Problem prediction
mpop predict   # "web2 disk full in 3 days"

# Auto-recovery
mpop heal      # Scan for problems (dry-run)
mpop heal -x   # Execute auto-fix

Voice News (Edge TTS + Local LLM)

Audio briefing of server/desktop status using local LLM and Edge TTS.

┌─────────────────────────────────────────────────────┐
│  1. Collect server metrics (SSH/Agent)             │
│  2. Generate script with local LLM (Ollama)        │
│  3. Convert to speech (Microsoft Edge TTS)         │
│  4. Play audio locally                             │
└─────────────────────────────────────────────────────┘

# Server status briefing (network overview)
mpop voice

# Specific server detailed report
mpop voice web1

# Desktop/laptop status
mpop voice me           # Today's activity
mpop voice me week      # Weekly summary
mpop voice me month     # Monthly summary

# Options
mpop voice --ko         # Korean briefing
mpop voice --no-play    # Generate only, don't play
mpop voice --save out.mp3

Requirements:

• Ollama running on configured LLM host
• Internet access for Edge TTS

Project Memory (No AI Hallucination)

AI answers based on code evidence only, no guessing.

# 1. Scan project
mpop memory scan ~/MeshPOP/mpop

# 2. Query (2-Agent system)
mpop memory ask mpop "Which LLM models are used?"

# Response:
# A) Historian (facts):
#    - ClaudeBackend defined in brain/claude.py
#    - OpenAIBackend defined in brain/openai.py
#
# B) Advisor (suggestions):
#    - Option A: Claude pros/cons...
#    - Option B: OpenAI pros/cons...

Deploy/Setup

# Deploy mpop to all servers (2 seconds via vssh)
mpop deploy mpop

# One-click new server setup
mpop setup g5 -x   # Install mpop + vssh + server-agent

# Individual setup
mpop setup vssh g5 -x
mpop setup agent g5 -x

Network

# VPN status (wire + tailscale)
mpop vpn

# Peer list
mpop peers

# SSH connection info
mpop network

# Server connectivity matrix
mpop matrix

VPN Routing (wire)

Architecture:

• Relay nodes (public IP): Direct connection
• NAT nodes: Via relay (/16)

Important: NAT nodes must NOT add other relay nodes as /32 direct peers. client.py checks is_relay so only relay nodes connect directly.

# NAT nodes skip this block - route only via relay (/16)
if self.is_relay:  # Relay nodes only
    for candidate in RELAY_CANDIDATES:
        self.wg.add_peer(..., f"{vpn_ip}/32")

Troubleshooting:

# Check VPN peers
sudo wg show awlite0

# Problem if NAT node has /32 peers
# Should only have relay (/16)

---

MCP Integration

Use mpop directly from Claude Desktop or Cursor. 58 tools in 12 categories.

Config (~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "mpop": {
      "command": "python3",
      "args": ["/path/to/mpop/mpop-mcp-server.py"]
    }
  }
}

MCP Tools (58):

Category	Tools	Description
[Overview]	dashboard, servers, info, full, hw	Server status at a glance
[Monitor]	temp, gpu, services, logs, watch, trend, matrix, diff	Detailed monitoring
[Security]	security, audit, ssh-attacks	Security audits
[Network]	vpn, peers, watchdog	VPN & connectivity
[Config]	config, setup, deploy, backup, secret	Configuration & deployment
[Execute]	exec, delete, raw, python, query_help	Run commands safely
[FileOps]	read_file, write_file, chunk_write, scp	File operations
[AI-Diag]	ai, report, heal, predict, advise, roles, chain	AI analysis
[Agent]	agent_list, agent_make, agent_deploy, agent_run, agent_workflow, agent_chain	Custom automation
[Knowledge]	brain, project, memory, rag	Project memory & RAG
[Integration]	notify, webhook, workflow	Notifications & automation
[Media]	tts, voice, news, export	Voice & reports
[Ops]	service, retry, logai	Operations & recovery
[Router]	ask	Natural language → command

All tools use mpop_ prefix (e.g., mpop_dashboard, mpop_exec).

Usage Examples:

• "Check web1 server status" → calls mpop_info
• "Run full security audit" → calls mpop_audit
• "Tech news briefing" → calls mpop_news
• "Show config structure from mpop memory" → calls mpop_memory
• "Find logs on web1" → calls mpop_exec (shell=true for pipe support)
• "Run complex Python script" → calls mpop_python (saves file then executes)

---

Server Configuration

Servers are defined in ~/.mpop/config.json:

{
  "servers": {
    "relay1": {"ip": "10.99.x.x", "user": "root", "desc": "Primary VPN", "type": "relay"},
    "relay2": {"ip": "10.99.y.y", "user": "root", "desc": "Secondary VPN", "type": "relay"},
    "gpu1": {"ip": "10.99.z.z", "user": "root", "desc": "GPU Server", "type": "gpu"},
    "app1": {"ip": "10.99.w.w", "user": "root", "desc": "App Server", "type": "app"}
  }
}

Use mpop servers to list all configured servers.

---

Architecture

                    ┌─────────────┐
                    │  INTERNET   │
                    └──────┬──────┘
                           │
       ┌───────────────────┼───────────────────┐
       │                   │                   │
  ┌────┴────┐        ┌─────┴────┐        ┌─────┴────┐
  │ Relay 1 │        │ Relay 2  │        │ Relay 3  │
  │ Primary │        │Secondary │        │ Backup   │
  └────┬────┘        └────┬─────┘        └────┬─────┘
       │                  │                   │
       └──────────────────┼───────────────────┘
                          │
               ┌──────────┴──────────┐
               │   VPN MESH (10.99.x) │
               └──────────┬──────────┘
                          │
    ┌──────┬──────┬───────┼───────┬──────┬──────┐
    │      │      │       │       │      │      │
  gpu1   app1   dev1    dev2    ai1    dev3   mac1
  (GPU)  (App)  (Dev)  (Dev)   (AI)   (Dev)  (Mac)

Components:

• mpop CLI: Runs locally, manages all servers
• server-agent: Collects metrics on each server → sends to dashboard
• vssh: Ultra-fast remote execution within VPN (port 48291)
• wire: VPN mesh network

---

Configuration

mpop config     # View config
mpop init       # Initialize config
# Config file: ~/.mpop/config.json

Example:

{
  "language": "en",
  "ollama_host": "gpu1",
  "ollama_model": "qwen2.5:7b",
  "llm_model": "gpt-4o-mini",
  "connection": {"vpn": "wire", "ssh_method": "ssh"}
}

Key Settings:

Key	Description	Default
language	Language (en/ko)	en
ollama_host	Ollama server	localhost
ollama_model	Ollama model	qwen2.5:7b
llm_model	OpenAI model	gpt-4o-mini
connection.vpn	VPN type	wire

---

File Locations

Path	Description
/usr/local/bin/mpop	Executable
~/.mpop/config.json	Configuration
~/.mpop/secrets.enc	Encrypted secrets
~/.mpop/memory/	Project memory
~/.mpop/peers.json	VPN peer cache
~/.mpop/patterns.yaml	Pattern matching (for mpop ask)
~/.mpop/workflows/	Workflow definitions

---

What's New

v4.9.0 (2026-03-07)

• Pattern Matching - Fast CLI without AI for common commands
  • ~/.mpop/patterns.yaml - 120+ English patterns + Korean aliases
  • Arguments: {1}, {2}, {*} placeholders
  • Example: mpop ask "docker restart nginx" → instant execution
• MCP 58 tools - Added mpop_hw, mpop_advise, mpop_roles
• Local LLM fallback - gpt_generate() → Ollama when no API key
• Voice fix - Correct LLM host configuration
• All English - README, CLI help, MCP docs in English (Korean as language pack)

v4.8.0 (2026-03-07)

• mpop ask - Natural language server management
  • Auto-fix awk quotes (awk "..." → awk '...')
  • Quote correction (}'" → }"')
  • Multi-server comma support (-t web1,web2,web3)
• mpop advise --ai - Local LLM (Ollama) infrastructure analysis
• mpop advise - Instant response with agent cache
• Config - language, ollama_host, ollama_model support
• server-agent v3.3 - Hardware/software reporting, delta updates

v4.7.0 (2026-03-06)

• mpop advise - Infrastructure advisor (hardware, software, security)
• mpop hw - Local machine hardware info

v4.6.0 (2026-03-04)

• mpop news - RSS news briefing (tech/korea/world/dev) + LLM summary + TTS
• mpop_python - Run complex Python code via MCP (saves file then executes)
• mpop ask - Enhanced AI prompt rules (ss, head/tail, awk simplification)
• MCP 46 tools - Added mpop_news, mpop_python

v4.5.0 (2026-03-02)

• RAG ask quality - TF-IDF threshold tuning, more context (3000 chars)
• RAG model config - mpop config set rag.ask_model qwen2.5:14b (default)
• Multilingual prompts - LLM references documents more accurately

v4.2.0 (2026-02-28)

• exec security check - 2-stage dangerous command confirmation (DANGER/WARNING)
• mpop delete - MCP-compatible safe delete (blocks system paths)
• mpop watchdog - Auto-recover VPN (5-minute check, restart on disconnect)
• MCP exec parsing fix - Commands like ls /home/user parse correctly
• MCP shell mode - shell=true for pipes (|), redirects (>)
• Timeout increase - heal 5min, backup 10min for heavy operations
• MCP 42 tools - Added watchdog, delete

v4.1.0 (2026-02-28)

• Project memory - No AI hallucination (2-Agent system)
• Unified setup - mpop setup <server> -x (mpop+vssh+agent)
• MCP server update - Extended setup tool
• Dashboard /docs - Added document page

v4.4.0 (2026-03-01)

• RAG improvements - Cumulative indexing (default), update, delete commands
• VPN routing - Fixed NAT node /32 peer bug
• Radio - 10-channel streaming restored

v4.10.0 (2026-03-09)

• Service management - Cluster service start/stop/status/logs
• Retry system - Retry failed commands from history
• Log analysis - AI-powered log analysis across servers
• i18n - English default, Korean language pack

v4.0.0 (2026-02-27)

• AI Agent - Natural language server management (mpop ask)
• Workflow - Multi-step automation
• RAG - Document-based Q&A
• MCP integration - Claude Desktop integration

v3.9.0 (2026-02-28)

• mpop heal - Auto problem detection & recovery
• mpop predict - Future problem prediction
• mpop diff - Server comparison
• mpop export - HTML report

v3.7.0 (2026-02-27)

• mpop secret - Level 3/4 secret management
• mpop voice - Voice briefing

---

Dependencies

• Python 3.8+
• No external packages (single file, ~650KB)
• VPN: wire or Tailscale
• AI: OpenAI API or local Ollama

---

MeshPOP Project - Swiss Army knife for server management + AI