mini-siem 1.0.0

A pip-installable cross-platform SIEM CLI tool with real-time threat detection

🛡️ Mini SIEM

A pip-installable, cross-platform Security Information & Event Management (SIEM) tool built in Python. Collects real macOS system logs, detects threats in real time, stores everything in a database, and displays it all on a password-protected web dashboard.

Inspired by enterprise tools like Splunk, IBM QRadar, and Microsoft Sentinel — built from scratch using only Python and distributed as a proper CLI package.

---

⚡ Install & Run in 3 Commands

pip install mini-siem
mini-siem init
mini-siem run --demo

That's it. No cloning. No manual file execution. Real software distribution.

---

📸 What It Looks Like

  ╔╦╗╦╔╗╔╦  ╔═╗╦╔═╗╔╦╗
  ║║║║║║║║  ╚═╗║║╣ ║║║
  ╩ ╩╩╝╚╝╩  ╚═╝╩╚═╝╩ ╩
  Cross-Platform Log Monitor & Alert Tool

[1/4] Collecting logs...        → 9 raw entries collected.
[2/4] Parsing & normalising...  → 9 events parsed.
[3/4] Running detection engine...
      → 2 alert(s) suppressed (trusted activity — audit trail preserved)
[4/4] Dispatching alerts...

============================================================
  🚨  SECURITY ALERTS
============================================================
[!!!!!! CRITICAL !!!!!!] THREAT_INTEL_MATCH | Score: 15
  ↳ IP 203.0.113.42 matched threat intelligence list
[!!! HIGH !!!] BRUTE_FORCE | Score: 42
  ↳ 8 failed login attempts in 120s from '198.51.100.7'
============================================================

Dashboard at http://127.0.0.1:5000 — login protected, live charts, attack timeline.

---

📁 Project Structure

Mini SIEM/
│
├── mini_siem/                  ← Python package (the actual tool)
│   ├── __init__.py
│   ├── __main__.py             ← enables: python -m mini_siem
│   ├── main.py                 ← CLI entry point (all commands live here)
│   └── core/                   ← engine modules
│       ├── __init__.py
│       ├── collector.py        ← macOS log collection
│       ├── parser.py           ← normalisation & validation
│       ├── detector.py         ← 6 detection rules + risk scoring
│       ├── alert.py            ← console, file, email alerts
│       ├── report.py           ← TXT + CSV report generator
│       ├── database.py         ← SQLite persistent storage
│       ├── threat_intel.py     ← malicious IP matching
│       ├── siem_logger.py      ← internal SIEM activity log
│       ├── dashboard.py        ← Flask web dashboard
│       └── whitelist.py        ← false positive suppression
│
├── tests/
│   ├── test_parser.py          ← 20+ parser tests
│   ├── test_detector.py        ← 25+ detection rule tests
│   └── test_database.py        ← 15+ database tests
│
├── data/
│   └── threat_intel.txt        ← known malicious IPs (edit this)
│
├── pyproject.toml              ← package config (how pip install works)
├── Dockerfile                  ← container support
├── requirements.txt            ← dependencies
├── whitelist.json              ← trusted users/IPs (auto-created)
├── LICENSE                     ← MIT
└── README.md

User data is stored in ~/.mini_siem/ — completely separate from the code:

~/.mini_siem/
├── siem.db              ← SQLite database (all events + alerts)
├── threat_intel.txt     ← your threat intel list
├── whitelist.json       ← your trusted users/IPs
└── logs/
    └── siem.log         ← internal SIEM activity log

---

🍎 Installation on macOS

Requirements

• Python 3.10 or higher
• macOS Monterey, Ventura, or Sonoma

Install from source (development)

# Clone the repo
git clone https://github.com/yourusername/mini-SIEM.git
cd mini-SIEM

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate

# Install the package
pip install .

# First time setup
mini-siem init

Shortcut — add to your shell so venv activates automatically

echo 'alias siem="cd ~/Documents/PROJECTS/Mini\ SIEM && source venv/bin/activate"' >> ~/.zshrc
source ~/.zshrc

Now just type siem in any terminal window to get started.

---

🚀 Daily Usage (macOS)

Every time you open Terminal

siem                        # go to project + activate venv
sudo mini-siem run          # collect real macOS logs
mini-siem dashboard         # open web dashboard

Then visit http://127.0.0.1:5000 — login: admin / siem2025

---

📟 All Commands

mini-siem init

First-time setup. Creates ~/.mini_siem/ with database and config files.

mini-siem init

mini-siem run — Analyse logs and fire alerts

# Real macOS logs (requires sudo)
sudo mini-siem run

# Demo mode — simulated logs, no sudo needed
mini-siem run --demo

# Look back further in time
sudo mini-siem run --hours 48

# Live monitoring — scans every 10 seconds continuously
sudo mini-siem run --live

# Live monitoring with custom interval
sudo mini-siem run --live --interval 30

# Skip saving to database
sudo mini-siem run --no-db

# Skip generating report files
sudo mini-siem run --no-report

# Send email alerts (configure env vars first)
sudo mini-siem run --email

mini-siem dashboard — Web dashboard

mini-siem dashboard                    # uses data from database
mini-siem dashboard --demo             # uses simulated demo data
mini-siem dashboard --port 8080        # use different port

mini-siem query — Search the database

mini-siem query --alerts               # all recent alerts
mini-siem query --ip 203.0.113.42      # events from specific IP
mini-siem query --user admin           # events for specific user
mini-siem query --type FAILED_LOGIN    # filter by event type
mini-siem query --since 2025-01-15     # events after a date
mini-siem query --alerts --limit 10    # limit results

mini-siem stats — Summary of everything stored

mini-siem stats

mini-siem logs — View SIEM's own internal log

mini-siem logs
mini-siem logs --lines 50

Get help anytime

mini-siem --help
mini-siem run --help
mini-siem query --help

Alternative — run without installing

python -m mini_siem run --demo
python -m mini_siem dashboard --demo

---

🔍 Detection Rules

Six independent rules run on every batch of events:

Rule	What Triggers It	Severity
BRUTE_FORCE	5+ failed logins from same IP within 120 seconds	HIGH
NEW_IP_LOGIN	Successful login from a previously unseen IP	MEDIUM
PRIVILEGE_ESCALATION	Any sudo or privilege escalation event	HIGH
ERROR_SPIKE	10+ failures within 60 seconds	MEDIUM
THREAT_INTEL_MATCH	Source IP found in threat intelligence list	CRITICAL
CRITICAL_RISK_THRESHOLD	Total system risk score exceeds 20 points	CRITICAL

---

🧠 Risk Scoring

Every event gets a numerical risk score. Scores accumulate to the system total:

Condition	Points
Failed login	+2
Privilege escalation	+5
Account lockout	+4
Explicit credential use	+3
Unknown external IP	+3
Part of brute-force burst	+8 bonus
Part of error spike	+6 bonus
IP matched threat intel	+10 bonus

When total score ≥ 20 → CRITICAL alert fires automatically.

---

🛡️ Whitelist — False Positive Suppression

Your own legitimate admin activity (like running sudo) would normally trigger PRIVILEGE_ESCALATION alerts. The whitelist suppresses these so you only see real threats.

Your username is added automatically when you run mini-siem init.

Edit the whitelist at ~/.mini_siem/whitelist.json:

{
  "trusted_users": ["jyotikuaner", "root"],
  "trusted_ips":   ["127.0.0.1", "::1", "192.168.1.5"],
  "trusted_processes": ["sudo", "python3", "venv"]
}

Whitelisted events are still saved to the database — full audit trail is preserved. They just don't show up as loud alerts.

---

🌐 Threat Intelligence

Every source IP is checked against ~/.mini_siem/threat_intel.txt. Any match fires a CRITICAL alert immediately.

Add your own malicious IPs — one per line:

# ~/.mini_siem/threat_intel.txt
203.0.113.42        # known scanner
198.51.100.7        # brute force bot
185.220.101.0/24    # tor exit nodes (CIDR block)

CIDR blocks (like /24) are supported — matches entire subnets.

In production, replace with real threat feeds:

• AbuseIPDB
• Emerging Threats
• Spamhaus DROP list

---

💻 Web Dashboard

mini-siem dashboard --demo

Visit http://127.0.0.1:5000 — login: admin / siem2025

Features:

• Login page — password protected
• 7 KPI cards — failed logins, escalations, critical alerts, risk score, threat intel hits
• Attack timeline — 5-minute bucket chart, bars turn red during burst attacks
• Failed vs successful logins by hour
• Top suspicious IPs chart
• Top targeted users chart
• Alert severity distribution
• Colour-coded alerts table
• Live SIEM internal log viewer
• REST API — /api/events and /api/alerts return JSON

Change the dashboard password — open mini_siem/core/dashboard.py:

ADMIN_USERNAME = "admin"
ADMIN_PASSWORD = "siem2025"    # change this

---

💾 Database

All events and alerts are stored permanently in ~/.mini_siem/siem.db (SQLite). Data builds up over time — your history is always there.

from mini_siem.core.database import query_events, query_alerts, get_top_ips

# Find all events from a suspicious IP
events = query_events(ip="203.0.113.42")

# Find critical alerts from last week
alerts = query_alerts(severity="CRITICAL", since="2025-01-08")

# Top 10 attacking IPs all time
top = get_top_ips(10)

Clean up old data:

from mini_siem.core.database import clear_old_data
clear_old_data(days=30)

---

📊 Output Files

After each run, files are saved to ~/.mini_siem/reports/:

File	Description
alerts.txt	All alerts, human-readable
security_report.txt	Full report with hourly activity chart
security_events.csv	All events — open in Excel or Numbers
security_alerts.csv	All alerts — open in Excel or Numbers

---

📧 Email Alerts (Optional)

export SIEM_SENDER="[e-mail]"
export SIEM_PASSWORD="your_app_password"
export SIEM_RECIPIENT="[e-mail]"
export SIEM_MIN_SEVERITY="HIGH"

sudo mini-siem run --email

Gmail requires an App Password — generate at: Google Account → Security → 2-Step Verification → App Passwords

---

🧪 Running Tests

python -m pytest tests/ -v

# Run individual test files
python -m pytest tests/test_detector.py -v
python -m pytest tests/test_parser.py -v
python -m pytest tests/test_database.py -v

# Run with coverage report
pip install pytest-cov
python -m pytest tests/ --cov=mini_siem --cov-report=term-missing

---

🐳 Docker

# Build
docker build -t mini-siem .

# Run dashboard
docker run -p 5000:5000 mini-siem

# Keep data between restarts
docker run -p 5000:5000 -v ~/.mini_siem:/root/.mini_siem mini-siem

---

🍎 macOS Log Collection

Real log collection uses two macOS sources:

log show command — macOS unified logging. Captures authentication failures, SSH events, sudo usage, and privilege escalation.

/var/log/system.log — Traditional syslog fallback.

Sudo is required for real logs:

sudo mini-siem run

Without sudo, the tool automatically falls back to demo mode.

What gets detected from real macOS logs:

• SSH brute-force attempts
• Failed password events
• Sudo / privilege escalation
• Invalid user login attempts
• Account lockouts

Generate real test events:

# In a second Terminal window
ssh wronguser@localhost      # creates failed login entries
sudo ls                      # creates privilege escalation entry
sudo whoami

Then scan:

sudo mini-siem run --hours 1

---

📦 Building for Distribution

pip install build
python -m build

Creates in dist/:

• mini_siem-1.0.0.tar.gz — source distribution
• mini_siem-1.0.0-py3-none-any.whl — installable wheel

Install from wheel:

pip install mini_siem-1.0.0-py3-none-any.whl

Publish to PyPI (optional):

pip install twine
twine upload dist/*

Then anyone in the world can install with:

pip install mini-siem

---

🔧 Troubleshooting

mini-siem: command not found

source venv/bin/activate

Port 5000 already in use

mini-siem dashboard --port 8080
# visit http://127.0.0.1:8080

Zero events collected (no sudo)

sudo mini-siem run

Zero events even with sudo

• Go to System Settings → Privacy & Security → Full Disk Access
• Add Terminal and enable it
• Restart Terminal and try again

Database issues

rm ~/.mini_siem/siem.db
mini-siem init

ModuleNotFoundError after code changes

pip uninstall mini-siem -y
pip install .

---

📚 Concepts This Project Demonstrates

Concept	Where
pip package distribution	pyproject.toml, mini_siem/ structure
CLI design	Click framework, mini_siem/main.py
macOS log collection	core/collector.py
Log parsing & normalisation	core/parser.py
Rule-based threat detection	core/detector.py
Risk scoring	Numerical danger scores per event
False positive suppression	core/whitelist.py
SQLite persistence	core/database.py
Threat intelligence	core/threat_intel.py, CIDR matching
Web dashboard + auth	core/dashboard.py, Flask sessions
Internal logging	core/siem_logger.py, log rotation
Email alerting	core/alert.py, smtplib
Unit testing	tests/, pytest, 60+ test cases
Docker containerisation	Dockerfile

---

📄 License

MIT License — see LICENSE for full text.

Free to use, modify, and distribute. Keep the copyright notice.

---

Built as a portfolio project demonstrating defensive security engineering. Inspired by Splunk, IBM QRadar, and Microsoft Sentinel.