crowdsec-blocklist-import

Import 32 threat intelligence feeds into CrowdSec with automatic deduplication, normalization, and real-time sync.

  Threat Feeds (32)           crowdsec-blocklist-import           CrowdSec LAPI
 ┌──────────────────┐        ┌────────────────────────┐        ┌──────────────┐
 │ IPsum            │───────>│  Fetch & Normalize     │        │              │
 │ Spamhaus DROP    │───────>│  Deduplicate vs LAPI   │───────>│  Decisions   │──> Bouncers
 │ Firehol L1/L2/L3 │───────>│  Batch Import          │        │  Database    │    (fw, CDN,
 │ Abuse.ch Feodo   │───────>│  Allowlist Filtering   │        │              │     nginx...)
 │ 27 more feeds... │───────>│  Webhook + Metrics     │        └──────────────┘
 └──────────────────┘        └────────────────────────┘

Table of Contents: Why This Tool | Features | Quickstart | Installation | Configuration | Blocklists | CLI Usage | Advanced Usage | Monitoring | Troubleshooting | Contributing

Why IP Freshness Matters

Most blocklist tools suffer from a critical flaw: staleness. They fetch blocklists on a schedule, cache them, and enforce stale entries for days or weeks. By then, threat actors have rotated to new infrastructure, but your firewall still blocks addresses that were compromised weeks ago.

crowdsec-blocklist-import solves this:

• Fresh IPs propagate instantly -- New threats from 32 feeds hit your network within minutes, not days
• Expired threats are removed immediately -- Recovered IPs are automatically delisted, not held for weeks
• No cron delays -- Run hourly or on-demand via built-in scheduler
• No stale drift -- Every sync is a complete refresh; no orphaned entries linger

This is the difference between reactive security (waiting for alerts) and active threat intelligence (staying ahead of attackers).

Core Features

• Deduplication Engine -- Detects IPs already in CrowdSec, eliminating redundant processing and API calls
• Normalization Layer -- Strips comments, validates CIDR blocks, removes duplicates, enforces consistent formatting across all feeds
• Real-Time Sync -- No caching, no delays. Every import is a complete refresh with live threat data
• 32 Threat Feeds -- IPsum, Spamhaus, Blocklist.de, Firehol, Abuse.ch, Emerging Threats, Binary Defense, DShield, AbuseIPDB, StopForumSpam, Tor nodes, scanner IPs, and more
• Per-Feed Control -- Enable or disable individual blocklists via environment variables
• Allowlist Support -- Three-tier system: static IP lists, CIDR ranges, and provider-specific exceptions (GitHub IPs)
• Built-in Scheduler -- Long-lived daemon mode with INTERVAL=3600. Graceful SIGTERM/SIGINT shutdown
• Webhook Notifications -- Push import results to Discord, Slack, or any generic webhook endpoint
• AbuseIPDB Integration -- Public mirror (no key needed) plus optional direct API for higher rate limits and fresher data
• Prometheus Metrics -- Push to Pushgateway for monitoring imports, deduplication rates, and feed health
• Grafana Dashboard -- Pre-built dashboard for visualizing import metrics
• Docker Secrets -- All credential variables support _FILE suffix for mounted secret files
• Consolidated Alerts -- Optionally batch all IPs into a single alert per run to save CrowdSec alert quota
• Machine Status -- Sends LAPI heartbeat and OS metadata so cscli machines list shows recent activity

Quickstart

1. Prerequisites

You need CrowdSec running with LAPI credentials:

# Create machine credentials (for writing decisions)
cscli machines add blocklist-import --password 'SecurePassword123'

# Create bouncer key (for reading existing decisions)
cscli bouncers add blocklist-import -o raw
# Save the output -- you'll need it below

2. Docker Compose (Recommended)

services:
  blocklist-import:
    image: ghcr.io/wolffcatskyy/crowdsec-blocklist-import:latest
    restart: unless-stopped
    networks:
      - crowdsec
    environment:
      - CROWDSEC_LAPI_URL=http://crowdsec:8080
      - CROWDSEC_LAPI_KEY=YOUR_BOUNCER_KEY
      - CROWDSEC_MACHINE_ID=blocklist-import
      - CROWDSEC_MACHINE_PASSWORD=SecurePassword123
      - DECISION_DURATION=24h
      - INTERVAL=3600           # Run every hour (built-in scheduler)
      - CROWDSEC_HEARTBEAT_INTERVAL=60
      - LOG_LEVEL=INFO

networks:
  crowdsec:
    external: true

docker compose up -d

With INTERVAL=3600, the container runs as a long-lived daemon and repeats every hour. No cron or systemd timer needed. Set INTERVAL=0 (default) for a single run.

3. One-Shot Mode (Cron/Timer)

If you prefer external scheduling, omit INTERVAL and use restart: "no":

# Daily at 4am
0 4 * * * docker compose -f /path/to/compose.yml up --abort-on-container-exit

Installation

Docker (Recommended)

docker run --rm --network crowdsec \
  -e CROWDSEC_LAPI_URL=http://crowdsec:8080 \
  -e CROWDSEC_LAPI_KEY=YOUR_KEY \
  -e CROWDSEC_MACHINE_ID=blocklist-import \
  -e CROWDSEC_MACHINE_PASSWORD=YourPassword \
  ghcr.io/wolffcatskyy/crowdsec-blocklist-import:latest

Homebrew (macOS/Linux)

brew tap wolffcatskyy/crowdsec
brew install crowdsec-blocklist-import

pip (Python 3.9+)

pip install git+https://github.com/wolffcatskyy/crowdsec-blocklist-import.git

cp .env.example .env
# Edit .env with your credentials

crowdsec-blocklist-import

From Source

git clone https://github.com/wolffcatskyy/crowdsec-blocklist-import.git
cd crowdsec-blocklist-import

pip install -r requirements.txt
cp .env.example .env
# Edit .env with your credentials

python blocklist_import.py

Build Docker Image Locally

git clone https://github.com/wolffcatskyy/crowdsec-blocklist-import.git
cd crowdsec-blocklist-import
docker build -t crowdsec-blocklist-import .
docker run --rm --network crowdsec -e ... crowdsec-blocklist-import

For detailed configuration options, see Configuration Reference.

Configuration

Minimal Setup

Edit .env with your CrowdSec credentials:

CROWDSEC_LAPI_URL=http://crowdsec:8080
CROWDSEC_LAPI_KEY=your_bouncer_key
CROWDSEC_MACHINE_ID=blocklist-import
CROWDSEC_MACHINE_PASSWORD=your_password
DECISION_DURATION=24h

All credential variables support Docker Secrets via _FILE suffix (e.g., CROWDSEC_LAPI_KEY_FILE=/run/secrets/lapi_key).

LAPI HTTPS and Client Certificate Settings

This importer is a client of CrowdSec LAPI. It has three independent TLS/auth settings:

• CROWDSEC_LAPI_CA_CERT_PATH verifies the HTTPS certificate served by LAPI. Use it when connecting to https://... and the LAPI server certificate is not trusted by the container's default CA store. This does not authenticate the importer to LAPI.
• CROWDSEC_LAPI_AGENT_CERT_PATH and CROWDSEC_LAPI_AGENT_KEY_PATH present an agent client certificate to LAPI. CrowdSec uses this for watcher login/JWT auth, which is needed for writing alerts and decisions.
• CROWDSEC_LAPI_BOUNCER_CERT_PATH and CROWDSEC_LAPI_BOUNCER_KEY_PATH present a bouncer client certificate to LAPI. CrowdSec uses this for bouncer auth, which is needed for reading existing decisions from /v1/decisions.

Agent and bouncer certificate pairs are optional, but each pair must be complete if used.

For normal HTTPS plus machine credentials/API key:

CROWDSEC_LAPI_URL=https://crowdsec:8080
CROWDSEC_LAPI_CA_CERT_PATH=/certs/crowdsec_lapi.pem
CROWDSEC_LAPI_KEY=your_bouncer_key
CROWDSEC_MACHINE_ID=blocklist-import
CROWDSEC_MACHINE_PASSWORD=your_password

For LAPI client certificate authentication:

CROWDSEC_LAPI_URL=https://crowdsec:8080
CROWDSEC_LAPI_AGENT_CERT_PATH=/certs/blocklist-import-agent.pem
CROWDSEC_LAPI_AGENT_KEY_PATH=/certs/blocklist-import-agent-key.pem
CROWDSEC_LAPI_BOUNCER_CERT_PATH=/certs/blocklist-import-bouncer.pem
CROWDSEC_LAPI_BOUNCER_KEY_PATH=/certs/blocklist-import-bouncer-key.pem
# Optional and unrelated to the client cert/key:
CROWDSEC_LAPI_CA_CERT_PATH=/certs/crowdsec_lapi.pem

When CROWDSEC_LAPI_BOUNCER_CERT_PATH and CROWDSEC_LAPI_BOUNCER_KEY_PATH are set, the importer presents that bouncer certificate for decision reads and does not send the X-Api-Key header on those requests. Existing API key and machine JWT authentication continue to work when client certificate/key pairs are not configured.

CrowdSec has two different CA settings that are easy to mix up:

• CROWDSEC_LAPI_CA_CERT_PATH is client-side. It tells this importer which CA to trust for the LAPI HTTPS server certificate.
• api.server.tls.ca_cert_path is server-side in CrowdSec. It tells LAPI which CA signed client certificates, so LAPI can verify the agent and bouncer client certificates.

A CrowdSec config with only cert_file and key_file enables HTTPS for LAPI. It does not require, or enable, client certificate authentication by itself. To use client certificates as authentication, LAPI also needs a client-cert CA and allowed OU, for example:

api:
  server:
    tls:
      cert_file: /usr/local/etc/crowdsec/ssl/crowdsec_lapi.pem
      key_file: /usr/local/etc/crowdsec/ssl/crowdsec_lapi_key.pem
      ca_cert_path: /usr/local/etc/crowdsec/ssl/client_ca.pem
      agents_allowed_ou:
        - agent-ou
      bouncers_allowed_ou:
        - bouncer-ou

Common Settings

Notification Settings

AbuseIPDB

ENABLE_ABUSE_IPDB=true fetches the public mirror maintained by @borestad — no API key required.

For higher rate limits and fresher data, you can optionally configure a direct API key:

Get a free API key at abuseipdb.com. The free tier allows 5 blacklist checks per day.

Feed Presets

PRESET picks a ready-made feed set: it sets the default for every ENABLE_* switch, and explicit ENABLE_* values always win.

PRESET=embedded also sets MAX_DECISIONS=15000 unless you set your own value (MAX_DECISIONS=0 keeps it unlimited). Other presets leave MAX_DECISIONS unlimited.

When neither PRESET nor BLOCKLISTS_OPT_IN is set, the legacy all-on default still applies, but every run logs a loud deprecation warning (and includes it in webhook notifications) ahead of the v4.0.0 flip. A run that ends up with zero enabled feeds exits non-zero instead of silently importing nothing.

Set FAIL_ON_DEAD_FEED=true to exit non-zero when any enabled feed fails to fetch.

Run --list-sources to see every feed with its preset membership and license metadata (license, attribution_required, commercial_ok).

Selective Blocklists

All blocklists are enabled by default. Disable feeds you don't need:

# Optional safe posture: disable every feed unless explicitly enabled below
BLOCKLISTS_OPT_IN=true
ENABLE_IPSUM=true           # Aggregated threats (review before enabling)
ENABLE_SPAMHAUS=true        # Spamhaus DROP
ENABLE_TOR=false            # Tor exit nodes (may cause false positives)
ENABLE_SCANNERS=false       # Shodan/Censys scanners

See Configuration Reference for the full list of ENABLE_* variables.

Custom Blocklists

Import any additional IP list via URL — e.g. cloud provider ranges, vendor-specific blocklists, or community feeds:

# Comma-separated list of URLs; each is fetched and imported like a built-in source
CUSTOM_BLOCKLISTS="https://example.com/alibaba.txt,https://example.com/facebook.txt"

Each URL is added as its own source (custom_blocklist_0, custom_blocklist_1, ...) and goes through the same dedup, allowlist, and MAX_DECISIONS logic as the built-in feeds.

Allowlists

Protect trusted IPs from being blocked:

# Comma-separated IPs and/or CIDR ranges
ALLOWLIST="1.2.3.4,5.6.7.8,192.168.0.0/16,10.0.0.0/8"

# Auto-fetch GitHub IP ranges (git, web, api, hooks, actions)
ALLOWLIST_GITHUB=true

Supported Blocklists

crowdsec-blocklist-import pulls from 32 threat intelligence sources:

For a complete list with URLs and threat types, see Examples. You can also generate a current machine-readable table with python blocklist_import.py --list-sources --format md.

CLI Usage

python blocklist_import.py [options]

Options:
  -h, --help                Show help
  -v, --version             Show version and exit
  -n, --dry-run             Preview without importing
  -d, --debug               Enable debug logging
  --setup                   Launch interactive setup wizard
  --lapi-url URL            Override LAPI URL
  --lapi-key KEY            Override LAPI key
  --duration DURATION       Override decision duration
  --batch-size SIZE         Override batch size
  --list-sources            List all available blocklist sources
  --format FORMAT           Output format for --list-sources: text (default) or md
  --validate                Validate configuration and exit
  --pushgateway-url URL     Override Prometheus Pushgateway URL
  --no-metrics              Disable Prometheus metrics for this run
  --interval SECONDS        Daemon mode: repeat every N seconds
  --webhook-url URL         Webhook URL for notifications
  --webhook-type TYPE       Webhook format: generic, discord, slack
  --mode MODE               Filter blocklist sources regarding rate-limiting: all, frequent, limited

Examples

# Launch interactive setup wizard
python blocklist_import.py --setup

# Dry-run to see what would be imported
python blocklist_import.py --dry-run

# List all available sources
python blocklist_import.py --list-sources

# Same list as a Markdown table, ready for docs
python blocklist_import.py --list-sources --format md

# Import with custom duration and batch size
python blocklist_import.py --duration 48h --batch-size 500

# Validate configuration without running
python blocklist_import.py --validate

Advanced Usage

Daemon Mode (Built-in Scheduler)

Run as a long-lived service instead of using cron:

INTERVAL=3600         # Run every hour
RUN_ON_START=false    # Skip the first run and wait for the interval
CROWDSEC_HEARTBEAT_INTERVAL=60  # Keep cscli machines list heartbeat fresh

The daemon sends a machine heartbeat every 60 seconds by default, matching CrowdSec's own client behavior. Set CROWDSEC_HEARTBEAT_INTERVAL=0 to disable it.

The daemon handles SIGTERM/SIGINT gracefully -- it finishes the current run, then exits. This makes it Docker-friendly with restart: unless-stopped.

Webhook Notifications

Get notified after each import run:

# Discord
WEBHOOK_URL=https://discord.com/api/webhooks/YOUR_WEBHOOK
WEBHOOK_TYPE=discord

# Slack
WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
WEBHOOK_TYPE=slack

# Generic JSON POST
WEBHOOK_URL=https://your-endpoint.example.com/webhook
WEBHOOK_TYPE=generic

AbuseIPDB Direct API

By default, ENABLE_ABUSE_IPDB=true fetches the public mirror (no API key needed). To use the direct API for higher rate limits and fresher data:

ABUSEIPDB_API_KEY=your_api_key_here
# or ABUSEIPDB_API_KEY_FILE=your_api_key_file_path_here
ABUSEIPDB_MIN_CONFIDENCE=90   # Only IPs with 90%+ confidence
ABUSEIPDB_LIMIT=10000         # Max IPs to fetch

Get a free API key at abuseipdb.com. The free tier allows 5 blacklist checks per day.

Note: The public mirror (via ENABLE_ABUSE_IPDB) and the direct API are complementary. When an API key is configured, both sources are queried and deduplicated.

Docker Secrets

All credential variables support the _FILE suffix for Docker Secrets:

services:
  blocklist-import:
    image: ghcr.io/wolffcatskyy/crowdsec-blocklist-import:latest
    environment:
      - CROWDSEC_LAPI_KEY_FILE=/run/secrets/lapi_key
      - CROWDSEC_MACHINE_PASSWORD_FILE=/run/secrets/machine_password
    secrets:
      - lapi_key
      - machine_password

secrets:
  lapi_key:
    file: ./secrets/lapi_key.txt
  machine_password:
    file: ./secrets/machine_password.txt

Docker with Custom Config

Mount your .env file:

docker run --rm \
  --network crowdsec \
  -v /path/to/.env:/app/.env:ro \
  ghcr.io/wolffcatskyy/crowdsec-blocklist-import:latest

Scheduling with Systemd Timer

[Unit]
Description=CrowdSec Blocklist Import
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=docker compose -f /opt/compose/blocklist-import.yml up --abort-on-container-exit
StandardOutput=journal
StandardError=journal

[Unit]
Description=CrowdSec Blocklist Import Timer
Requires=blocklist-import.service

[Timer]
OnBootSec=5min
OnUnitActiveSec=1h
AccuracySec=1min

[Install]
WantedBy=timers.target

systemctl daemon-reload
systemctl enable --now blocklist-import.timer

For more examples, see Advanced Usage.

Monitoring

Prometheus Metrics

Push metrics to Prometheus Pushgateway:

METRICS_PUSHGATEWAY_URL=http://prometheus:9091

Metrics tracked:

• Total IPs imported per source
• Deduplicated entries skipped
• Failed imports per source
• Import duration per run
• Feed health per source (v3.9): last successful fetch timestamp, raw entry count, unique IP contribution, and HTTP status

Grafana Dashboard

A pre-built Grafana dashboard is included at grafana-dashboard.json. Import it into Grafana to visualize import activity, deduplication rates, and feed health over time.

Part of the CrowdSec UniFi Ecosystem

crowdsec-blocklist-import works great with any CrowdSec setup. It also pairs perfectly with the UniFi-specific projects below for a complete detect → decide → enforce stack on UniFi hardware:

Troubleshooting

CrowdSec Connection Failed

# Verify LAPI is reachable
curl http://crowdsec:8080/health

# If using Docker, ensure containers share a network
docker network inspect crowdsec

Authentication Error

# Test bouncer key
curl -H "X-Api-Key: YOUR_KEY" http://crowdsec:8080/decisions

# Test TLS client certificate auth
curl --cacert /certs/ca.pem \
  --cert /certs/blocklist-import.pem \
  --key /certs/blocklist-import-key.pem \
  https://crowdsec:8080/v1/watchers/login \
  -H "Content-Type: application/json" \
  -d '{"machine_id":"blocklist-import","password":"YourPassword","scenarios":["external/blocklist"]}'

# Test machine login
curl -X POST http://crowdsec:8080/watchers/login \
  -H "Content-Type: application/json" \
  -d '{"machine_id":"blocklist-import","password":"YourPassword"}'

No IPs Imported

docker logs blocklist-import        # Docker logs
python blocklist_import.py --debug  # Detailed output

Common causes:

• All blocklists disabled (check ENABLE_* variables)
• CrowdSec already has all IPs (check deduplication count in logs)
• Network connectivity issue (check curl https://example.com)

Memory Issues

BATCH_SIZE=100          # Reduce from default 1000
ENABLE_IPSUM=false      # IPsum is the largest feed

For more troubleshooting, see FAQ.

Technical Details

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

• Report bugs: GitHub Issues
• Suggest features: GitHub Discussions
• Changelog: CHANGELOG.md
• Roadmap: ROADMAP.md

Contributors

• @gaelj — Major contributions including IP refresh before expiration, Grafana dashboard improvements, CI/CD pipeline, bug fixes, and NixOS packaging

License

MIT License -- See LICENSE for details.

Credits

Maintained by wolffcatskyy.

Special Thanks:

• CrowdSec for the threat detection platform
• The security community for maintaining public threat feeds
• Awesome CrowdSec community