runbooks/incus-crowdsec-architecture.md

Multi-Layer Security Architecture with CrowdSec and OpenAppSec

🏗️ Infrastructure Overview

This document describes a multi-layer security architecture implemented using Incus containers, featuring CrowdSec community-driven threat intelligence and OpenAppSec ML-based WAF protection.

📋 Container Inventory

Container	IP Address	Role	Technology
ab-test	10.90.135.123	Attack Simulation	curl/testing tools
openresty	10.90.135.17	Layer 1 Security	OpenResty + CrowdSec Bouncer
crowdsec	10.90.135.49	Threat Intelligence	CrowdSec Engine
openappsec	10.90.135.186	Layer 2 Security	ML-based WAF
caddy	10.90.135.16	Backend Server	Caddy HTTP Server
php-auth	10.90.135.83	Test Endpoint	PHP Authentication Service

🏗️ Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    Incus Container Infrastructure                │
│                          (10.90.135.0/24)                     │
└─────────────────────────────────────────────────────────────────┘
                                    │
┌─────────────────────────────────────────────────────────────────┐
│  🌐 External Traffic Entry Point                               │
│                                                                 │
│  ┌─────────────────┐                                           │
│  │   ab-test       │  ← Attack simulation container            │
│  │ 10.90.135.123   │                                           │
│  └─────────────────┘                                           │
└─────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────┐
│  🛡️ Layer 1: Community-Driven Threat Intelligence             │
│                                                                 │
│  ┌─────────────────┐    ┌─────────────────┐                   │
│  │   openresty     │◄──►│    crowdsec     │                   │
│  │ 10.90.135.17    │    │ 10.90.135.49    │                   │
│  │                 │    │                 │                   │
│  │ • OpenResty     │    │ • Threat Intel  │                   │
│  │ • CrowdSec      │    │ • Scenarios     │                   │
│  │   Bouncer       │    │ • Community DB  │                   │
│  │ • Rate Limiting │    │ • API Server    │                   │
│  │ • IP Blocking   │    │ • Log Analysis  │                   │
│  └─────────────────┘    └─────────────────┘                   │
│           │                       │                           │
│           └──── nginx-logs ────────┘                           │
│           (Shared Volume: /var/log/nginx)                     │
└─────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────┐
│  🧠 Layer 2: ML-Based Application Security                     │
│                                                                 │
│  ┌─────────────────┐                                           │
│  │   openappsec    │                                           │
│  │ 10.90.135.186   │                                           │
│  │                 │                                           │
│  │ • ML WAF        │                                           │
│  │ • Simple Model  │                                           │
│  │   V1.0          │                                           │
│  │ • Zero-day      │                                           │
│  │   Detection     │                                           │
│  │ • No Signatures │                                           │
│  └─────────────────┘                                           │
└─────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────┐
│  🔗 Layer 3: Backend Application Server                        │
│                                                                 │
│  ┌─────────────────┐                                           │
│  │     caddy       │                                           │
│  │ 10.90.135.16    │                                           │
│  │                 │                                           │
│  │ • HTTP Server   │                                           │
│  │ • Reverse Proxy │                                           │
│  │ • Auto HTTPS    │                                           │
│  └─────────────────┘                                           │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  🧪 Supporting Services                                         │
│                                                                 │
│  ┌─────────────────┐                                           │
│  │    php-auth     │  ← Authentication test endpoint           │
│  │ 10.90.135.83    │    (Returns 401 for brute force testing) │
│  └─────────────────┘                                           │
└─────────────────────────────────────────────────────────────────┘

📊 Traffic Flow & Security Processing

🌐 Attack Request
        │
        ▼
┌─────────────────┐
│   openresty     │ ← CrowdSec Bouncer
│ 10.90.135.17    │   • Community threat intel
│                 │   • Behavioral analysis
│                 │   • IP reputation
│                 │   • Rate limiting
└─────────────────┘
        │ ✅ Allowed
        ▼
┌─────────────────┐
│   openappsec    │ ← ML-based WAF
│ 10.90.135.186   │   • Zero-day detection
│                 │   • ML pattern analysis
│                 │   • Application-layer security
└─────────────────┘
        │ ✅ Clean
        ▼
┌─────────────────┐
│     caddy       │ ← Backend Server
│ 10.90.135.16    │   • Content delivery
│                 │   • Application logic
└─────────────────┘

🔒 Security Layers Detail

Layer 1: CrowdSec Community Intelligence

Container: openresty + crowdsec

Key Features:

• 🌍 Global threat intelligence sharing
• 🔍 HTTP brute force detection (LePresidente/http-generic-401-bf)
• 📊 Scenario-based behavioral analysis
• ⏱️ Real-time IP blocking (4-hour default ban)
• 🔄 Dynamic bouncer integration with OpenResty

Scenarios Enabled:

• crowdsecurity/http-generic-bf - Generic HTTP brute force detection
• crowdsecurity/http-admin-interface-probing - Admin interface scanning
• crowdsecurity/http-probing - General HTTP probing
• crowdsecurity/http-cve-probing - CVE exploitation attempts
• crowdsecurity/http-sqli-probing - SQL injection attempts

Configuration:

# CrowdSec Acquisition
filenames:
  - /var/log/nginx/access.log
  - /var/log/nginx/error.log
labels:
  type: nginx

Acquisition 디렉토리 구조화 (권장):

# /etc/crowdsec/acquis.d/ 디렉토리에 서비스별 파일 분리
/etc/crowdsec/acquis.d/
├── nginx.yaml      # Nginx 로그
├── apache.yaml     # Apache 로그 (필요시)
├── sshd.yaml       # SSH 로그
└── traefik.yaml    # Traefik 로그 (필요시)

nginx.yaml 예시:

filenames:
  - /var/log/nginx/access.log
  - /var/log/nginx/error.log
labels:
  type: nginx
---
# JSON 형식 로그 (별도 처리)
filenames:
  - /var/log/nginx/json_access.log
labels:
  type: nginx
  format: json

sshd.yaml 예시:

filenames:
  - /var/log/auth.log
labels:
  type: syslog

Layer 2: OpenAppSec ML WAF

Container: openappsec

Key Features:

• 🧠 Machine learning-based attack detection
• 🆕 Zero-day vulnerability protection
• 📝 No signature updates required
• ⚡ Simple Model V1.0 for fast processing
• 🔄 Automatic threat adaptation

ML Model: Simple Model V1.0

• Behavioral analysis
• Pattern recognition
• Anomaly detection
• Real-time learning

Layer 3: Backend Application

Container: caddy

Key Features:

• 🔐 Automatic HTTPS with Let's Encrypt
• 🔄 Reverse proxy capabilities
• 📈 High-performance HTTP/2 support
• 🎯 Clean traffic processing

🧪 Testing Infrastructure

Attack Simulation

• Container: ab-test
• Purpose: Simulate various attack patterns for testing
• Tools: curl, custom scripts

Authentication Endpoint

• Container: php-auth
• Purpose: Generate 401/403 responses for brute force testing
• Endpoint: /login.php - Always returns 401 Unauthorized

🔧 Implementation Details

Volume Mounts

# Shared nginx logs between openresty and crowdsec
nginx-logs: /var/log/nginx

Network Configuration

• Network: incus default bridge (10.90.135.0/24)
• Traffic Flow: ab-test → openresty → openappsec → caddy
• Special Routes: /admin/login → php-auth (testing)

Security Configuration

CrowdSec Bouncer (수동 등록)

API_URL: http://10.90.135.49:8080
API_KEY: s0ENc/6Tw+6m6tr0Qkjt/WAYU1QlC5/MzH7SQOCJX50

Bouncer 자동 등록 (권장)

컨테이너 시작 시 환경변수로 Bouncer 자동 등록. 수동 cscli bouncers add 불필요.

환경변수 방식:

# CrowdSec 컨테이너 시작 시 Bouncer 자동 등록
incus launch docker.io/crowdsecurity/crowdsec crowdsec \
  --config environment.BOUNCER_KEY_openresty=my-secure-bouncer-key-here \
  --config environment.BOUNCER_KEY_firewall=another-bouncer-key-here

Docker Compose 예시:

services:
  crowdsec:
    image: crowdsecurity/crowdsec:latest
    environment:
      # Bouncer 자동 등록 (BOUNCER_KEY_<name>=<key>)
      - BOUNCER_KEY_openresty=${OPENRESTY_BOUNCER_KEY}
      - BOUNCER_KEY_firewall=${FIREWALL_BOUNCER_KEY}
      # 컬렉션 자동 설치
      - COLLECTIONS=crowdsecurity/nginx crowdsecurity/http-cve
    volumes:
      - crowdsec-data:/var/lib/crowdsec/data
      - crowdsec-config:/etc/crowdsec
      - /var/log/nginx:/var/log/nginx:ro
    restart: unless-stopped

  openresty:
    image: openresty/openresty:latest
    environment:
      - CROWDSEC_API_URL=http://crowdsec:8080
      - CROWDSEC_API_KEY=${OPENRESTY_BOUNCER_KEY}
    depends_on:
      - crowdsec

Docker Secret 방식 (더 안전):

services:
  crowdsec:
    image: crowdsecurity/crowdsec:latest
    secrets:
      - bouncer_key_openresty
      - bouncer_key_firewall
    # Secret 파일은 /run/secrets/bouncer_key_<name> 으로 마운트됨

secrets:
  bouncer_key_openresty:
    file: ./secrets/openresty_bouncer_key
  bouncer_key_firewall:
    file: ./secrets/firewall_bouncer_key

# Secret 파일 생성
mkdir -p secrets
openssl rand -base64 32 > secrets/openresty_bouncer_key
openssl rand -base64 32 > secrets/firewall_bouncer_key
chmod 600 secrets/*

주의: 환경변수/Secret 방식은 초기 배포 시에만 동작. 기존 Bouncer 업데이트는 cscli 사용.

OpenResty Configuration

upstream openappsec_backend {
    server 10.90.135.186:80;
}

upstream auth_backend {
    server 10.90.135.83:8080;
}

server {
    listen 80;
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;
    
    location /admin/login {
        proxy_pass http://auth_backend/login.php;
    }
    
    location / {
        proxy_pass http://openappsec_backend;
    }
}

✅ Testing Results

Successful Test Cases

1. Brute Force Detection

   • ✅ 6 failed login attempts → CrowdSec alert triggered
   • ✅ IP 10.90.135.123 automatically banned
   • ✅ Scenario: LePresidente/http-generic-401-bf

2. IP Blocking

   • ✅ Banned IP receives 403 Forbidden
   • ✅ Traffic completely blocked at Layer 1

3. IP Unbanning

   • ✅ Manual unban via cscli decisions delete
   • ✅ Immediate traffic restoration

4. Multi-Layer Coordination

   • ✅ OpenResty logs → CrowdSec analysis
   • ✅ Real-time threat intelligence updates
   • ✅ Seamless traffic flow through all layers

Attack Patterns Tested

• HTTP brute force attacks (401 responses)
• Multiple rapid authentication attempts
• IP-based blocking and recovery

🚀 Deployment Commands

Container Management

# List all containers
incus list

# Start security stack
incus start crowdsec openresty openappsec caddy

# Monitor CrowdSec
incus exec crowdsec -- cscli alerts list
incus exec crowdsec -- cscli decisions list
incus exec crowdsec -- cscli metrics

Security Operations

# Ban an IP manually
incus exec crowdsec -- cscli decisions add --ip 1.2.3.4 --duration 4h --reason "manual-ban"

# Unban an IP
incus exec crowdsec -- cscli decisions delete --ip 1.2.3.4

# Check bouncer status
incus exec crowdsec -- cscli bouncers list

# View real-time logs
incus exec openresty -- tail -f /var/log/nginx/access.log

Testing Commands

# Simulate brute force attack
incus exec ab-test -- bash -c 'for i in {1..6}; do
  curl -X POST http://10.90.135.17/admin/login -d "username=admin&password=wrong$i"
  sleep 1
done'

# Test blocked IP
incus exec ab-test -- curl -X POST http://10.90.135.17/admin/login -d "test=blocked"

🔍 Monitoring & Alerting

CrowdSec Metrics

• Acquisition: Lines read, parsed, whitelisted
• Scenarios: Active scenarios, overflows, instantiations
• Decisions: Active bans, ban duration, ban reasons
• API: Bouncer heartbeats, decision pulls

Log Locations

• OpenResty: /var/log/nginx/access.log, /var/log/nginx/error.log
• CrowdSec: Built-in metrics via cscli metrics
• OpenAppSec: Container logs via incus exec openappsec -- logs

📚 Key Learnings

1. Whitelist Management: Internal IP ranges need careful configuration for testing
2. Log Format Compatibility: OpenResty logs work well with CrowdSec nginx parsers
3. Volume Sharing: Critical for log analysis between containers
4. Response Codes: 401/403 responses trigger different CrowdSec scenarios
5. Real-time Detection: Sub-minute detection and blocking capabilities

🔮 Future Enhancements

• Custom CrowdSec scenarios for application-specific attacks
• Integration with external threat intelligence feeds
• Automated incident response workflows
• Performance monitoring and optimization
• Distributed deployment across multiple nodes
• SSL/TLS termination at OpenResty layer
• Rate limiting configuration fine-tuning

---

Created: 2025-08-17
Last Updated: 2025-08-17
Architecture Version: 1.0
Status: Production Ready ✅