runbooks/incus-crowdsec-architecture.md

# Multi-Layer Security Architecture with CrowdSec and OpenAppSec

## 🏗️ Infrastructure Overview

This document describes a multi-layer security architecture implemented using Incus containers, featuring CrowdSec community-driven threat intelligence and OpenAppSec ML-based WAF protection.

## 📋 Container Inventory

| Container | IP Address | Role | Technology |
|-----------|------------|------|------------|
| `ab-test` | 10.90.135.123 | Attack Simulation | curl/testing tools |
| `openresty` | 10.90.135.17 | Layer 1 Security | OpenResty + CrowdSec Bouncer |
| `crowdsec` | 10.90.135.49 | Threat Intelligence | CrowdSec Engine |
| `openappsec` | 10.90.135.186 | Layer 2 Security | ML-based WAF |
| `caddy` | 10.90.135.16 | Backend Server | Caddy HTTP Server |
| `php-auth` | 10.90.135.83 | Test Endpoint | PHP Authentication Service |

## 🏗️ Architecture Diagram

```
┌─────────────────────────────────────────────────────────────────┐
│                    Incus Container Infrastructure                │
│                          (10.90.135.0/24)                     │
└─────────────────────────────────────────────────────────────────┘
                                    │
┌─────────────────────────────────────────────────────────────────┐
│  🌐 External Traffic Entry Point                               │
│                                                                 │
│  ┌─────────────────┐                                           │
│  │   ab-test       │  ← Attack simulation container            │
│  │ 10.90.135.123   │                                           │
│  └─────────────────┘                                           │
└─────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────┐
│  🛡️ Layer 1: Community-Driven Threat Intelligence             │
│                                                                 │
│  ┌─────────────────┐    ┌─────────────────┐                   │
│  │   openresty     │◄──►│    crowdsec     │                   │
│  │ 10.90.135.17    │    │ 10.90.135.49    │                   │
│  │                 │    │                 │                   │
│  │ • OpenResty     │    │ • Threat Intel  │                   │
│  │ • CrowdSec      │    │ • Scenarios     │                   │
│  │   Bouncer       │    │ • Community DB  │                   │
│  │ • Rate Limiting │    │ • API Server    │                   │
│  │ • IP Blocking   │    │ • Log Analysis  │                   │
│  └─────────────────┘    └─────────────────┘                   │
│           │                       │                           │
│           └──── nginx-logs ────────┘                           │
│           (Shared Volume: /var/log/nginx)                     │
└─────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────┐
│  🧠 Layer 2: ML-Based Application Security                     │
│                                                                 │
│  ┌─────────────────┐                                           │
│  │   openappsec    │                                           │
│  │ 10.90.135.186   │                                           │
│  │                 │                                           │
│  │ • ML WAF        │                                           │
│  │ • Simple Model  │                                           │
│  │   V1.0          │                                           │
│  │ • Zero-day      │                                           │
│  │   Detection     │                                           │
│  │ • No Signatures │                                           │
│  └─────────────────┘                                           │
└─────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────┐
│  🔗 Layer 3: Backend Application Server                        │
│                                                                 │
│  ┌─────────────────┐                                           │
│  │     caddy       │                                           │
│  │ 10.90.135.16    │                                           │
│  │                 │                                           │
│  │ • HTTP Server   │                                           │
│  │ • Reverse Proxy │                                           │
│  │ • Auto HTTPS    │                                           │
│  └─────────────────┘                                           │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  🧪 Supporting Services                                         │
│                                                                 │
│  ┌─────────────────┐                                           │
│  │    php-auth     │  ← Authentication test endpoint           │
│  │ 10.90.135.83    │    (Returns 401 for brute force testing) │
│  └─────────────────┘                                           │
└─────────────────────────────────────────────────────────────────┘
```

## 📊 Traffic Flow & Security Processing

```
🌐 Attack Request
        │
        ▼
┌─────────────────┐
│   openresty     │ ← CrowdSec Bouncer
│ 10.90.135.17    │   • Community threat intel
│                 │   • Behavioral analysis
│                 │   • IP reputation
│                 │   • Rate limiting
└─────────────────┘
        │ ✅ Allowed
        ▼
┌─────────────────┐
│   openappsec    │ ← ML-based WAF
│ 10.90.135.186   │   • Zero-day detection
│                 │   • ML pattern analysis
│                 │   • Application-layer security
└─────────────────┘
        │ ✅ Clean
        ▼
┌─────────────────┐
│     caddy       │ ← Backend Server
│ 10.90.135.16    │   • Content delivery
│                 │   • Application logic
└─────────────────┘
```

## 🔒 Security Layers Detail

### Layer 1: CrowdSec Community Intelligence

**Container**: `openresty` + `crowdsec`

**Key Features**:
- 🌍 Global threat intelligence sharing
- 🔍 HTTP brute force detection (`LePresidente/http-generic-401-bf`)
- 📊 Scenario-based behavioral analysis
- ⏱️ Real-time IP blocking (4-hour default ban)
- 🔄 Dynamic bouncer integration with OpenResty

**Scenarios Enabled**:
- `crowdsecurity/http-generic-bf` - Generic HTTP brute force detection
- `crowdsecurity/http-admin-interface-probing` - Admin interface scanning
- `crowdsecurity/http-probing` - General HTTP probing
- `crowdsecurity/http-cve-probing` - CVE exploitation attempts
- `crowdsecurity/http-sqli-probing` - SQL injection attempts

**Configuration**:
```yaml
# CrowdSec Acquisition
filenames:
  - /var/log/nginx/access.log
  - /var/log/nginx/error.log
labels:
  type: nginx
```

**Acquisition 디렉토리 구조화** (권장):
```bash
# /etc/crowdsec/acquis.d/ 디렉토리에 서비스별 파일 분리
/etc/crowdsec/acquis.d/
├── nginx.yaml      # Nginx 로그
├── apache.yaml     # Apache 로그 (필요시)
├── sshd.yaml       # SSH 로그
└── traefik.yaml    # Traefik 로그 (필요시)
```

**nginx.yaml 예시**:
```yaml
filenames:
  - /var/log/nginx/access.log
  - /var/log/nginx/error.log
labels:
  type: nginx
---
# JSON 형식 로그 (별도 처리)
filenames:
  - /var/log/nginx/json_access.log
labels:
  type: nginx
  format: json
```

**sshd.yaml 예시**:
```yaml
filenames:
  - /var/log/auth.log
labels:
  type: syslog
```

### Layer 2: OpenAppSec ML WAF

**Container**: `openappsec`

**Key Features**:
- 🧠 Machine learning-based attack detection
- 🆕 Zero-day vulnerability protection
- 📝 No signature updates required
- ⚡ Simple Model V1.0 for fast processing
- 🔄 Automatic threat adaptation

**ML Model**: Simple Model V1.0
- Behavioral analysis
- Pattern recognition
- Anomaly detection
- Real-time learning

### Layer 3: Backend Application

**Container**: `caddy`

**Key Features**:
- 🔐 Automatic HTTPS with Let's Encrypt
- 🔄 Reverse proxy capabilities
- 📈 High-performance HTTP/2 support
- 🎯 Clean traffic processing

## 🧪 Testing Infrastructure

### Attack Simulation
- **Container**: `ab-test`
- **Purpose**: Simulate various attack patterns for testing
- **Tools**: curl, custom scripts

### Authentication Endpoint
- **Container**: `php-auth`
- **Purpose**: Generate 401/403 responses for brute force testing
- **Endpoint**: `/login.php` - Always returns 401 Unauthorized

## 🔧 Implementation Details

### Volume Mounts
```bash
# Shared nginx logs between openresty and crowdsec
nginx-logs: /var/log/nginx
```

### Network Configuration
- **Network**: incus default bridge (10.90.135.0/24)
- **Traffic Flow**: ab-test → openresty → openappsec → caddy
- **Special Routes**: `/admin/login` → php-auth (testing)

### Security Configuration

#### CrowdSec Bouncer (수동 등록)
```yaml
API_URL: http://10.90.135.49:8080
API_KEY: s0ENc/6Tw+6m6tr0Qkjt/WAYU1QlC5/MzH7SQOCJX50
```

#### Bouncer 자동 등록 (권장)

> 컨테이너 시작 시 환경변수로 Bouncer 자동 등록. 수동 `cscli bouncers add` 불필요.

**환경변수 방식**:
```bash
# CrowdSec 컨테이너 시작 시 Bouncer 자동 등록
incus launch docker.io/crowdsecurity/crowdsec crowdsec \
  --config environment.BOUNCER_KEY_openresty=my-secure-bouncer-key-here \
  --config environment.BOUNCER_KEY_firewall=another-bouncer-key-here
```

**Docker Compose 예시**:
```yaml
services:
  crowdsec:
    image: crowdsecurity/crowdsec:latest
    environment:
      # Bouncer 자동 등록 (BOUNCER_KEY_<name>=<key>)
      - BOUNCER_KEY_openresty=${OPENRESTY_BOUNCER_KEY}
      - BOUNCER_KEY_firewall=${FIREWALL_BOUNCER_KEY}
      # 컬렉션 자동 설치
      - COLLECTIONS=crowdsecurity/nginx crowdsecurity/http-cve
    volumes:
      - crowdsec-data:/var/lib/crowdsec/data
      - crowdsec-config:/etc/crowdsec
      - /var/log/nginx:/var/log/nginx:ro
    restart: unless-stopped

  openresty:
    image: openresty/openresty:latest
    environment:
      - CROWDSEC_API_URL=http://crowdsec:8080
      - CROWDSEC_API_KEY=${OPENRESTY_BOUNCER_KEY}
    depends_on:
      - crowdsec
```

**Docker Secret 방식 (더 안전)**:
```yaml
services:
  crowdsec:
    image: crowdsecurity/crowdsec:latest
    secrets:
      - bouncer_key_openresty
      - bouncer_key_firewall
    # Secret 파일은 /run/secrets/bouncer_key_<name> 으로 마운트됨

secrets:
  bouncer_key_openresty:
    file: ./secrets/openresty_bouncer_key
  bouncer_key_firewall:
    file: ./secrets/firewall_bouncer_key
```

```bash
# Secret 파일 생성
mkdir -p secrets
openssl rand -base64 32 > secrets/openresty_bouncer_key
openssl rand -base64 32 > secrets/firewall_bouncer_key
chmod 600 secrets/*
```

> **주의**: 환경변수/Secret 방식은 초기 배포 시에만 동작. 기존 Bouncer 업데이트는 `cscli` 사용.

#### OpenResty Configuration
```nginx
upstream openappsec_backend {
    server 10.90.135.186:80;
}

upstream auth_backend {
    server 10.90.135.83:8080;
}

server {
    listen 80;
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;
    
    location /admin/login {
        proxy_pass http://auth_backend/login.php;
    }
    
    location / {
        proxy_pass http://openappsec_backend;
    }
}
```

## ✅ Testing Results

### Successful Test Cases

1. **Brute Force Detection**
   - ✅ 6 failed login attempts → CrowdSec alert triggered
   - ✅ IP `10.90.135.123` automatically banned
   - ✅ Scenario: `LePresidente/http-generic-401-bf`

2. **IP Blocking**
   - ✅ Banned IP receives `403 Forbidden`
   - ✅ Traffic completely blocked at Layer 1

3. **IP Unbanning**
   - ✅ Manual unban via `cscli decisions delete`
   - ✅ Immediate traffic restoration

4. **Multi-Layer Coordination**
   - ✅ OpenResty logs → CrowdSec analysis
   - ✅ Real-time threat intelligence updates
   - ✅ Seamless traffic flow through all layers

### Attack Patterns Tested
- HTTP brute force attacks (401 responses)
- Multiple rapid authentication attempts
- IP-based blocking and recovery

## 🚀 Deployment Commands

### Container Management
```bash
# List all containers
incus list

# Start security stack
incus start crowdsec openresty openappsec caddy

# Monitor CrowdSec
incus exec crowdsec -- cscli alerts list
incus exec crowdsec -- cscli decisions list
incus exec crowdsec -- cscli metrics
```

### Security Operations
```bash
# Ban an IP manually
incus exec crowdsec -- cscli decisions add --ip 1.2.3.4 --duration 4h --reason "manual-ban"

# Unban an IP
incus exec crowdsec -- cscli decisions delete --ip 1.2.3.4

# Check bouncer status
incus exec crowdsec -- cscli bouncers list

# View real-time logs
incus exec openresty -- tail -f /var/log/nginx/access.log
```

### Testing Commands
```bash
# Simulate brute force attack
incus exec ab-test -- bash -c 'for i in {1..6}; do
  curl -X POST http://10.90.135.17/admin/login -d "username=admin&password=wrong$i"
  sleep 1
done'

# Test blocked IP
incus exec ab-test -- curl -X POST http://10.90.135.17/admin/login -d "test=blocked"
```

## 🔍 Monitoring & Alerting

### CrowdSec Metrics
- **Acquisition**: Lines read, parsed, whitelisted
- **Scenarios**: Active scenarios, overflows, instantiations
- **Decisions**: Active bans, ban duration, ban reasons
- **API**: Bouncer heartbeats, decision pulls

### Log Locations
- **OpenResty**: `/var/log/nginx/access.log`, `/var/log/nginx/error.log`
- **CrowdSec**: Built-in metrics via `cscli metrics`
- **OpenAppSec**: Container logs via `incus exec openappsec -- logs`

## 📚 Key Learnings

1. **Whitelist Management**: Internal IP ranges need careful configuration for testing
2. **Log Format Compatibility**: OpenResty logs work well with CrowdSec nginx parsers
3. **Volume Sharing**: Critical for log analysis between containers
4. **Response Codes**: 401/403 responses trigger different CrowdSec scenarios
5. **Real-time Detection**: Sub-minute detection and blocking capabilities

## 🔮 Future Enhancements

- [ ] Custom CrowdSec scenarios for application-specific attacks
- [ ] Integration with external threat intelligence feeds
- [ ] Automated incident response workflows
- [ ] Performance monitoring and optimization
- [ ] Distributed deployment across multiple nodes
- [ ] SSL/TLS termination at OpenResty layer
- [ ] Rate limiting configuration fine-tuning

---

**Created**: 2025-08-17  
**Last Updated**: 2025-08-17  
**Architecture Version**: 1.0  
**Status**: Production Ready ✅