Initial commit: Add runbooks collection
- anvil-load-test-report.md - claude_communication_flow.md - incus-crowdsec-architecture.md - incus-meilisearch-manual.md - kitty-setup-guide.md - OpenAppSec_WAF_Operations_Guide.md - openappsec-setup-20250817.md - xray-proxy-guide.md Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
kappa
346
incus-crowdsec-architecture.md
Normal file
346
incus-crowdsec-architecture.md
Normal file
@@ -0,0 +1,346 @@
|
||||
# Multi-Layer Security Architecture with CrowdSec and OpenAppSec
|
||||
|
||||
## 🏗️ Infrastructure Overview
|
||||
|
||||
This document describes a multi-layer security architecture implemented using Incus containers, featuring CrowdSec community-driven threat intelligence and OpenAppSec ML-based WAF protection.
|
||||
|
||||
## 📋 Container Inventory
|
||||
|
||||
| Container | IP Address | Role | Technology |
|
||||
|-----------|------------|------|------------|
|
||||
| `ab-test` | 10.90.135.123 | Attack Simulation | curl/testing tools |
|
||||
| `openresty` | 10.90.135.17 | Layer 1 Security | OpenResty + CrowdSec Bouncer |
|
||||
| `crowdsec` | 10.90.135.49 | Threat Intelligence | CrowdSec Engine |
|
||||
| `openappsec` | 10.90.135.186 | Layer 2 Security | ML-based WAF |
|
||||
| `caddy` | 10.90.135.16 | Backend Server | Caddy HTTP Server |
|
||||
| `php-auth` | 10.90.135.83 | Test Endpoint | PHP Authentication Service |
|
||||
|
||||
## 🏗️ Architecture Diagram
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Incus Container Infrastructure │
|
||||
│ (10.90.135.0/24) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 🌐 External Traffic Entry Point │
|
||||
│ │
|
||||
│ ┌─────────────────┐ │
|
||||
│ │ ab-test │ ← Attack simulation container │
|
||||
│ │ 10.90.135.123 │ │
|
||||
│ └─────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 🛡️ Layer 1: Community-Driven Threat Intelligence │
|
||||
│ │
|
||||
│ ┌─────────────────┐ ┌─────────────────┐ │
|
||||
│ │ openresty │◄──►│ crowdsec │ │
|
||||
│ │ 10.90.135.17 │ │ 10.90.135.49 │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ • OpenResty │ │ • Threat Intel │ │
|
||||
│ │ • CrowdSec │ │ • Scenarios │ │
|
||||
│ │ Bouncer │ │ • Community DB │ │
|
||||
│ │ • Rate Limiting │ │ • API Server │ │
|
||||
│ │ • IP Blocking │ │ • Log Analysis │ │
|
||||
│ └─────────────────┘ └─────────────────┘ │
|
||||
│ │ │ │
|
||||
│ └──── nginx-logs ────────┘ │
|
||||
│ (Shared Volume: /var/log/nginx) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 🧠 Layer 2: ML-Based Application Security │
|
||||
│ │
|
||||
│ ┌─────────────────┐ │
|
||||
│ │ openappsec │ │
|
||||
│ │ 10.90.135.186 │ │
|
||||
│ │ │ │
|
||||
│ │ • ML WAF │ │
|
||||
│ │ • Simple Model │ │
|
||||
│ │ V1.0 │ │
|
||||
│ │ • Zero-day │ │
|
||||
│ │ Detection │ │
|
||||
│ │ • No Signatures │ │
|
||||
│ └─────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 🔗 Layer 3: Backend Application Server │
|
||||
│ │
|
||||
│ ┌─────────────────┐ │
|
||||
│ │ caddy │ │
|
||||
│ │ 10.90.135.16 │ │
|
||||
│ │ │ │
|
||||
│ │ • HTTP Server │ │
|
||||
│ │ • Reverse Proxy │ │
|
||||
│ │ • Auto HTTPS │ │
|
||||
│ └─────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 🧪 Supporting Services │
|
||||
│ │
|
||||
│ ┌─────────────────┐ │
|
||||
│ │ php-auth │ ← Authentication test endpoint │
|
||||
│ │ 10.90.135.83 │ (Returns 401 for brute force testing) │
|
||||
│ └─────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 📊 Traffic Flow & Security Processing
|
||||
|
||||
```
|
||||
🌐 Attack Request
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ openresty │ ← CrowdSec Bouncer
|
||||
│ 10.90.135.17 │ • Community threat intel
|
||||
│ │ • Behavioral analysis
|
||||
│ │ • IP reputation
|
||||
│ │ • Rate limiting
|
||||
└─────────────────┘
|
||||
│ ✅ Allowed
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ openappsec │ ← ML-based WAF
|
||||
│ 10.90.135.186 │ • Zero-day detection
|
||||
│ │ • ML pattern analysis
|
||||
│ │ • Application-layer security
|
||||
└─────────────────┘
|
||||
│ ✅ Clean
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ caddy │ ← Backend Server
|
||||
│ 10.90.135.16 │ • Content delivery
|
||||
│ │ • Application logic
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
## 🔒 Security Layers Detail
|
||||
|
||||
### Layer 1: CrowdSec Community Intelligence
|
||||
|
||||
**Container**: `openresty` + `crowdsec`
|
||||
|
||||
**Key Features**:
|
||||
- 🌍 Global threat intelligence sharing
|
||||
- 🔍 HTTP brute force detection (`LePresidente/http-generic-401-bf`)
|
||||
- 📊 Scenario-based behavioral analysis
|
||||
- ⏱️ Real-time IP blocking (4-hour default ban)
|
||||
- 🔄 Dynamic bouncer integration with OpenResty
|
||||
|
||||
**Scenarios Enabled**:
|
||||
- `crowdsecurity/http-generic-bf` - Generic HTTP brute force detection
|
||||
- `crowdsecurity/http-admin-interface-probing` - Admin interface scanning
|
||||
- `crowdsecurity/http-probing` - General HTTP probing
|
||||
- `crowdsecurity/http-cve-probing` - CVE exploitation attempts
|
||||
- `crowdsecurity/http-sqli-probing` - SQL injection attempts
|
||||
|
||||
**Configuration**:
|
||||
```yaml
|
||||
# CrowdSec Acquisition
|
||||
filenames:
|
||||
- /var/log/nginx/access.log
|
||||
- /var/log/nginx/error.log
|
||||
labels:
|
||||
type: nginx
|
||||
```
|
||||
|
||||
### Layer 2: OpenAppSec ML WAF
|
||||
|
||||
**Container**: `openappsec`
|
||||
|
||||
**Key Features**:
|
||||
- 🧠 Machine learning-based attack detection
|
||||
- 🆕 Zero-day vulnerability protection
|
||||
- 📝 No signature updates required
|
||||
- ⚡ Simple Model V1.0 for fast processing
|
||||
- 🔄 Automatic threat adaptation
|
||||
|
||||
**ML Model**: Simple Model V1.0
|
||||
- Behavioral analysis
|
||||
- Pattern recognition
|
||||
- Anomaly detection
|
||||
- Real-time learning
|
||||
|
||||
### Layer 3: Backend Application
|
||||
|
||||
**Container**: `caddy`
|
||||
|
||||
**Key Features**:
|
||||
- 🔐 Automatic HTTPS with Let's Encrypt
|
||||
- 🔄 Reverse proxy capabilities
|
||||
- 📈 High-performance HTTP/2 support
|
||||
- 🎯 Clean traffic processing
|
||||
|
||||
## 🧪 Testing Infrastructure
|
||||
|
||||
### Attack Simulation
|
||||
- **Container**: `ab-test`
|
||||
- **Purpose**: Simulate various attack patterns for testing
|
||||
- **Tools**: curl, custom scripts
|
||||
|
||||
### Authentication Endpoint
|
||||
- **Container**: `php-auth`
|
||||
- **Purpose**: Generate 401/403 responses for brute force testing
|
||||
- **Endpoint**: `/login.php` - Always returns 401 Unauthorized
|
||||
|
||||
## 🔧 Implementation Details
|
||||
|
||||
### Volume Mounts
|
||||
```bash
|
||||
# Shared nginx logs between openresty and crowdsec
|
||||
nginx-logs: /var/log/nginx
|
||||
```
|
||||
|
||||
### Network Configuration
|
||||
- **Network**: incus default bridge (10.90.135.0/24)
|
||||
- **Traffic Flow**: ab-test → openresty → openappsec → caddy
|
||||
- **Special Routes**: `/admin/login` → php-auth (testing)
|
||||
|
||||
### Security Configuration
|
||||
|
||||
#### CrowdSec Bouncer
|
||||
```yaml
|
||||
API_URL: http://10.90.135.49:8080
|
||||
API_KEY: s0ENc/6Tw+6m6tr0Qkjt/WAYU1QlC5/MzH7SQOCJX50
|
||||
```
|
||||
|
||||
#### OpenResty Configuration
|
||||
```nginx
|
||||
upstream openappsec_backend {
|
||||
server 10.90.135.186:80;
|
||||
}
|
||||
|
||||
upstream auth_backend {
|
||||
server 10.90.135.83:8080;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
access_log /var/log/nginx/access.log;
|
||||
error_log /var/log/nginx/error.log;
|
||||
|
||||
location /admin/login {
|
||||
proxy_pass http://auth_backend/login.php;
|
||||
}
|
||||
|
||||
location / {
|
||||
proxy_pass http://openappsec_backend;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## ✅ Testing Results
|
||||
|
||||
### Successful Test Cases
|
||||
|
||||
1. **Brute Force Detection**
|
||||
- ✅ 6 failed login attempts → CrowdSec alert triggered
|
||||
- ✅ IP `10.90.135.123` automatically banned
|
||||
- ✅ Scenario: `LePresidente/http-generic-401-bf`
|
||||
|
||||
2. **IP Blocking**
|
||||
- ✅ Banned IP receives `403 Forbidden`
|
||||
- ✅ Traffic completely blocked at Layer 1
|
||||
|
||||
3. **IP Unbanning**
|
||||
- ✅ Manual unban via `cscli decisions delete`
|
||||
- ✅ Immediate traffic restoration
|
||||
|
||||
4. **Multi-Layer Coordination**
|
||||
- ✅ OpenResty logs → CrowdSec analysis
|
||||
- ✅ Real-time threat intelligence updates
|
||||
- ✅ Seamless traffic flow through all layers
|
||||
|
||||
### Attack Patterns Tested
|
||||
- HTTP brute force attacks (401 responses)
|
||||
- Multiple rapid authentication attempts
|
||||
- IP-based blocking and recovery
|
||||
|
||||
## 🚀 Deployment Commands
|
||||
|
||||
### Container Management
|
||||
```bash
|
||||
# List all containers
|
||||
incus list
|
||||
|
||||
# Start security stack
|
||||
incus start crowdsec openresty openappsec caddy
|
||||
|
||||
# Monitor CrowdSec
|
||||
incus exec crowdsec -- cscli alerts list
|
||||
incus exec crowdsec -- cscli decisions list
|
||||
incus exec crowdsec -- cscli metrics
|
||||
```
|
||||
|
||||
### Security Operations
|
||||
```bash
|
||||
# Ban an IP manually
|
||||
incus exec crowdsec -- cscli decisions add --ip 1.2.3.4 --duration 4h --reason "manual-ban"
|
||||
|
||||
# Unban an IP
|
||||
incus exec crowdsec -- cscli decisions delete --ip 1.2.3.4
|
||||
|
||||
# Check bouncer status
|
||||
incus exec crowdsec -- cscli bouncers list
|
||||
|
||||
# View real-time logs
|
||||
incus exec openresty -- tail -f /var/log/nginx/access.log
|
||||
```
|
||||
|
||||
### Testing Commands
|
||||
```bash
|
||||
# Simulate brute force attack
|
||||
incus exec ab-test -- bash -c 'for i in {1..6}; do
|
||||
curl -X POST http://10.90.135.17/admin/login -d "username=admin&password=wrong$i"
|
||||
sleep 1
|
||||
done'
|
||||
|
||||
# Test blocked IP
|
||||
incus exec ab-test -- curl -X POST http://10.90.135.17/admin/login -d "test=blocked"
|
||||
```
|
||||
|
||||
## 🔍 Monitoring & Alerting
|
||||
|
||||
### CrowdSec Metrics
|
||||
- **Acquisition**: Lines read, parsed, whitelisted
|
||||
- **Scenarios**: Active scenarios, overflows, instantiations
|
||||
- **Decisions**: Active bans, ban duration, ban reasons
|
||||
- **API**: Bouncer heartbeats, decision pulls
|
||||
|
||||
### Log Locations
|
||||
- **OpenResty**: `/var/log/nginx/access.log`, `/var/log/nginx/error.log`
|
||||
- **CrowdSec**: Built-in metrics via `cscli metrics`
|
||||
- **OpenAppSec**: Container logs via `incus exec openappsec -- logs`
|
||||
|
||||
## 📚 Key Learnings
|
||||
|
||||
1. **Whitelist Management**: Internal IP ranges need careful configuration for testing
|
||||
2. **Log Format Compatibility**: OpenResty logs work well with CrowdSec nginx parsers
|
||||
3. **Volume Sharing**: Critical for log analysis between containers
|
||||
4. **Response Codes**: 401/403 responses trigger different CrowdSec scenarios
|
||||
5. **Real-time Detection**: Sub-minute detection and blocking capabilities
|
||||
|
||||
## 🔮 Future Enhancements
|
||||
|
||||
- [ ] Custom CrowdSec scenarios for application-specific attacks
|
||||
- [ ] Integration with external threat intelligence feeds
|
||||
- [ ] Automated incident response workflows
|
||||
- [ ] Performance monitoring and optimization
|
||||
- [ ] Distributed deployment across multiple nodes
|
||||
- [ ] SSL/TLS termination at OpenResty layer
|
||||
- [ ] Rate limiting configuration fine-tuning
|
||||
|
||||
---
|
||||
|
||||
**Created**: 2025-08-17
|
||||
**Last Updated**: 2025-08-17
|
||||
**Architecture Version**: 1.0
|
||||
**Status**: Production Ready ✅
|
||||
Reference in New Issue
Block a user