dropshell-templates/logserver/TODO.md

# LogServer Template - Implementation TODO

## Phase 1: Core Infrastructure (Priority 1)

### Configuration Files
- [ ] Create `config/.template_info.env` with template metadata
- [ ] Create `config/service.env` with user-configurable settings
- [ ] Define all required environment variables (ports, passwords, heap sizes)
- [ ] Set appropriate default values for zero-config experience

### Docker Compose Setup
- [ ] Create `docker-compose.yml` with ELK stack services
- [ ] Configure Elasticsearch single-node setup
- [ ] Configure Logstash with Beats input pipeline
- [ ] Configure Kibana with Elasticsearch connection
- [ ] Set up proper networking between services
- [ ] Define named volumes for data persistence
- [ ] Configure health checks for each service

### Required Scripts
- [ ] Implement `install.sh` - Pull images, create volumes, start services
- [ ] Implement `uninstall.sh` - Stop and remove containers (preserve volumes!)
- [ ] Implement `start.sh` - Start all ELK services with docker-compose
- [ ] Implement `stop.sh` - Gracefully stop all services
- [ ] Implement `status.sh` - Check health of all three services

## Phase 2: Logstash Configuration (Priority 1)

### Input Configuration
- [ ] Configure Beats input on port 5044 with TLS/SSL
- [ ] Set up mutual TLS (mTLS) authentication
- [ ] Configure client certificate validation
- [ ] Add API key authentication option
- [ ] Implement IP whitelisting
- [ ] Add Syslog input on port 514 (UDP/TCP) - unauthenticated
- [ ] Add Docker Fluentd input on port 24224 (optional)

### Filter Pipeline
- [ ] Create Docker log parser (extract container metadata)
- [ ] Create Syslog parser (RFC3164 and RFC5424)
- [ ] Add JSON parser for structured logs
- [ ] Implement multiline pattern for stack traces
- [ ] Add timestamp extraction and normalization
- [ ] Create field enrichment (add host metadata)
- [ ] Implement conditional routing based on log type

### Output Configuration
- [ ] Configure Elasticsearch output with index patterns
- [ ] Set up index templates for different log types
- [ ] Configure index lifecycle management (ILM)

## Phase 3: Elasticsearch Setup (Priority 1)

### System Configuration
- [ ] Set appropriate heap size defaults (ES_HEAP_SIZE)
- [ ] Configure vm.max_map_count requirement check
- [ ] Set up single-node discovery settings
- [ ] Configure data persistence volume
- [ ] Set up index templates for:
  - [ ] Docker logs (docker-*)
  - [ ] System logs (syslog-*)
  - [ ] Application logs (app-*)
  - [ ] Error logs (errors-*)

### Index Management
- [ ] Configure ILM policies for log rotation
- [ ] Set retention period (default 30 days)
- [ ] Configure max index size limits
- [ ] Set up automatic cleanup of old indices
- [ ] Create snapshot repository configuration

## Phase 4: Kibana Configuration (Priority 2)

### Initial Setup
- [ ] Configure Kibana with Elasticsearch URL
- [ ] Set up basic authentication
- [ ] Configure server base path
- [ ] Set appropriate memory limits

### Pre-built Dashboards
- [ ] Create System Overview dashboard
- [ ] Create Docker Containers dashboard
- [ ] Create Error Analysis dashboard
- [ ] Create Security Events dashboard
- [ ] Create Host Metrics dashboard

### Saved Searches
- [ ] Error logs across all sources
- [ ] Authentication events
- [ ] Container lifecycle events
- [ ] Slow queries/performance issues
- [ ] Critical system events

### Index Patterns
- [ ] Configure docker-* pattern
- [ ] Configure syslog-* pattern
- [ ] Configure app-* pattern
- [ ] Configure filebeat-* pattern

## Phase 5: Optional Scripts (Priority 2)

### Operational Scripts
- [ ] Implement `logs.sh` - Show logs from all ELK services
- [ ] Implement `backup.sh` - Snapshot Elasticsearch indices
- [ ] Implement `restore.sh` - Restore from snapshots
- [ ] Implement `destroy.sh` - Complete removal including volumes
- [ ] Implement `ports.sh` - Display all exposed ports
- [ ] Implement `ssh.sh` - Shell into specific container

### Helper Scripts
- [ ] Create `_volumes.sh` for volume management helpers
- [ ] Add health check script for all services
- [ ] Create performance tuning script
- [ ] Add certificate generation script for SSL

## Phase 6: Security Features (Priority 1 - CRITICAL)

### Certificate Authority Setup
- [ ] Create CA certificate and key for signing client certs
- [ ] Generate server certificate for Logstash
- [ ] Create certificate generation script for clients
- [ ] Set up certificate storage structure
- [ ] Implement certificate rotation mechanism

### mTLS Authentication
- [ ] Configure Logstash for mutual TLS
- [ ] Set up client certificate validation
- [ ] Create client certificate generation script
- [ ] Implement certificate revocation list (CRL)
- [ ] Add certificate expiry monitoring

### API Key Authentication
- [ ] Create API key generation script
- [ ] Configure Logstash to accept API keys
- [ ] Implement API key storage (encrypted)
- [ ] Add API key rotation mechanism
- [ ] Create API key revocation process

### Network Security
- [ ] Implement IP whitelisting in Logstash
- [ ] Configure firewall rules
- [ ] Set up rate limiting
- [ ] Add connection throttling
- [ ] Implement DDoS protection

### Kibana Security
- [ ] Configure Kibana HTTPS
- [ ] Set up basic authentication
- [ ] Create user management scripts
- [ ] Implement session management
- [ ] Add audit logging

## Phase 7: Performance & Optimization (Priority 3)

### Resource Management
- [ ] Configure CPU limits for each service
- [ ] Set memory limits appropriately
- [ ] Add swap handling configuration
- [ ] Configure JVM options files
- [ ] Add performance monitoring

### Optimization
- [ ] Configure pipeline workers
- [ ] Set batch sizes for optimal throughput
- [ ] Configure queue sizes
- [ ] Add caching configuration
- [ ] Optimize index refresh intervals

## Phase 8: Testing & Documentation (Priority 3)

### Testing
- [ ] Test installation process
- [ ] Test uninstall (verify volume preservation)
- [ ] Test log ingestion from sample client
- [ ] Test all dashboard functionality
- [ ] Test backup and restore procedures
- [ ] Load test with high log volume
- [ ] Test failover and recovery

### Documentation
- [ ] Create README.txt for dropshell format
- [ ] Document all configuration options
- [ ] Add troubleshooting guide
- [ ] Create quick start guide
- [ ] Document upgrade procedures
- [ ] Add performance tuning guide

## Phase 9: Integration Testing (Priority 3)

### With LogClient
- [ ] Test automatic discovery
- [ ] Verify log flow from client to server
- [ ] Test reconnection scenarios
- [ ] Verify all log types are parsed correctly
- [ ] Test SSL communication
- [ ] Measure end-to-end latency

### Compatibility Testing
- [ ] Test with different Docker versions
- [ ] Test on various Linux distributions
- [ ] Verify with different log formats
- [ ] Test with high-volume producers
- [ ] Validate resource usage

## Phase 10: Production Readiness (Priority 4)

### Monitoring & Alerting
- [ ] Add Elasticsearch monitoring
- [ ] Configure disk space alerts
- [ ] Set up index health monitoring
- [ ] Add performance metrics collection
- [ ] Create alert rules in Kibana

### Maintenance Features
- [ ] Add automatic update check
- [ ] Create maintenance mode
- [ ] Add data export functionality
- [ ] Create migration scripts
- [ ] Add configuration validation

## Notes

### Design Principles
1. **Minimum configuration**: Should work with just `dropshell install logserver`
2. **Data safety**: Never delete volumes in uninstall.sh
3. **Non-interactive**: All scripts must run without user input
4. **Idempotent**: Scripts can be run multiple times safely
5. **Clear feedback**: Provide clear status and error messages

### Dependencies
- Docker and Docker Compose
- Sufficient system resources (4GB+ RAM recommended)
- Network connectivity for clients
- Persistent storage for logs

### Testing Checklist
- [ ] All required scripts present and executable
- [ ] Template validates with dropshell test-template
- [ ] Services start and connect properly
- [ ] Logs flow from client to Kibana
- [ ] Data persists across container restarts
- [ ] Uninstall preserves data volumes
- [ ] Resource limits are enforced
- [ ] Error handling works correctly