dropshell-templates/logserver/TODO.md

LogServer Template - Implementation TODO

Phase 1: Core Infrastructure (Priority 1)

Configuration Files

• Create config/.template_info.env with template metadata
• Create config/service.env with user-configurable settings
• Define all required environment variables (ports, passwords, heap sizes)
• Set appropriate default values for zero-config experience

Docker Compose Setup

• Create docker-compose.yml with ELK stack services
• Configure Elasticsearch single-node setup
• Configure Logstash with Beats input pipeline
• Configure Kibana with Elasticsearch connection
• Set up proper networking between services
• Define named volumes for data persistence
• Configure health checks for each service

Required Scripts

• Implement install.sh - Pull images, create volumes, start services
• Implement uninstall.sh - Stop and remove containers (preserve volumes!)
• Implement start.sh - Start all ELK services with docker-compose
• Implement stop.sh - Gracefully stop all services
• Implement status.sh - Check health of all three services

Phase 2: Logstash Configuration (Priority 1)

Input Configuration

• Configure Beats input on port 5044 with TLS/SSL
• Set up mutual TLS (mTLS) authentication
• Configure client certificate validation
• Add API key authentication option
• Implement IP whitelisting
• Add Syslog input on port 514 (UDP/TCP) - unauthenticated
• Add Docker Fluentd input on port 24224 (optional)

Filter Pipeline

• Create Docker log parser (extract container metadata)
• Create Syslog parser (RFC3164 and RFC5424)
• Add JSON parser for structured logs
• Implement multiline pattern for stack traces
• Add timestamp extraction and normalization
• Create field enrichment (add host metadata)
• Implement conditional routing based on log type

Output Configuration

• Configure Elasticsearch output with index patterns
• Set up index templates for different log types
• Configure index lifecycle management (ILM)

Phase 3: Elasticsearch Setup (Priority 1)

System Configuration

• Set appropriate heap size defaults (ES_HEAP_SIZE)
• Configure vm.max_map_count requirement check
• Set up single-node discovery settings
• Configure data persistence volume
• Set up index templates for:
  • Docker logs (docker-*)
  • System logs (syslog-*)
  • Application logs (app-*)
  • Error logs (errors-*)

Index Management

• Configure ILM policies for log rotation
• Set retention period (default 30 days)
• Configure max index size limits
• Set up automatic cleanup of old indices
• Create snapshot repository configuration

Phase 4: Kibana Configuration (Priority 2)

Initial Setup

• Configure Kibana with Elasticsearch URL
• Set up basic authentication
• Configure server base path
• Set appropriate memory limits

Pre-built Dashboards

• Create System Overview dashboard
• Create Docker Containers dashboard
• Create Error Analysis dashboard
• Create Security Events dashboard
• Create Host Metrics dashboard

Saved Searches

• Error logs across all sources
• Authentication events
• Container lifecycle events
• Slow queries/performance issues
• Critical system events

Index Patterns

• Configure docker-* pattern
• Configure syslog-* pattern
• Configure app-* pattern
• Configure filebeat-* pattern

Phase 5: Optional Scripts (Priority 2)

Operational Scripts

• Implement logs.sh - Show logs from all ELK services
• Implement backup.sh - Snapshot Elasticsearch indices
• Implement restore.sh - Restore from snapshots
• Implement destroy.sh - Complete removal including volumes
• Implement ports.sh - Display all exposed ports
• Implement ssh.sh - Shell into specific container

Helper Scripts

• Create _volumes.sh for volume management helpers
• Add health check script for all services
• Create performance tuning script
• Add certificate generation script for SSL

Phase 6: Security Features (Priority 1 - CRITICAL)

Certificate Authority Setup

• Create CA certificate and key for signing client certs
• Generate server certificate for Logstash
• Create certificate generation script for clients
• Set up certificate storage structure
• Implement certificate rotation mechanism

mTLS Authentication

• Configure Logstash for mutual TLS
• Set up client certificate validation
• Create client certificate generation script
• Implement certificate revocation list (CRL)
• Add certificate expiry monitoring

API Key Authentication

• Create API key generation script
• Configure Logstash to accept API keys
• Implement API key storage (encrypted)
• Add API key rotation mechanism
• Create API key revocation process

Network Security

• Implement IP whitelisting in Logstash
• Configure firewall rules
• Set up rate limiting
• Add connection throttling
• Implement DDoS protection

Kibana Security

• Configure Kibana HTTPS
• Set up basic authentication
• Create user management scripts
• Implement session management
• Add audit logging

Phase 7: Performance & Optimization (Priority 3)

Resource Management

• Configure CPU limits for each service
• Set memory limits appropriately
• Add swap handling configuration
• Configure JVM options files
• Add performance monitoring

Optimization

• Configure pipeline workers
• Set batch sizes for optimal throughput
• Configure queue sizes
• Add caching configuration
• Optimize index refresh intervals

Phase 8: Testing & Documentation (Priority 3)

Testing

• Test installation process
• Test uninstall (verify volume preservation)
• Test log ingestion from sample client
• Test all dashboard functionality
• Test backup and restore procedures
• Load test with high log volume
• Test failover and recovery

Documentation

• Create README.txt for dropshell format
• Document all configuration options
• Add troubleshooting guide
• Create quick start guide
• Document upgrade procedures
• Add performance tuning guide

Phase 9: Integration Testing (Priority 3)

With LogClient

• Test automatic discovery
• Verify log flow from client to server
• Test reconnection scenarios
• Verify all log types are parsed correctly
• Test SSL communication
• Measure end-to-end latency

Compatibility Testing

• Test with different Docker versions
• Test on various Linux distributions
• Verify with different log formats
• Test with high-volume producers
• Validate resource usage

Phase 10: Production Readiness (Priority 4)

Monitoring & Alerting

• Add Elasticsearch monitoring
• Configure disk space alerts
• Set up index health monitoring
• Add performance metrics collection
• Create alert rules in Kibana

Maintenance Features

• Add automatic update check
• Create maintenance mode
• Add data export functionality
• Create migration scripts
• Add configuration validation

Notes

Design Principles

1. Minimum configuration: Should work with just dropshell install logserver
2. Data safety: Never delete volumes in uninstall.sh
3. Non-interactive: All scripts must run without user input
4. Idempotent: Scripts can be run multiple times safely
5. Clear feedback: Provide clear status and error messages

Dependencies

• Docker and Docker Compose
• Sufficient system resources (4GB+ RAM recommended)
• Network connectivity for clients
• Persistent storage for logs

Testing Checklist

• All required scripts present and executable
• Template validates with dropshell test-template
• Services start and connect properly
• Logs flow from client to Kibana
• Data persists across container restarts
• Uninstall preserves data volumes
• Resource limits are enforced
• Error handling works correctly