dropshell-templates/logclient/TODO.md

LogClient Template - Implementation TODO

Phase 1: Core Infrastructure (Priority 1)

Configuration Files

• Create config/.template_info.env with template metadata
• Create config/service.env with minimal required settings
• Define LOGSERVER_HOST and LOGSERVER_PORT variables
• Add AUTH_MODE variable (mtls, apikey, basic)
• Add certificate/key path variables for mTLS
• Add API_KEY variable for API key auth
• Add USERNAME/PASSWORD for basic auth
• Add optional performance and filtering variables
• Set sensible defaults where possible

Filebeat Configuration

• Create base filebeat.yml configuration template
• Configure Docker input using Docker API (not autodiscover with hints)
• Set containers.ids: ["*"] to collect from all containers
• Set up system log inputs for host logs
• Configure output to Logstash
• Add error handling and retry logic
• Set up local disk buffering
• Configure stream: "all" to get both stdout and stderr

Required Scripts

• Implement install.sh - Pull Filebeat image, configure auth, start
• Implement uninstall.sh - Stop and remove container (preserve config and certs)
• Implement start.sh - Start Filebeat with auth config and proper mounts
• Implement stop.sh - Gracefully stop Filebeat
• Implement status.sh - Check Filebeat health and auth status
• Create setup-auth.sh - Helper script to configure authentication

Phase 2: Docker API Log Collection (Priority 1)

Docker API Input Configuration

• Configure Docker input type (NOT autodiscover, use direct Docker input)
• Mount Docker socket (/var/run/docker.sock) with proper permissions
• Configure Docker API endpoint (unix:///var/run/docker.sock)
• Set up real-time log streaming from Docker daemon
• Enable collection from ALL logging drivers (local, json-file, journald, etc.)
• Configure since_time to get recent logs on startup

Container Metadata Extraction

• Extract container name, ID, image name, and image tag
• Map container labels to fields
• Handle docker-compose project names and service names
• Add container state information
• Include container environment variables (filtered)
• Handle container lifecycle events (start, stop, restart)

Container Filtering

• Implement include/exclude by container name patterns
• Add label-based filtering (containers.labels)
• Create ignore patterns for system containers
• Add support for custom filter expressions
• Configure combine_partial to handle partial log lines
• Document filtering examples with Docker API syntax

Phase 3: System Log Collection (Priority 1)

Log File Inputs

• Configure /var/log/syslog or /var/log/messages
• Add /var/log/auth.log or /var/log/secure
• Include /var/log/kern.log
• Monitor /var/log/dpkg.log or /var/log/yum.log
• Add custom log path support via environment variable

Journald Integration

• Detect if systemd/journald is available
• Configure journald input if present
• Set up unit filtering
• Extract systemd metadata
• Handle binary journal format

Log Parsing

• Configure syslog parsing
• Extract severity levels
• Parse timestamps correctly
• Handle different syslog formats
• Add timezone handling

Phase 4: Output Configuration (Priority 1)

Logstash Output

• Configure primary Logstash endpoint
• Set up connection parameters (timeout, retry)
• Configure bulk operations settings
• Add compression support
• Implement backpressure handling

Connection Management

• Configure automatic reconnection
• Set exponential backoff for retries
• Add connection pooling
• Configure keepalive settings
• Handle DNS resolution failures

Authentication Configuration (Priority 1 - CRITICAL)

• Implement mTLS authentication support
• Configure client certificate and key loading
• Add CA certificate validation
• Implement API key authentication
• Add basic auth as fallback option
• Create authentication mode selection logic
• Handle authentication failures gracefully
• Add certificate expiry checking
• Implement secure credential storage
• Document authentication setup process

Phase 5: Reliability Features (Priority 2)

Local Buffering

• Configure disk queue for reliability
• Set queue size limits
• Configure memory queue settings
• Add overflow handling
• Set up automatic cleanup of old events

Error Handling

• Add retry logic for failed sends
• Configure dead letter queue
• Add circuit breaker pattern
• Log transmission errors appropriately
• Add metrics for monitoring failures

Performance Optimization

• Configure worker count
• Set batch size for sending
• Add compression level setting
• Configure CPU and memory limits
• Optimize for high-volume scenarios

Phase 6: Optional Scripts (Priority 2)

Operational Scripts

• Implement logs.sh - Show Filebeat logs
• Implement destroy.sh - Complete removal
• Implement ssh.sh - Shell into Filebeat container
• Create test.sh - Test connectivity to server
• Add metrics.sh - Show Filebeat statistics

Diagnostic Scripts

• Create connectivity test script
• Add configuration validation script
• Create debug mode enabler
• Add log sampling script
• Create performance benchmark script

Phase 7: Monitoring & Health (Priority 2)

Health Checks

• Configure Filebeat HTTP endpoint
• Add Docker health check
• Monitor queue status
• Check connection to Logstash
• Track dropped events

Metrics Collection

• Enable Filebeat monitoring
• Export metrics endpoint
• Track events sent/failed
• Monitor resource usage
• Add performance counters

Status Reporting

• Implement detailed status in status.sh
• Show connection state
• Display queue status
• Report recent errors
• Show throughput metrics

Phase 8: Advanced Features (Priority 3)

Processors

• Add field renaming processor
• Configure drop_event conditions
• Add rate limiting processor
• Include fingerprinting for deduplication
• Add custom field enrichment

Multiline Handling

• Configure patterns for common languages
• Java stack trace handling
• Python traceback handling
• Go panic handling
• Custom pattern support via environment

Field Management

• Configure field inclusion/exclusion
• Add custom fields via environment
• Set up field type conversions
• Add timestamp parsing
• Configure field aliasing

Phase 9: Testing (Priority 3)

Unit Testing

• Test configuration generation
• Verify volume mounts
• Test environment variable substitution
• Validate filtering logic
• Test error conditions

Integration Testing

• Test with logserver template
• Verify Docker log collection
• Test system log collection
• Validate SSL connectivity
• Test reconnection scenarios
• Verify buffering during outages

Load Testing

• Test with high log volume
• Measure resource usage
• Test queue overflow handling
• Verify rate limiting
• Benchmark throughput

Phase 10: Documentation (Priority 3)

User Documentation

• Create README.txt for dropshell
• Document all configuration options
• Add troubleshooting guide
• Create quick start guide
• Add FAQ section

Configuration Examples

• Minimal configuration example
• High-volume configuration
• Secure SSL configuration
• Filtered configuration
• Custom paths configuration

Integration Guides

• Integration with logserver
• Docker Compose examples
• Kubernetes DaemonSet example
• Swarm mode configuration
• Custom application integration

Phase 11: Production Readiness (Priority 4)

Security Hardening

• Run as non-root user where possible
• Minimize container capabilities
• Add secrets management
• Configure log sanitization
• Add audit logging

Updates & Maintenance

• Add update notification
• Create upgrade script
• Add configuration migration
• Document breaking changes
• Create rollback procedure

Compatibility

• Test with different Filebeat versions
• Verify Docker API compatibility
• Test on different Linux distributions
• Validate with various log formats
• Ensure Logstash version compatibility

Notes

Design Principles

1. Minimal configuration: Just needs LOGSERVER_HOST to work
2. Docker API access: Use Docker API for driver-independent log collection
3. Automatic discovery: Find all container logs without manual configuration
4. Reliability first: Never lose logs, buffer locally if needed
5. Low overhead: Minimal resource usage on host
6. Non-intrusive: No changes to existing containers needed
7. Driver flexibility: Allow containers to use any logging driver (especially local)

Key Requirements

• Must work with zero configuration beyond server address
• Must use Docker API input, not file-based collection
• Must support all Docker logging drivers (local, json-file, etc.)
• Must handle Docker socket permissions properly
• Must be resilient to network failures
• Must not impact host performance significantly
• Must preserve configuration on uninstall

Testing Checklist

• Validates with dropshell test-template
• Connects to logserver successfully
• Collects Docker logs automatically
• Collects system logs properly
• Handles server downtime gracefully
• Reconnects automatically
• Resource usage stays within limits
• Uninstall preserves configuration