dropshell-templates/logserver/DOCUMENTATION.md

Dropshell LogServer Template

A comprehensive centralized logging solution using the ELK Stack (Elasticsearch, Logstash, Kibana) for receiving, processing, and visualizing logs from multiple hosts.

Overview

This template deploys a full-featured ELK stack that:

• Receives logs from multiple sources via Beats protocol
• Stores and indexes logs in Elasticsearch
• Provides powerful search and visualization through Kibana
• Supports automatic log parsing and enrichment
• Handles Docker container logs and system logs from clients

Architecture

Components

1. Elasticsearch (7.17.x)

   • Distributed search and analytics engine
   • Stores and indexes all log data
   • Provides fast full-text search capabilities
   • Single-node configuration for simplicity (can be scaled)

2. Logstash (7.17.x)

   • Log processing pipeline
   • Receives logs from Filebeat clients
   • Parses and enriches log data
   • Routes logs to appropriate Elasticsearch indices

3. Kibana (7.17.x)

   • Web UI for log exploration and visualization
   • Create dashboards and alerts
   • Real-time log streaming
   • Advanced search queries

Features

Minimum Configuration Design

• Auto-discovery of log formats
• Pre-configured dashboards for common services
• Automatic index lifecycle management
• Built-in parsing for Docker and syslog formats
• Zero-configuration client connectivity

Log Processing

• Automatic timestamp extraction
• Docker metadata enrichment (container name, image, labels)
• Syslog parsing with severity levels
• JSON log support
• Multi-line log handling (stacktraces, etc.)
• Grok pattern matching for common formats

Security & Performance

• Mutual TLS (mTLS) authentication for client connections
• API key authentication as an alternative to certificates
• Per-client authentication with unique keys/certificates
• SSL/TLS encryption for all client connections
• Basic authentication for Kibana web access
• IP whitelisting for additional security
• Index lifecycle management for storage optimization
• Automatic old log cleanup
• Resource limits to prevent overconsumption

Port Configuration

• 5601: Kibana Web UI (HTTP/HTTPS with authentication)
• 9200: Elasticsearch REST API (HTTP) - internal only
• 5044: Logstash Beats input (TCP/TLS) - authenticated client connections
• 514: Syslog input (UDP/TCP) - optional, unauthenticated
• 24224: Fluentd forward input - optional Docker logging driver

Storage Requirements

• Minimum: 10GB for basic operation
• Recommended: 50GB+ depending on log volume
• Log Retention: Default 30 days (configurable)

Client Authentication

Authentication Methods

1. Mutual TLS (mTLS) - Recommended

   • Each client gets a unique certificate signed by the server's CA
   • Strongest security with mutual authentication
   • Automatic certificate validation

2. API Keys

   • Each client gets a unique API key
   • Simpler to manage than certificates
   • Good for environments where certificate management is difficult

3. Basic Auth (Not Recommended)

   • Shared username/password
   • Least secure, only for testing

Client Configuration

Clients using the logclient template will:

1. Authenticate using provided credentials (cert/key or API key)
2. Establish encrypted TLS connection
3. Ship all Docker container logs
4. Ship system logs (syslog, auth, kernel)
5. Maintain connection with automatic reconnection
6. Buffer logs locally during network outages

Dashboard Features

Pre-configured Dashboards

• System Overview: Overall health and log volume metrics
• Docker Containers: Container-specific logs and metrics
• Error Analysis: Aggregated error logs from all sources
• Security Events: Authentication and access logs
• Application Logs: Parsed application-specific logs

Search Capabilities

• Full-text search across all logs
• Filter by time range, host, container, severity
• Save and share search queries
• Export search results

Resource Requirements

Minimum

• CPU: 2 cores
• RAM: 4GB
• Storage: 10GB

Recommended

• CPU: 4+ cores
• RAM: 8GB+
• Storage: 50GB+ SSD

Configuration Options

Environment Variables (service.env)

# Elasticsearch settings
ES_HEAP_SIZE=2g
ES_MAX_MAP_COUNT=262144

# Logstash settings
LS_HEAP_SIZE=1g
LS_PIPELINE_WORKERS=2

# Kibana settings
KIBANA_PASSWORD=changeme
KIBANA_BASE_PATH=/

# Log retention
LOG_RETENTION_DAYS=30
LOG_MAX_SIZE_GB=50

# Authentication Mode
AUTH_MODE=mtls  # Options: mtls, apikey, basic
ENABLE_TLS=true

# mTLS Settings (if AUTH_MODE=mtls)
CA_CERT_PATH=/certs/ca.crt
SERVER_CERT_PATH=/certs/server.crt
SERVER_KEY_PATH=/certs/server.key
CLIENT_CERT_REQUIRED=true

# API Key Settings (if AUTH_MODE=apikey)
API_KEYS_PATH=/config/api-keys.yml

# Network Security
ALLOWED_IPS=""  # Comma-separated list, empty = all

Usage

Installation

dropshell install logserver

Generate Client Credentials

For mTLS Authentication:

# Generate client certificate for a new host
dropshell exec logserver /scripts/generate-client-cert.sh hostname
# This creates hostname.crt and hostname.key files

For API Key Authentication:

# Generate API key for a new client
dropshell exec logserver /scripts/generate-api-key.sh hostname
# Returns an API key to configure in the client

Access Kibana

Navigate to https://<server-ip>:5601 in your browser.

Default credentials:

• Username: elastic
• Password: changeme (change in service.env)

View Logs

dropshell logs logserver

Backup

dropshell backup logserver

Troubleshooting

Common Issues

1. Elasticsearch failing to start

   • Check vm.max_map_count: sysctl vm.max_map_count (should be 262144+)
   • Verify sufficient memory available

2. No logs appearing in Kibana

   • Check Logstash is receiving data: port 5044 should be open
   • Verify client connectivity
   • Check index patterns in Kibana

3. High memory usage

   • Adjust heap sizes in service.env
   • Configure index lifecycle management
   • Reduce retention period

Integration

This template is designed to work seamlessly with the logclient template. Simply:

1. Deploy this logserver
2. Deploy logclient on each host you want to monitor
3. Configure logclient with the logserver address
4. Logs will automatically start flowing

Security Considerations

1. Authentication Setup

   • Use mTLS for production environments
   • Generate unique credentials for each client
   • Rotate certificates/keys regularly
   • Store credentials securely

2. Network Security

   • Always use TLS encryption for client connections
   • Configure IP whitelisting when possible
   • Use firewall rules to restrict access
   • Consider VPN or private networks

3. Access Control

   • Change default Kibana password immediately
   • Create read-only users for viewing logs
   • Implement role-based access control (RBAC)
   • Audit access logs regularly

4. Data Protection

   • Regular backups of Elasticsearch indices
   • Encrypt data at rest (optional)
   • Monitor disk usage to prevent data loss
   • Implement log retention policies

Maintenance

Daily Tasks

• Monitor disk usage
• Check for failed log shipments
• Review error dashboards

Weekly Tasks

• Verify all clients are reporting
• Check index health
• Review and optimize slow queries

Monthly Tasks

• Update ELK stack components
• Archive old indices
• Review retention policies
• Performance tuning based on usage patterns