dropshell-templates/logserver/README.md

# Dropshell LogServer Template

A comprehensive centralized logging solution using the ELK Stack (Elasticsearch, Logstash, Kibana) for receiving, processing, and visualizing logs from multiple hosts.

## Overview

This template deploys a full-featured ELK stack that:
- Receives logs from multiple sources via Beats protocol
- Stores and indexes logs in Elasticsearch
- Provides powerful search and visualization through Kibana
- Supports automatic log parsing and enrichment
- Handles Docker container logs and system logs from clients

## Architecture

### Components

1. **Elasticsearch** (7.17.x)
   - Distributed search and analytics engine
   - Stores and indexes all log data
   - Provides fast full-text search capabilities
   - Single-node configuration for simplicity (can be scaled)

2. **Logstash** (7.17.x)
   - Log processing pipeline
   - Receives logs from Filebeat clients
   - Parses and enriches log data
   - Routes logs to appropriate Elasticsearch indices

3. **Kibana** (7.17.x)
   - Web UI for log exploration and visualization
   - Create dashboards and alerts
   - Real-time log streaming
   - Advanced search queries

## Features

### Minimum Configuration Design
- Auto-discovery of log formats
- Pre-configured dashboards for common services
- Automatic index lifecycle management
- Built-in parsing for Docker and syslog formats
- Zero-configuration client connectivity

### Log Processing
- Automatic timestamp extraction
- Docker metadata enrichment (container name, image, labels)
- Syslog parsing with severity levels
- JSON log support
- Multi-line log handling (stacktraces, etc.)
- Grok pattern matching for common formats

### Security & Performance
- **Mutual TLS (mTLS)** authentication for client connections
- **API key authentication** as an alternative to certificates
- **Per-client authentication** with unique keys/certificates
- **SSL/TLS encryption** for all client connections
- **Basic authentication** for Kibana web access
- **IP whitelisting** for additional security
- Index lifecycle management for storage optimization
- Automatic old log cleanup
- Resource limits to prevent overconsumption

## Port Configuration

- **5601**: Kibana Web UI (HTTP/HTTPS with authentication)
- **9200**: Elasticsearch REST API (HTTP) - internal only
- **5044**: Logstash Beats input (TCP/TLS) - authenticated client connections
- **514**: Syslog input (UDP/TCP) - optional, unauthenticated
- **24224**: Fluentd forward input - optional Docker logging driver

## Storage Requirements

- **Minimum**: 10GB for basic operation
- **Recommended**: 50GB+ depending on log volume
- **Log Retention**: Default 30 days (configurable)

## Client Authentication

### Authentication Methods

1. **Mutual TLS (mTLS) - Recommended**
   - Each client gets a unique certificate signed by the server's CA
   - Strongest security with mutual authentication
   - Automatic certificate validation

2. **API Keys**
   - Each client gets a unique API key
   - Simpler to manage than certificates
   - Good for environments where certificate management is difficult

3. **Basic Auth (Not Recommended)**
   - Shared username/password
   - Least secure, only for testing

### Client Configuration

Clients using the `logclient` template will:
1. Authenticate using provided credentials (cert/key or API key)
2. Establish encrypted TLS connection
3. Ship all Docker container logs
4. Ship system logs (syslog, auth, kernel)
5. Maintain connection with automatic reconnection
6. Buffer logs locally during network outages

## Dashboard Features

### Pre-configured Dashboards
- **System Overview**: Overall health and log volume metrics
- **Docker Containers**: Container-specific logs and metrics
- **Error Analysis**: Aggregated error logs from all sources
- **Security Events**: Authentication and access logs
- **Application Logs**: Parsed application-specific logs

### Search Capabilities
- Full-text search across all logs
- Filter by time range, host, container, severity
- Save and share search queries
- Export search results

## Resource Requirements

### Minimum
- CPU: 2 cores
- RAM: 4GB
- Storage: 10GB

### Recommended
- CPU: 4+ cores
- RAM: 8GB+
- Storage: 50GB+ SSD

## Configuration Options

### Environment Variables (service.env)

```bash
# Elasticsearch settings
ES_HEAP_SIZE=2g
ES_MAX_MAP_COUNT=262144

# Logstash settings
LS_HEAP_SIZE=1g
LS_PIPELINE_WORKERS=2

# Kibana settings
KIBANA_PASSWORD=changeme
KIBANA_BASE_PATH=/

# Log retention
LOG_RETENTION_DAYS=30
LOG_MAX_SIZE_GB=50

# Authentication Mode
AUTH_MODE=mtls  # Options: mtls, apikey, basic
ENABLE_TLS=true

# mTLS Settings (if AUTH_MODE=mtls)
CA_CERT_PATH=/certs/ca.crt
SERVER_CERT_PATH=/certs/server.crt
SERVER_KEY_PATH=/certs/server.key
CLIENT_CERT_REQUIRED=true

# API Key Settings (if AUTH_MODE=apikey)
API_KEYS_PATH=/config/api-keys.yml

# Network Security
ALLOWED_IPS=""  # Comma-separated list, empty = all
```

## Usage

### Installation
```bash
dropshell install logserver
```

### Generate Client Credentials

#### For mTLS Authentication:
```bash
# Generate client certificate for a new host
dropshell exec logserver /scripts/generate-client-cert.sh hostname
# This creates hostname.crt and hostname.key files
```

#### For API Key Authentication:
```bash
# Generate API key for a new client
dropshell exec logserver /scripts/generate-api-key.sh hostname
# Returns an API key to configure in the client
```

### Access Kibana
Navigate to `https://<server-ip>:5601` in your browser.

Default credentials:
- Username: `elastic`
- Password: `changeme` (change in service.env)

### View Logs
```bash
dropshell logs logserver
```

### Backup
```bash
dropshell backup logserver
```

## Troubleshooting

### Common Issues

1. **Elasticsearch failing to start**
   - Check vm.max_map_count: `sysctl vm.max_map_count` (should be 262144+)
   - Verify sufficient memory available

2. **No logs appearing in Kibana**
   - Check Logstash is receiving data: port 5044 should be open
   - Verify client connectivity
   - Check index patterns in Kibana

3. **High memory usage**
   - Adjust heap sizes in service.env
   - Configure index lifecycle management
   - Reduce retention period

## Integration

This template is designed to work seamlessly with the `logclient` template. Simply:
1. Deploy this logserver
2. Deploy logclient on each host you want to monitor
3. Configure logclient with the logserver address
4. Logs will automatically start flowing

## Security Considerations

1. **Authentication Setup**
   - Use mTLS for production environments
   - Generate unique credentials for each client
   - Rotate certificates/keys regularly
   - Store credentials securely

2. **Network Security**
   - Always use TLS encryption for client connections
   - Configure IP whitelisting when possible
   - Use firewall rules to restrict access
   - Consider VPN or private networks

3. **Access Control**
   - Change default Kibana password immediately
   - Create read-only users for viewing logs
   - Implement role-based access control (RBAC)
   - Audit access logs regularly

4. **Data Protection**
   - Regular backups of Elasticsearch indices
   - Encrypt data at rest (optional)
   - Monitor disk usage to prevent data loss
   - Implement log retention policies

## Maintenance

### Daily Tasks
- Monitor disk usage
- Check for failed log shipments
- Review error dashboards

### Weekly Tasks
- Verify all clients are reporting
- Check index health
- Review and optimize slow queries

### Monthly Tasks
- Update ELK stack components
- Archive old indices
- Review retention policies
- Performance tuning based on usage patterns