dropshell-templates/logclient/README.md

# Dropshell LogClient Template

An auto-configuring Filebeat agent that collects Docker container logs via the Docker API and system logs, shipping them to a centralized logging server with minimal configuration.

## Overview

This template deploys a lightweight Filebeat agent that:
- Uses Docker API to collect logs from all containers (regardless of logging driver)
- Allows containers to use any Docker logging driver (json-file, local, journald, etc.)
- Collects system logs (syslog, auth logs, kernel logs)
- Ships logs to a centralized ELK stack (logserver)
- Requires minimal configuration - just the server address
- Handles connection failures with local buffering
- Auto-reconnects when the server becomes available

## Features

### Docker API Log Collection
- **Direct API access**: Reads logs via Docker API, not from files
- **Driver independent**: Works with any Docker logging driver (local, json-file, journald)
- **Automatic discovery**: Finds all running containers dynamically
- **Container metadata**: Enriches logs with container names, images, labels
- **Real-time streaming**: Gets logs as they're generated
- **Multi-line handling**: Properly handles stack traces and multi-line logs
- **JSON parsing**: Automatically parses JSON-formatted logs
- **Label-based filtering**: Can include/exclude containers based on labels

### System Log Collection
- **/var/log/syslog** or **/var/log/messages**: System events
- **/var/log/auth.log** or **/var/log/secure**: Authentication events
- **/var/log/kern.log**: Kernel messages
- **journald**: SystemD journal (if available)
- **Custom paths**: Configurable additional log paths

### Reliability Features
- **Local buffering**: Stores logs locally when server is unreachable
- **Automatic retry**: Reconnects automatically with exponential backoff
- **Compression**: Compresses logs before sending to save bandwidth
- **Secure transmission**: Optional TLS/SSL encryption
- **Backpressure handling**: Slows down when server is overwhelmed

## Architecture

### How It Works
1. Filebeat runs as a container with Docker socket access
2. Uses Docker API to stream logs from all containers
3. Monitors Docker API for container lifecycle events
4. Automatically starts collecting logs from new containers
5. Reads host system logs from mounted volumes
6. Ships all logs to configured Logstash/Elasticsearch endpoint
7. Maintains connection state and buffering information

### Log Flow
```
Docker Containers → Docker API →
                                 ↘
                                  Filebeat → Logstash → Elasticsearch → Kibana
                                 ↗
System Logs (mounted volumes)  →
```

### Why Docker API Instead of Log Files?
- **Logging driver flexibility**: Containers can use `local`, `json-file`, `journald`, or any driver
- **No log file management**: Don't need to worry about log rotation or file paths
- **Better performance**: Direct streaming without file I/O overhead
- **Consistent access**: Same method regardless of storage backend
- **Real-time streaming**: Get logs immediately as they're generated
- **Simplified permissions**: Only need Docker socket access

## Minimum Configuration

The template requires minimal configuration - server address and authentication:

```bash
# Required - Server connection
LOGSERVER_HOST=192.168.1.100
LOGSERVER_PORT=5044

# Required - Authentication (choose one method)
AUTH_MODE=mtls  # Options: mtls, apikey, basic

# For mTLS authentication
CLIENT_CERT_PATH=/certs/client.crt
CLIENT_KEY_PATH=/certs/client.key
CA_CERT_PATH=/certs/ca.crt

# For API key authentication
API_KEY=your-api-key-here

# For basic auth (not recommended)
USERNAME=filebeat
PASSWORD=changeme
```

## Configuration Options

### Environment Variables (service.env)

```bash
# REQUIRED: Log server connection
LOGSERVER_HOST=logserver.example.com
LOGSERVER_PORT=5044

# REQUIRED: Authentication method
AUTH_MODE=mtls  # mtls, apikey, or basic

# mTLS Authentication (if AUTH_MODE=mtls)
CLIENT_CERT_PATH=/certs/${HOSTNAME}.crt
CLIENT_KEY_PATH=/certs/${HOSTNAME}.key
CA_CERT_PATH=/certs/ca.crt
SSL_VERIFICATION_MODE=full

# API Key Authentication (if AUTH_MODE=apikey)
API_KEY=""  # Will be provided by logserver admin

# Basic Authentication (if AUTH_MODE=basic)
USERNAME=filebeat
PASSWORD=changeme

# Optional: Performance tuning
BULK_MAX_SIZE=2048          # Maximum batch size
WORKER_THREADS=1            # Number of worker threads
QUEUE_SIZE=4096            # Internal queue size
MAX_BACKOFF=60s           # Maximum retry backoff

# Optional: Filtering
EXCLUDE_CONTAINERS=""      # Comma-separated container names to exclude
INCLUDE_CONTAINERS=""      # Only include these containers (if set)
EXCLUDE_LABELS=""         # Exclude containers with these labels
INCLUDE_LABELS=""         # Only include containers with these labels

# Optional: Additional log paths
CUSTOM_LOG_PATHS=""       # Comma-separated additional paths to monitor

# Optional: Resource limits
MAX_CPU=50               # Maximum CPU usage percentage
MAX_MEMORY=200MB         # Maximum memory usage
```

## Collected Log Types

### Docker Container Logs (via Docker API)
- **stdout/stderr**: All container output regardless of logging driver
- **Container metadata**: Name, ID, image, labels
- **Docker events**: Start, stop, die, kill events
- **Health check results**: If configured
- **Works with all logging drivers**: local, json-file, journald, syslog, etc.

### System Logs
- **System messages**: Service starts/stops, errors
- **Authentication**: SSH logins, sudo usage
- **Kernel messages**: Hardware events, driver messages
- **Package management**: apt/yum operations
- **Cron jobs**: Scheduled task execution

## Log Enrichment

Logs are automatically enriched with:
- **Hostname**: Source host identification
- **Timestamp**: Precise event time with timezone
- **Log level**: Parsed from log content when possible
- **Container info**: For Docker logs
- **Process info**: PID, command for system logs
- **File path**: Source log file

## Resource Requirements

### Minimum
- CPU: 0.5 cores
- RAM: 128MB
- Storage: 1GB (for buffer)

### Typical Usage
- CPU: 1-5% of one core
- RAM: 150-200MB
- Network: Varies with log volume
- Storage: Depends on buffer size

## Installation

### Prerequisites
1. A running logserver (ELK stack)
2. Network connectivity to logserver
3. Docker installed on host
4. Authentication credentials from logserver admin

### Setup Authentication

#### For mTLS (Recommended):
```bash
# Get client certificate from logserver admin
# They will run: dropshell exec logserver /scripts/generate-client-cert.sh $(hostname)
# Copy the generated certificate files to this client
mkdir -p /etc/dropshell/certs
# Copy ca.crt, client.crt, and client.key to /etc/dropshell/certs/
```

#### For API Key:
```bash
# Get API key from logserver admin
# They will run: dropshell exec logserver /scripts/generate-api-key.sh $(hostname)
# Add the API key to service.env
```

### Deploy
```bash
# Configure authentication in service.env
dropshell install logclient
```

## Monitoring

### Check Status
```bash
dropshell status logclient
```

### View Filebeat Logs
```bash
dropshell logs logclient
```

### Verify Connectivity
```bash
# Check if logs are being shipped
docker exec logclient-filebeat filebeat test output
```

### Monitor Metrics
```bash
# View Filebeat statistics
docker exec logclient-filebeat curl -s http://localhost:5066/stats
```

## Troubleshooting

### No Logs Appearing on Server

1. **Check connectivity**
   ```bash
   telnet $LOGSERVER_HOST $LOGSERVER_PORT
   ```

2. **Verify Filebeat is running**
   ```bash
   dropshell status logclient
   ```

3. **Check Filebeat logs**
   ```bash
   dropshell logs logclient | tail -50
   ```

4. **Test configuration**
   ```bash
   docker exec logclient-filebeat filebeat test config
   ```

### High CPU Usage

1. Reduce worker threads in service.env
2. Increase bulk_max_size to send larger batches
3. Add exclude filters for noisy containers

### Missing Container Logs

1. Verify Docker socket is mounted
2. Check container isn't in exclude list
3. Ensure Filebeat has permissions to Docker socket
4. Verify container is actually producing output
5. Check if container uses a supported logging driver

### Buffer Full Errors

1. Increase queue_size in service.env
2. Check network connectivity to server
3. Verify server isn't overwhelmed

## Security Considerations

1. **Authentication**:
   - Always use mTLS or API keys in production
   - Never use basic auth except for testing
   - Store credentials securely
   - Rotate certificates/keys regularly

2. **Docker Socket Access**:
   - Requires Docker socket access to read logs via API
   - Understand security implications of socket access
   - Consider read-only socket access if possible

3. **Network Security**:
   - All connections are TLS encrypted
   - Verify server certificates
   - Configure firewall rules appropriately
   - Use private networks when possible

4. **Data Protection**:
   - Logs may contain sensitive data
   - Filter sensitive information before shipping
   - Exclude containers with sensitive data if needed

5. **Resource Limits**:
   - Set CPU and memory limits
   - Monitor resource usage
   - Prevent resource exhaustion attacks

## Performance Tuning

### For High-Volume Environments
```bash
# Increase workers and batch size
WORKER_THREADS=4
BULK_MAX_SIZE=4096
QUEUE_SIZE=8192
```

### For Low-Resource Hosts
```bash
# Reduce resource usage
WORKER_THREADS=1
BULK_MAX_SIZE=512
MAX_MEMORY=100MB
MAX_CPU=25
```

### Network Optimization
```bash
# Enable compression (CPU vs bandwidth tradeoff)
COMPRESSION_LEVEL=3  # 0-9, higher = more compression
```

## Integration with LogServer

This template is designed to work seamlessly with the `logserver` template:

1. Deploy logserver first
2. Note the logserver's IP/hostname
3. Configure logclient with server address
4. Logs automatically start flowing

## Maintenance

### Regular Tasks
- Monitor buffer usage
- Check for connection errors
- Review excluded/included containers
- Update Filebeat version

### Logs Rotation
Filebeat handles log rotation automatically:
- Detects renamed/rotated files
- Continues reading from correct position
- Cleans up old file handles

## Advanced Configuration

### Custom Filebeat Configuration
Create a custom `filebeat.yml` in the config directory for advanced scenarios:
- Custom processors
- Additional inputs
- Complex filtering rules
- Multiple outputs

### Docker Labels for Control
Control log collection per container:
```yaml
# In docker-compose.yml
services:
  myapp:
    logging:
      driver: local  # Can use any driver - Filebeat reads via API
      options:
        max-size: "10m"
        max-file: "3"
    labels:
      - "filebeat.enable=false"  # Exclude this container
      - "filebeat.multiline.pattern=^\\["  # Custom multiline pattern
```

### Logging Driver Compatibility
The Docker API input works with all Docker logging drivers:
- **local**: Recommended for production (efficient, no file access needed)
- **json-file**: Traditional default driver
- **journald**: SystemD journal integration
- **syslog**: Forward to syslog
- **none**: Disables logging (Filebeat won't collect)

You can use the `local` driver for better performance since Filebeat doesn't need to read files.