swtich from ELK to Loki!
All checks were successful
Test and Publish Templates / test-and-publish (push) Successful in 40s
All checks were successful
Test and Publish Templates / test-and-publish (push) Successful in 40s
This commit is contained in:
Your Name
@@ -1,389 +0,0 @@
|
||||
# Dropshell LogClient Template
|
||||
|
||||
An auto-configuring Filebeat agent that collects Docker container logs via the Docker API and system logs, shipping them to a centralized logging server with minimal configuration.
|
||||
|
||||
## Overview
|
||||
|
||||
This template deploys a lightweight Filebeat agent that:
|
||||
- Uses Docker API to collect logs from all containers (regardless of logging driver)
|
||||
- Allows containers to use any Docker logging driver (json-file, local, journald, etc.)
|
||||
- Collects system logs (syslog, auth logs, kernel logs)
|
||||
- Ships logs to a centralized ELK stack (logserver)
|
||||
- Requires minimal configuration - just the server address
|
||||
- Handles connection failures with local buffering
|
||||
- Auto-reconnects when the server becomes available
|
||||
|
||||
## Features
|
||||
|
||||
### Docker API Log Collection
|
||||
- **Direct API access**: Reads logs via Docker API, not from files
|
||||
- **Driver independent**: Works with any Docker logging driver (local, json-file, journald)
|
||||
- **Automatic discovery**: Finds all running containers dynamically
|
||||
- **Container metadata**: Enriches logs with container names, images, labels
|
||||
- **Real-time streaming**: Gets logs as they're generated
|
||||
- **Multi-line handling**: Properly handles stack traces and multi-line logs
|
||||
- **JSON parsing**: Automatically parses JSON-formatted logs
|
||||
- **Label-based filtering**: Can include/exclude containers based on labels
|
||||
|
||||
### System Log Collection
|
||||
- **/var/log/syslog** or **/var/log/messages**: System events
|
||||
- **/var/log/auth.log** or **/var/log/secure**: Authentication events
|
||||
- **/var/log/kern.log**: Kernel messages
|
||||
- **journald**: SystemD journal (if available)
|
||||
- **Custom paths**: Configurable additional log paths
|
||||
|
||||
### Reliability Features
|
||||
- **Local buffering**: Stores logs locally when server is unreachable
|
||||
- **Automatic retry**: Reconnects automatically with exponential backoff
|
||||
- **Compression**: Compresses logs before sending to save bandwidth
|
||||
- **Secure transmission**: Optional TLS/SSL encryption
|
||||
- **Backpressure handling**: Slows down when server is overwhelmed
|
||||
|
||||
## Architecture
|
||||
|
||||
### How It Works
|
||||
1. Filebeat runs as a container with Docker socket access
|
||||
2. Uses Docker API to stream logs from all containers
|
||||
3. Monitors Docker API for container lifecycle events
|
||||
4. Automatically starts collecting logs from new containers
|
||||
5. Reads host system logs from mounted volumes
|
||||
6. Ships all logs to configured Logstash/Elasticsearch endpoint
|
||||
7. Maintains connection state and buffering information
|
||||
|
||||
### Log Flow
|
||||
```
|
||||
Docker Containers → Docker API →
|
||||
↘
|
||||
Filebeat → Logstash → Elasticsearch → Kibana
|
||||
↗
|
||||
System Logs (mounted volumes) →
|
||||
```
|
||||
|
||||
### Why Docker API Instead of Log Files?
|
||||
- **Logging driver flexibility**: Containers can use `local`, `json-file`, `journald`, or any driver
|
||||
- **No log file management**: Don't need to worry about log rotation or file paths
|
||||
- **Better performance**: Direct streaming without file I/O overhead
|
||||
- **Consistent access**: Same method regardless of storage backend
|
||||
- **Real-time streaming**: Get logs immediately as they're generated
|
||||
- **Simplified permissions**: Only need Docker socket access
|
||||
|
||||
## Minimum Configuration
|
||||
|
||||
The template requires minimal configuration - server address and authentication:
|
||||
|
||||
```bash
|
||||
# Required - Server connection
|
||||
LOGSERVER_HOST=192.168.1.100
|
||||
LOGSERVER_PORT=5044
|
||||
|
||||
# Required - Authentication (choose one method)
|
||||
AUTH_MODE=mtls # Options: mtls, apikey, basic
|
||||
|
||||
# For mTLS authentication
|
||||
CLIENT_CERT_PATH=/certs/client.crt
|
||||
CLIENT_KEY_PATH=/certs/client.key
|
||||
CA_CERT_PATH=/certs/ca.crt
|
||||
|
||||
# For API key authentication
|
||||
API_KEY=your-api-key-here
|
||||
|
||||
# For basic auth (not recommended)
|
||||
USERNAME=filebeat
|
||||
PASSWORD=changeme
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
### Environment Variables (service.env)
|
||||
|
||||
```bash
|
||||
# REQUIRED: Log server connection
|
||||
LOGSERVER_HOST=logserver.example.com
|
||||
LOGSERVER_PORT=5044
|
||||
|
||||
# REQUIRED: Authentication method
|
||||
AUTH_MODE=mtls # mtls, apikey, or basic
|
||||
|
||||
# mTLS Authentication (if AUTH_MODE=mtls)
|
||||
CLIENT_CERT_PATH=/certs/${HOSTNAME}.crt
|
||||
CLIENT_KEY_PATH=/certs/${HOSTNAME}.key
|
||||
CA_CERT_PATH=/certs/ca.crt
|
||||
SSL_VERIFICATION_MODE=full
|
||||
|
||||
# API Key Authentication (if AUTH_MODE=apikey)
|
||||
API_KEY="" # Will be provided by logserver admin
|
||||
|
||||
# Basic Authentication (if AUTH_MODE=basic)
|
||||
USERNAME=filebeat
|
||||
PASSWORD=changeme
|
||||
|
||||
# Optional: Performance tuning
|
||||
BULK_MAX_SIZE=2048 # Maximum batch size
|
||||
WORKER_THREADS=1 # Number of worker threads
|
||||
QUEUE_SIZE=4096 # Internal queue size
|
||||
MAX_BACKOFF=60s # Maximum retry backoff
|
||||
|
||||
# Optional: Filtering
|
||||
EXCLUDE_CONTAINERS="" # Comma-separated container names to exclude
|
||||
INCLUDE_CONTAINERS="" # Only include these containers (if set)
|
||||
EXCLUDE_LABELS="" # Exclude containers with these labels
|
||||
INCLUDE_LABELS="" # Only include containers with these labels
|
||||
|
||||
# Optional: Additional log paths
|
||||
CUSTOM_LOG_PATHS="" # Comma-separated additional paths to monitor
|
||||
|
||||
# Optional: Resource limits
|
||||
MAX_CPU=50 # Maximum CPU usage percentage
|
||||
MAX_MEMORY=200MB # Maximum memory usage
|
||||
```
|
||||
|
||||
## Collected Log Types
|
||||
|
||||
### Docker Container Logs (via Docker API)
|
||||
- **stdout/stderr**: All container output regardless of logging driver
|
||||
- **Container metadata**: Name, ID, image, labels
|
||||
- **Docker events**: Start, stop, die, kill events
|
||||
- **Health check results**: If configured
|
||||
- **Works with all logging drivers**: local, json-file, journald, syslog, etc.
|
||||
|
||||
### System Logs
|
||||
- **System messages**: Service starts/stops, errors
|
||||
- **Authentication**: SSH logins, sudo usage
|
||||
- **Kernel messages**: Hardware events, driver messages
|
||||
- **Package management**: apt/yum operations
|
||||
- **Cron jobs**: Scheduled task execution
|
||||
|
||||
## Log Enrichment
|
||||
|
||||
Logs are automatically enriched with:
|
||||
- **Hostname**: Source host identification
|
||||
- **Timestamp**: Precise event time with timezone
|
||||
- **Log level**: Parsed from log content when possible
|
||||
- **Container info**: For Docker logs
|
||||
- **Process info**: PID, command for system logs
|
||||
- **File path**: Source log file
|
||||
|
||||
## Resource Requirements
|
||||
|
||||
### Minimum
|
||||
- CPU: 0.5 cores
|
||||
- RAM: 128MB
|
||||
- Storage: 1GB (for buffer)
|
||||
|
||||
### Typical Usage
|
||||
- CPU: 1-5% of one core
|
||||
- RAM: 150-200MB
|
||||
- Network: Varies with log volume
|
||||
- Storage: Depends on buffer size
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
1. A running logserver (ELK stack)
|
||||
2. Network connectivity to logserver
|
||||
3. Docker installed on host
|
||||
4. Authentication credentials from logserver admin
|
||||
|
||||
### Setup Authentication
|
||||
|
||||
#### For mTLS (Recommended):
|
||||
```bash
|
||||
# Get client certificate from logserver admin
|
||||
# They will run: dropshell exec logserver /scripts/generate-client-cert.sh $(hostname)
|
||||
# Copy the generated certificate files to this client
|
||||
mkdir -p /etc/dropshell/certs
|
||||
# Copy ca.crt, client.crt, and client.key to /etc/dropshell/certs/
|
||||
```
|
||||
|
||||
#### For API Key:
|
||||
```bash
|
||||
# Get API key from logserver admin
|
||||
# They will run: dropshell exec logserver /scripts/generate-api-key.sh $(hostname)
|
||||
# Add the API key to service.env
|
||||
```
|
||||
|
||||
### Deploy
|
||||
```bash
|
||||
# Configure authentication in service.env
|
||||
dropshell install logclient
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Check Status
|
||||
```bash
|
||||
dropshell status logclient
|
||||
```
|
||||
|
||||
### View Filebeat Logs
|
||||
```bash
|
||||
dropshell logs logclient
|
||||
```
|
||||
|
||||
### Verify Connectivity
|
||||
```bash
|
||||
# Check if logs are being shipped
|
||||
docker exec logclient-filebeat filebeat test output
|
||||
```
|
||||
|
||||
### Monitor Metrics
|
||||
```bash
|
||||
# View Filebeat statistics
|
||||
docker exec logclient-filebeat curl -s http://localhost:5066/stats
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### No Logs Appearing on Server
|
||||
|
||||
1. **Check connectivity**
|
||||
```bash
|
||||
telnet $LOGSERVER_HOST $LOGSERVER_PORT
|
||||
```
|
||||
|
||||
2. **Verify Filebeat is running**
|
||||
```bash
|
||||
dropshell status logclient
|
||||
```
|
||||
|
||||
3. **Check Filebeat logs**
|
||||
```bash
|
||||
dropshell logs logclient | tail -50
|
||||
```
|
||||
|
||||
4. **Test configuration**
|
||||
```bash
|
||||
docker exec logclient-filebeat filebeat test config
|
||||
```
|
||||
|
||||
### High CPU Usage
|
||||
|
||||
1. Reduce worker threads in service.env
|
||||
2. Increase bulk_max_size to send larger batches
|
||||
3. Add exclude filters for noisy containers
|
||||
|
||||
### Missing Container Logs
|
||||
|
||||
1. Verify Docker socket is mounted
|
||||
2. Check container isn't in exclude list
|
||||
3. Ensure Filebeat has permissions to Docker socket
|
||||
4. Verify container is actually producing output
|
||||
5. Check if container uses a supported logging driver
|
||||
|
||||
### Buffer Full Errors
|
||||
|
||||
1. Increase queue_size in service.env
|
||||
2. Check network connectivity to server
|
||||
3. Verify server isn't overwhelmed
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Authentication**:
|
||||
- Always use mTLS or API keys in production
|
||||
- Never use basic auth except for testing
|
||||
- Store credentials securely
|
||||
- Rotate certificates/keys regularly
|
||||
|
||||
2. **Docker Socket Access**:
|
||||
- Requires Docker socket access to read logs via API
|
||||
- Understand security implications of socket access
|
||||
- Consider read-only socket access if possible
|
||||
|
||||
3. **Network Security**:
|
||||
- All connections are TLS encrypted
|
||||
- Verify server certificates
|
||||
- Configure firewall rules appropriately
|
||||
- Use private networks when possible
|
||||
|
||||
4. **Data Protection**:
|
||||
- Logs may contain sensitive data
|
||||
- Filter sensitive information before shipping
|
||||
- Exclude containers with sensitive data if needed
|
||||
|
||||
5. **Resource Limits**:
|
||||
- Set CPU and memory limits
|
||||
- Monitor resource usage
|
||||
- Prevent resource exhaustion attacks
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### For High-Volume Environments
|
||||
```bash
|
||||
# Increase workers and batch size
|
||||
WORKER_THREADS=4
|
||||
BULK_MAX_SIZE=4096
|
||||
QUEUE_SIZE=8192
|
||||
```
|
||||
|
||||
### For Low-Resource Hosts
|
||||
```bash
|
||||
# Reduce resource usage
|
||||
WORKER_THREADS=1
|
||||
BULK_MAX_SIZE=512
|
||||
MAX_MEMORY=100MB
|
||||
MAX_CPU=25
|
||||
```
|
||||
|
||||
### Network Optimization
|
||||
```bash
|
||||
# Enable compression (CPU vs bandwidth tradeoff)
|
||||
COMPRESSION_LEVEL=3 # 0-9, higher = more compression
|
||||
```
|
||||
|
||||
## Integration with LogServer
|
||||
|
||||
This template is designed to work seamlessly with the `logserver` template:
|
||||
|
||||
1. Deploy logserver first
|
||||
2. Note the logserver's IP/hostname
|
||||
3. Configure logclient with server address
|
||||
4. Logs automatically start flowing
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Regular Tasks
|
||||
- Monitor buffer usage
|
||||
- Check for connection errors
|
||||
- Review excluded/included containers
|
||||
- Update Filebeat version
|
||||
|
||||
### Logs Rotation
|
||||
Filebeat handles log rotation automatically:
|
||||
- Detects renamed/rotated files
|
||||
- Continues reading from correct position
|
||||
- Cleans up old file handles
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
### Custom Filebeat Configuration
|
||||
Create a custom `filebeat.yml` in the config directory for advanced scenarios:
|
||||
- Custom processors
|
||||
- Additional inputs
|
||||
- Complex filtering rules
|
||||
- Multiple outputs
|
||||
|
||||
### Docker Labels for Control
|
||||
Control log collection per container:
|
||||
```yaml
|
||||
# In docker-compose.yml
|
||||
services:
|
||||
myapp:
|
||||
logging:
|
||||
driver: local # Can use any driver - Filebeat reads via API
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
labels:
|
||||
- "filebeat.enable=false" # Exclude this container
|
||||
- "filebeat.multiline.pattern=^\\[" # Custom multiline pattern
|
||||
```
|
||||
|
||||
### Logging Driver Compatibility
|
||||
The Docker API input works with all Docker logging drivers:
|
||||
- **local**: Recommended for production (efficient, no file access needed)
|
||||
- **json-file**: Traditional default driver
|
||||
- **journald**: SystemD journal integration
|
||||
- **syslog**: Forward to syslog
|
||||
- **none**: Disables logging (Filebeat won't collect)
|
||||
|
||||
You can use the `local` driver for better performance since Filebeat doesn't need to read files.
|
||||
@@ -1,34 +1,56 @@
|
||||
# LogClient
|
||||
# Log Client (Promtail)
|
||||
|
||||
Ships Docker container and system logs to LogServer using Filebeat.
|
||||
Ships Docker and system logs to Log Server. Zero configuration!
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. **Get API Key**
|
||||
- Ask LogServer admin to run `./generate-api-key.sh`
|
||||
- They'll provide your API key
|
||||
|
||||
2. **Configure**
|
||||
Edit `config/service.env`:
|
||||
1. **Configure** (edit `config/service.env`):
|
||||
```bash
|
||||
LOGSERVER_HOST=<server-ip>
|
||||
LOGSERVER_PORT=5044
|
||||
API_KEY=<your-api-key>
|
||||
LOGSERVER_HOST=<server-ip> # Your log server IP
|
||||
LOGSERVER_PORT=3100 # Default Loki port
|
||||
LOKI_USER=logclient # Default username
|
||||
LOKI_PASSWORD=<password> # Get from server admin
|
||||
```
|
||||
|
||||
3. **Install**
|
||||
2. **Install**:
|
||||
```bash
|
||||
dropshell install logclient
|
||||
```
|
||||
|
||||
## What It Does
|
||||
- Collects all Docker container logs via API
|
||||
- Collects system logs (/var/log)
|
||||
- Ships to central LogServer
|
||||
- Works with any Docker logging driver
|
||||
That's it! Logs are now shipping to your central server.
|
||||
|
||||
## Requirements
|
||||
- Docker socket access
|
||||
- Network connection to LogServer port 5044
|
||||
## What Gets Sent
|
||||
- All Docker container logs (via Docker API)
|
||||
- System logs (/var/log/syslog)
|
||||
- Auth logs (/var/log/auth.log)
|
||||
- Hostname is automatically included
|
||||
|
||||
See [DOCUMENTATION.md](DOCUMENTATION.md) for full details.
|
||||
## View Your Logs
|
||||
Go to: `http://<logserver-ip>:3000`
|
||||
- Click Dashboards → Central Logs
|
||||
- Your server's logs appear with its hostname
|
||||
|
||||
## Features
|
||||
- **Zero maintenance** - Just runs
|
||||
- **Lightweight** - ~20MB RAM
|
||||
- **Automatic** - Finds all containers
|
||||
- **Labeled** - Hostname included automatically
|
||||
|
||||
## Simple Authentication
|
||||
Just a username/password - much simpler than ELK's API key system!
|
||||
|
||||
## Troubleshooting
|
||||
Check if client is running:
|
||||
```bash
|
||||
./status.sh
|
||||
```
|
||||
|
||||
View client logs:
|
||||
```bash
|
||||
./logs.sh
|
||||
```
|
||||
|
||||
Test connection to server:
|
||||
```bash
|
||||
nc -zv <logserver-ip> 3100
|
||||
```
|
||||
@@ -1,294 +0,0 @@
|
||||
# LogClient Template - Implementation TODO
|
||||
|
||||
## Phase 1: Core Infrastructure (Priority 1)
|
||||
|
||||
### Configuration Files
|
||||
- [ ] Create `config/.template_info.env` with template metadata
|
||||
- [ ] Create `config/service.env` with minimal required settings
|
||||
- [ ] Define LOGSERVER_HOST and LOGSERVER_PORT variables
|
||||
- [ ] Add AUTH_MODE variable (mtls, apikey, basic)
|
||||
- [ ] Add certificate/key path variables for mTLS
|
||||
- [ ] Add API_KEY variable for API key auth
|
||||
- [ ] Add USERNAME/PASSWORD for basic auth
|
||||
- [ ] Add optional performance and filtering variables
|
||||
- [ ] Set sensible defaults where possible
|
||||
|
||||
### Filebeat Configuration
|
||||
- [ ] Create base `filebeat.yml` configuration template
|
||||
- [ ] Configure Docker input using Docker API (not autodiscover with hints)
|
||||
- [ ] Set containers.ids: ["*"] to collect from all containers
|
||||
- [ ] Set up system log inputs for host logs
|
||||
- [ ] Configure output to Logstash
|
||||
- [ ] Add error handling and retry logic
|
||||
- [ ] Set up local disk buffering
|
||||
- [ ] Configure stream: "all" to get both stdout and stderr
|
||||
|
||||
### Required Scripts
|
||||
- [ ] Implement `install.sh` - Pull Filebeat image, configure auth, start
|
||||
- [ ] Implement `uninstall.sh` - Stop and remove container (preserve config and certs)
|
||||
- [ ] Implement `start.sh` - Start Filebeat with auth config and proper mounts
|
||||
- [ ] Implement `stop.sh` - Gracefully stop Filebeat
|
||||
- [ ] Implement `status.sh` - Check Filebeat health and auth status
|
||||
- [ ] Create `setup-auth.sh` - Helper script to configure authentication
|
||||
|
||||
## Phase 2: Docker API Log Collection (Priority 1)
|
||||
|
||||
### Docker API Input Configuration
|
||||
- [ ] Configure Docker input type (NOT autodiscover, use direct Docker input)
|
||||
- [ ] Mount Docker socket (/var/run/docker.sock) with proper permissions
|
||||
- [ ] Configure Docker API endpoint (unix:///var/run/docker.sock)
|
||||
- [ ] Set up real-time log streaming from Docker daemon
|
||||
- [ ] Enable collection from ALL logging drivers (local, json-file, journald, etc.)
|
||||
- [ ] Configure since_time to get recent logs on startup
|
||||
|
||||
### Container Metadata Extraction
|
||||
- [ ] Extract container name, ID, image name, and image tag
|
||||
- [ ] Map container labels to fields
|
||||
- [ ] Handle docker-compose project names and service names
|
||||
- [ ] Add container state information
|
||||
- [ ] Include container environment variables (filtered)
|
||||
- [ ] Handle container lifecycle events (start, stop, restart)
|
||||
|
||||
### Container Filtering
|
||||
- [ ] Implement include/exclude by container name patterns
|
||||
- [ ] Add label-based filtering (containers.labels)
|
||||
- [ ] Create ignore patterns for system containers
|
||||
- [ ] Add support for custom filter expressions
|
||||
- [ ] Configure combine_partial to handle partial log lines
|
||||
- [ ] Document filtering examples with Docker API syntax
|
||||
|
||||
## Phase 3: System Log Collection (Priority 1)
|
||||
|
||||
### Log File Inputs
|
||||
- [ ] Configure /var/log/syslog or /var/log/messages
|
||||
- [ ] Add /var/log/auth.log or /var/log/secure
|
||||
- [ ] Include /var/log/kern.log
|
||||
- [ ] Monitor /var/log/dpkg.log or /var/log/yum.log
|
||||
- [ ] Add custom log path support via environment variable
|
||||
|
||||
### Journald Integration
|
||||
- [ ] Detect if systemd/journald is available
|
||||
- [ ] Configure journald input if present
|
||||
- [ ] Set up unit filtering
|
||||
- [ ] Extract systemd metadata
|
||||
- [ ] Handle binary journal format
|
||||
|
||||
### Log Parsing
|
||||
- [ ] Configure syslog parsing
|
||||
- [ ] Extract severity levels
|
||||
- [ ] Parse timestamps correctly
|
||||
- [ ] Handle different syslog formats
|
||||
- [ ] Add timezone handling
|
||||
|
||||
## Phase 4: Output Configuration (Priority 1)
|
||||
|
||||
### Logstash Output
|
||||
- [ ] Configure primary Logstash endpoint
|
||||
- [ ] Set up connection parameters (timeout, retry)
|
||||
- [ ] Configure bulk operations settings
|
||||
- [ ] Add compression support
|
||||
- [ ] Implement backpressure handling
|
||||
|
||||
### Connection Management
|
||||
- [ ] Configure automatic reconnection
|
||||
- [ ] Set exponential backoff for retries
|
||||
- [ ] Add connection pooling
|
||||
- [ ] Configure keepalive settings
|
||||
- [ ] Handle DNS resolution failures
|
||||
|
||||
### Authentication Configuration (Priority 1 - CRITICAL)
|
||||
- [ ] Implement mTLS authentication support
|
||||
- [ ] Configure client certificate and key loading
|
||||
- [ ] Add CA certificate validation
|
||||
- [ ] Implement API key authentication
|
||||
- [ ] Add basic auth as fallback option
|
||||
- [ ] Create authentication mode selection logic
|
||||
- [ ] Handle authentication failures gracefully
|
||||
- [ ] Add certificate expiry checking
|
||||
- [ ] Implement secure credential storage
|
||||
- [ ] Document authentication setup process
|
||||
|
||||
## Phase 5: Reliability Features (Priority 2)
|
||||
|
||||
### Local Buffering
|
||||
- [ ] Configure disk queue for reliability
|
||||
- [ ] Set queue size limits
|
||||
- [ ] Configure memory queue settings
|
||||
- [ ] Add overflow handling
|
||||
- [ ] Set up automatic cleanup of old events
|
||||
|
||||
### Error Handling
|
||||
- [ ] Add retry logic for failed sends
|
||||
- [ ] Configure dead letter queue
|
||||
- [ ] Add circuit breaker pattern
|
||||
- [ ] Log transmission errors appropriately
|
||||
- [ ] Add metrics for monitoring failures
|
||||
|
||||
### Performance Optimization
|
||||
- [ ] Configure worker count
|
||||
- [ ] Set batch size for sending
|
||||
- [ ] Add compression level setting
|
||||
- [ ] Configure CPU and memory limits
|
||||
- [ ] Optimize for high-volume scenarios
|
||||
|
||||
## Phase 6: Optional Scripts (Priority 2)
|
||||
|
||||
### Operational Scripts
|
||||
- [ ] Implement `logs.sh` - Show Filebeat logs
|
||||
- [ ] Implement `destroy.sh` - Complete removal
|
||||
- [ ] Implement `ssh.sh` - Shell into Filebeat container
|
||||
- [ ] Create `test.sh` - Test connectivity to server
|
||||
- [ ] Add `metrics.sh` - Show Filebeat statistics
|
||||
|
||||
### Diagnostic Scripts
|
||||
- [ ] Create connectivity test script
|
||||
- [ ] Add configuration validation script
|
||||
- [ ] Create debug mode enabler
|
||||
- [ ] Add log sampling script
|
||||
- [ ] Create performance benchmark script
|
||||
|
||||
## Phase 7: Monitoring & Health (Priority 2)
|
||||
|
||||
### Health Checks
|
||||
- [ ] Configure Filebeat HTTP endpoint
|
||||
- [ ] Add Docker health check
|
||||
- [ ] Monitor queue status
|
||||
- [ ] Check connection to Logstash
|
||||
- [ ] Track dropped events
|
||||
|
||||
### Metrics Collection
|
||||
- [ ] Enable Filebeat monitoring
|
||||
- [ ] Export metrics endpoint
|
||||
- [ ] Track events sent/failed
|
||||
- [ ] Monitor resource usage
|
||||
- [ ] Add performance counters
|
||||
|
||||
### Status Reporting
|
||||
- [ ] Implement detailed status in status.sh
|
||||
- [ ] Show connection state
|
||||
- [ ] Display queue status
|
||||
- [ ] Report recent errors
|
||||
- [ ] Show throughput metrics
|
||||
|
||||
## Phase 8: Advanced Features (Priority 3)
|
||||
|
||||
### Processors
|
||||
- [ ] Add field renaming processor
|
||||
- [ ] Configure drop_event conditions
|
||||
- [ ] Add rate limiting processor
|
||||
- [ ] Include fingerprinting for deduplication
|
||||
- [ ] Add custom field enrichment
|
||||
|
||||
### Multiline Handling
|
||||
- [ ] Configure patterns for common languages
|
||||
- [ ] Java stack trace handling
|
||||
- [ ] Python traceback handling
|
||||
- [ ] Go panic handling
|
||||
- [ ] Custom pattern support via environment
|
||||
|
||||
### Field Management
|
||||
- [ ] Configure field inclusion/exclusion
|
||||
- [ ] Add custom fields via environment
|
||||
- [ ] Set up field type conversions
|
||||
- [ ] Add timestamp parsing
|
||||
- [ ] Configure field aliasing
|
||||
|
||||
## Phase 9: Testing (Priority 3)
|
||||
|
||||
### Unit Testing
|
||||
- [ ] Test configuration generation
|
||||
- [ ] Verify volume mounts
|
||||
- [ ] Test environment variable substitution
|
||||
- [ ] Validate filtering logic
|
||||
- [ ] Test error conditions
|
||||
|
||||
### Integration Testing
|
||||
- [ ] Test with logserver template
|
||||
- [ ] Verify Docker log collection
|
||||
- [ ] Test system log collection
|
||||
- [ ] Validate SSL connectivity
|
||||
- [ ] Test reconnection scenarios
|
||||
- [ ] Verify buffering during outages
|
||||
|
||||
### Load Testing
|
||||
- [ ] Test with high log volume
|
||||
- [ ] Measure resource usage
|
||||
- [ ] Test queue overflow handling
|
||||
- [ ] Verify rate limiting
|
||||
- [ ] Benchmark throughput
|
||||
|
||||
## Phase 10: Documentation (Priority 3)
|
||||
|
||||
### User Documentation
|
||||
- [ ] Create README.txt for dropshell
|
||||
- [ ] Document all configuration options
|
||||
- [ ] Add troubleshooting guide
|
||||
- [ ] Create quick start guide
|
||||
- [ ] Add FAQ section
|
||||
|
||||
### Configuration Examples
|
||||
- [ ] Minimal configuration example
|
||||
- [ ] High-volume configuration
|
||||
- [ ] Secure SSL configuration
|
||||
- [ ] Filtered configuration
|
||||
- [ ] Custom paths configuration
|
||||
|
||||
### Integration Guides
|
||||
- [ ] Integration with logserver
|
||||
- [ ] Docker Compose examples
|
||||
- [ ] Kubernetes DaemonSet example
|
||||
- [ ] Swarm mode configuration
|
||||
- [ ] Custom application integration
|
||||
|
||||
## Phase 11: Production Readiness (Priority 4)
|
||||
|
||||
### Security Hardening
|
||||
- [ ] Run as non-root user where possible
|
||||
- [ ] Minimize container capabilities
|
||||
- [ ] Add secrets management
|
||||
- [ ] Configure log sanitization
|
||||
- [ ] Add audit logging
|
||||
|
||||
### Updates & Maintenance
|
||||
- [ ] Add update notification
|
||||
- [ ] Create upgrade script
|
||||
- [ ] Add configuration migration
|
||||
- [ ] Document breaking changes
|
||||
- [ ] Create rollback procedure
|
||||
|
||||
### Compatibility
|
||||
- [ ] Test with different Filebeat versions
|
||||
- [ ] Verify Docker API compatibility
|
||||
- [ ] Test on different Linux distributions
|
||||
- [ ] Validate with various log formats
|
||||
- [ ] Ensure Logstash version compatibility
|
||||
|
||||
## Notes
|
||||
|
||||
### Design Principles
|
||||
1. **Minimal configuration**: Just needs LOGSERVER_HOST to work
|
||||
2. **Docker API access**: Use Docker API for driver-independent log collection
|
||||
3. **Automatic discovery**: Find all container logs without manual configuration
|
||||
4. **Reliability first**: Never lose logs, buffer locally if needed
|
||||
5. **Low overhead**: Minimal resource usage on host
|
||||
6. **Non-intrusive**: No changes to existing containers needed
|
||||
7. **Driver flexibility**: Allow containers to use any logging driver (especially `local`)
|
||||
|
||||
### Key Requirements
|
||||
- Must work with zero configuration beyond server address
|
||||
- Must use Docker API input, not file-based collection
|
||||
- Must support all Docker logging drivers (local, json-file, etc.)
|
||||
- Must handle Docker socket permissions properly
|
||||
- Must be resilient to network failures
|
||||
- Must not impact host performance significantly
|
||||
- Must preserve configuration on uninstall
|
||||
|
||||
### Testing Checklist
|
||||
- [ ] Validates with dropshell test-template
|
||||
- [ ] Connects to logserver successfully
|
||||
- [ ] Collects Docker logs automatically
|
||||
- [ ] Collects system logs properly
|
||||
- [ ] Handles server downtime gracefully
|
||||
- [ ] Reconnects automatically
|
||||
- [ ] Resource usage stays within limits
|
||||
- [ ] Uninstall preserves configuration
|
||||
@@ -1,7 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Define volume items for logclient container
|
||||
# These are used across backup, restore, create, and destroy operations
|
||||
|
||||
get_logclient_volumes() {
|
||||
echo "volume:datavolume:${DATA_VOLUME}" "volume:configvolume:${CONFIG_VOLUME}"
|
||||
}
|
||||
@@ -1,32 +0,0 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
||||
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
echo "Backing up ${CONTAINER_NAME} volumes..."
|
||||
|
||||
# Stop the container to ensure data consistency
|
||||
bash ./stop.sh || true
|
||||
|
||||
# Backup volumes
|
||||
BACKUP_DIR="${CONFIG_PATH}/backups/$(date +%Y%m%d_%H%M%S)"
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
# Export volumes
|
||||
for volume in $(get_logclient_volumes); do
|
||||
volume_name=$(echo $volume | cut -d: -f3)
|
||||
echo "Backing up volume: $volume_name"
|
||||
docker run --rm -v "$volume_name:/source:ro" -v "$BACKUP_DIR:/backup" alpine \
|
||||
tar -czf "/backup/${volume_name}.tar.gz" -C /source .
|
||||
done
|
||||
|
||||
# Backup configuration
|
||||
cp -r "${CONFIG_PATH}" "$BACKUP_DIR/config_backup"
|
||||
|
||||
echo "Backup completed to: $BACKUP_DIR"
|
||||
|
||||
# Restart the container
|
||||
bash ./start.sh
|
||||
|
||||
echo "Container restarted"
|
||||
@@ -1,16 +0,0 @@
|
||||
# Template identifier - MUST match the directory name
|
||||
TEMPLATE=logclient
|
||||
|
||||
# Requirements
|
||||
REQUIRES_HOST_ROOT=false # No root access on host needed
|
||||
REQUIRES_DOCKER=true # Docker is required
|
||||
REQUIRES_DOCKER_ROOT=false # Docker root privileges not specifically needed
|
||||
|
||||
# Docker image settings
|
||||
IMAGE_REGISTRY="docker.elastic.co"
|
||||
IMAGE_REPO="beats/filebeat"
|
||||
IMAGE_TAG="7.17.23"
|
||||
|
||||
# Volume definitions
|
||||
DATA_VOLUME=logclient_data
|
||||
CONFIG_VOLUME=logclient_config
|
||||
@@ -1,34 +1,16 @@
|
||||
# Service identification
|
||||
CONTAINER_NAME=logclient-filebeat
|
||||
# Log Client Configuration (Promtail)
|
||||
CONTAINER_NAME=logclient
|
||||
|
||||
# Server settings (REQUIRED by dropshell)
|
||||
SSH_USER="root"
|
||||
|
||||
# Docker image tag override (optional)
|
||||
IMAGE_TAG="7.17.23"
|
||||
|
||||
# REQUIRED: Log server connection
|
||||
LOGSERVER_HOST=
|
||||
LOGSERVER_PORT=5044
|
||||
LOGSERVER_PORT=3100
|
||||
|
||||
# REQUIRED: API Key Authentication
|
||||
API_KEY="" # Get from logserver admin using generate-api-key.sh
|
||||
# REQUIRED: Authentication (get from log server admin)
|
||||
LOKI_USER=logclient
|
||||
LOKI_PASSWORD=
|
||||
|
||||
# Performance tuning
|
||||
BULK_MAX_SIZE=2048 # Maximum batch size
|
||||
WORKER_THREADS=1 # Number of worker threads
|
||||
QUEUE_SIZE=4096 # Internal queue size
|
||||
MAX_BACKOFF=60s # Maximum retry backoff
|
||||
|
||||
# Filtering
|
||||
EXCLUDE_CONTAINERS="" # Comma-separated container names to exclude
|
||||
INCLUDE_CONTAINERS="" # Only include these containers (if set)
|
||||
EXCLUDE_LABELS="" # Exclude containers with these labels
|
||||
INCLUDE_LABELS="" # Only include containers with these labels
|
||||
|
||||
# Additional log paths
|
||||
CUSTOM_LOG_PATHS="" # Comma-separated additional paths to monitor
|
||||
|
||||
# Resource limits
|
||||
MAX_CPU=50 # Maximum CPU usage percentage
|
||||
MAX_MEMORY=200MB # Maximum memory usage
|
||||
# Optional: Set a custom hostname label (defaults to actual hostname)
|
||||
# HOSTNAME_LABEL=
|
||||
@@ -1,22 +0,0 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
||||
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
echo "WARNING: This will permanently delete all data for ${CONTAINER_NAME}"
|
||||
echo "This action cannot be undone!"
|
||||
echo ""
|
||||
|
||||
# Since scripts must be non-interactive, only proceed if explicitly called
|
||||
bash ./stop.sh || true
|
||||
_remove_container "$CONTAINER_NAME" || true
|
||||
|
||||
# Remove all volumes
|
||||
for volume in $(get_logclient_volumes); do
|
||||
volume_name=$(echo $volume | cut -d: -f3)
|
||||
echo "Removing volume: $volume_name"
|
||||
docker volume rm "$volume_name" 2>/dev/null || true
|
||||
done
|
||||
|
||||
echo "Service and all data destroyed"
|
||||
19
logclient/docker-compose.yml
Normal file
19
logclient/docker-compose.yml
Normal file
@@ -0,0 +1,19 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
promtail:
|
||||
image: grafana/promtail:latest
|
||||
container_name: ${CONTAINER_NAME}
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- /var/log:/var/log:ro
|
||||
- /var/lib/docker/containers:/var/lib/docker/containers:ro
|
||||
- ${CONFIG_PATH}/promtail.yaml:/etc/promtail/config.yml:ro
|
||||
- promtail_positions:/tmp/positions
|
||||
command: -config.file=/etc/promtail/config.yml
|
||||
environment:
|
||||
- HOSTNAME=${HOSTNAME_LABEL:-$(hostname)}
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
promtail_positions:
|
||||
@@ -2,52 +2,39 @@
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# Check required environment variables
|
||||
_check_required_env_vars "CONTAINER_NAME" "IMAGE_REGISTRY" "IMAGE_REPO" "IMAGE_TAG" "LOGSERVER_HOST" "LOGSERVER_PORT" "API_KEY"
|
||||
_check_required_env_vars "CONTAINER_NAME" "LOGSERVER_HOST" "LOGSERVER_PORT" "LOKI_PASSWORD"
|
||||
|
||||
# Validate API key
|
||||
if [ -z "$API_KEY" ]; then
|
||||
echo ""
|
||||
echo "ERROR: API_KEY is not configured"
|
||||
echo ""
|
||||
echo "To get an API key:"
|
||||
echo "1. On the logserver, run: ./generate-api-key.sh"
|
||||
echo "2. Enter this client's hostname when prompted"
|
||||
echo "3. Copy the generated API_KEY to this client's service.env"
|
||||
echo ""
|
||||
_die "Missing API_KEY configuration"
|
||||
fi
|
||||
|
||||
# Check Docker is available
|
||||
# Check Docker
|
||||
_check_docker_installed || _die "Docker test failed"
|
||||
|
||||
# Test connectivity to logserver
|
||||
echo "Testing connectivity to logserver at ${LOGSERVER_HOST}:${LOGSERVER_PORT}..."
|
||||
nc -zv "$LOGSERVER_HOST" "$LOGSERVER_PORT" 2>/dev/null || echo "WARNING: Cannot connect to logserver. Will retry when container starts."
|
||||
|
||||
# Pull the Docker image
|
||||
docker pull "$IMAGE_REGISTRY/$IMAGE_REPO:$IMAGE_TAG" || _die "Failed to pull Filebeat image"
|
||||
echo "Testing connectivity to log server at ${LOGSERVER_HOST}:${LOGSERVER_PORT}..."
|
||||
nc -zv "$LOGSERVER_HOST" "$LOGSERVER_PORT" 2>/dev/null || echo "WARNING: Cannot connect to log server. Will retry when container starts."
|
||||
|
||||
# Stop any existing container
|
||||
bash ./stop.sh || true
|
||||
bash ./stop.sh 2>/dev/null || true
|
||||
|
||||
# Remove old container
|
||||
_remove_container "$CONTAINER_NAME" || true
|
||||
|
||||
# Generate Filebeat configuration (with actual hostname)
|
||||
echo "Generating Filebeat configuration..."
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# Generate configuration
|
||||
echo "Generating configuration..."
|
||||
export HOSTNAME=$(hostname)
|
||||
bash "$SCRIPT_DIR/scripts/generate-config.sh" || _die "Failed to generate configuration"
|
||||
|
||||
# Create volumes using common function
|
||||
source "$SCRIPT_DIR/_volumes.sh"
|
||||
echo "Creating volumes..."
|
||||
create_items $(get_logclient_volumes)
|
||||
# Start the client
|
||||
echo "Starting Log Client..."
|
||||
docker compose up -d || _die "Failed to start"
|
||||
|
||||
# Start the new container
|
||||
bash ./start.sh || _die "Failed to start Filebeat"
|
||||
|
||||
echo "Installation of ${CONTAINER_NAME} complete"
|
||||
echo "Collecting logs from Docker API and shipping to ${LOGSERVER_HOST}:${LOGSERVER_PORT}"
|
||||
echo "Using API key authentication"
|
||||
echo ""
|
||||
echo "========================================="
|
||||
echo "Log Client Installed!"
|
||||
echo "========================================="
|
||||
echo ""
|
||||
echo "Shipping logs to: ${LOGSERVER_HOST}:${LOGSERVER_PORT}"
|
||||
echo "Using authentication: ${LOKI_USER:-logclient}"
|
||||
echo "Hostname label: $(hostname)"
|
||||
echo ""
|
||||
echo "Collecting:"
|
||||
echo " - All Docker container logs"
|
||||
echo " - System logs (/var/log)"
|
||||
echo ""
|
||||
echo "View logs at: http://${LOGSERVER_HOST}:3000"
|
||||
echo "========================================="
|
||||
5
logclient/logs.sh
Executable file
5
logclient/logs.sh
Executable file
@@ -0,0 +1,5 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
docker logs "${CONTAINER_NAME}" --tail 50
|
||||
@@ -1,47 +0,0 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
||||
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
if [ -z "$1" ]; then
|
||||
echo "Usage: $0 <backup_directory>"
|
||||
echo "Available backups:"
|
||||
ls -la "${CONFIG_PATH}/backups/" 2>/dev/null || echo "No backups found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
BACKUP_DIR="$1"
|
||||
|
||||
if [ ! -d "$BACKUP_DIR" ]; then
|
||||
_die "Backup directory not found: $BACKUP_DIR"
|
||||
fi
|
||||
|
||||
echo "Restoring from backup: $BACKUP_DIR"
|
||||
|
||||
# Stop the container
|
||||
bash ./stop.sh || true
|
||||
|
||||
# Restore volumes
|
||||
for volume in $(get_logclient_volumes); do
|
||||
volume_name=$(echo $volume | cut -d: -f3)
|
||||
backup_file="$BACKUP_DIR/${volume_name}.tar.gz"
|
||||
|
||||
if [ -f "$backup_file" ]; then
|
||||
echo "Restoring volume: $volume_name"
|
||||
docker run --rm -v "$volume_name:/target" -v "$BACKUP_DIR:/backup:ro" alpine \
|
||||
sh -c "rm -rf /target/* && tar -xzf /backup/${volume_name}.tar.gz -C /target"
|
||||
fi
|
||||
done
|
||||
|
||||
# Restore configuration if exists
|
||||
if [ -d "$BACKUP_DIR/config_backup" ]; then
|
||||
cp -r "$BACKUP_DIR/config_backup/"* "${CONFIG_PATH}/"
|
||||
fi
|
||||
|
||||
echo "Restore completed"
|
||||
|
||||
# Start the container
|
||||
bash ./start.sh
|
||||
|
||||
echo "Container restarted"
|
||||
@@ -1,163 +1,111 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Generate Filebeat configuration from template
|
||||
# This script creates a filebeat.yml configuration file with proper authentication
|
||||
# Generate Promtail configuration for Log Client
|
||||
|
||||
# Check required variables
|
||||
if [ -z "$LOGSERVER_HOST" ] || [ -z "$LOGSERVER_PORT" ]; then
|
||||
if [ -z "$LOGSERVER_HOST" ] || [ -z "$LOGSERVER_PORT" ] || [ -z "$LOKI_PASSWORD" ]; then
|
||||
echo "ERROR: Required environment variables not set"
|
||||
echo " LOGSERVER_HOST: ${LOGSERVER_HOST:-NOT SET}"
|
||||
echo " LOGSERVER_PORT: ${LOGSERVER_PORT:-NOT SET}"
|
||||
echo " LOKI_PASSWORD: ${LOKI_PASSWORD:-NOT SET}"
|
||||
echo ""
|
||||
echo "Please set these in config/service.env before running install"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Determine config directory - use CONFIG_PATH from dropshell or fallback
|
||||
if [ -n "$CONFIG_PATH" ]; then
|
||||
CONFIG_DIR="$CONFIG_PATH"
|
||||
elif [ -d "./config" ]; then
|
||||
CONFIG_DIR="./config"
|
||||
else
|
||||
CONFIG_DIR="."
|
||||
fi
|
||||
# Get actual hostname
|
||||
ACTUAL_HOSTNAME=${HOSTNAME_LABEL:-${HOSTNAME:-$(hostname 2>/dev/null || echo "unknown")}}
|
||||
|
||||
# Ensure config directory exists
|
||||
# Determine config directory
|
||||
CONFIG_DIR="${CONFIG_PATH:-./config}"
|
||||
mkdir -p "$CONFIG_DIR"
|
||||
|
||||
# Set defaults for variables if not set
|
||||
BULK_MAX_SIZE=${BULK_MAX_SIZE:-2048}
|
||||
WORKER_THREADS=${WORKER_THREADS:-1}
|
||||
QUEUE_SIZE=${QUEUE_SIZE:-4096}
|
||||
MAX_BACKOFF=${MAX_BACKOFF:-60s}
|
||||
# Generate promtail.yaml configuration
|
||||
cat > "$CONFIG_DIR/promtail.yaml" << EOF
|
||||
server:
|
||||
http_listen_port: 9080
|
||||
grpc_listen_port: 0
|
||||
|
||||
# Get actual hostname from the host system
|
||||
ACTUAL_HOSTNAME=${HOSTNAME:-$(hostname 2>/dev/null || echo "unknown")}
|
||||
positions:
|
||||
filename: /tmp/positions/positions.yaml
|
||||
|
||||
# Generate filebeat.yml configuration with variable substitution
|
||||
(
|
||||
cat << 'TEMPLATE_EOF'
|
||||
# Filebeat Configuration for LogClient
|
||||
# Generated by generate-config.sh
|
||||
clients:
|
||||
- url: http://${LOKI_USER:-logclient}:${LOKI_PASSWORD}@${LOGSERVER_HOST}:${LOGSERVER_PORT}/loki/api/v1/push
|
||||
# Authentication via URL (HTTP Basic Auth)
|
||||
|
||||
# ======================== Docker Input Configuration =========================
|
||||
# Use Docker input to collect logs via Docker API
|
||||
filebeat.inputs:
|
||||
- type: docker
|
||||
enabled: true
|
||||
# Collect from all containers
|
||||
containers.ids:
|
||||
- '*'
|
||||
# Collect both stdout and stderr
|
||||
containers.stream: all
|
||||
# Combine partial log lines
|
||||
combine_partial: true
|
||||
# Add Docker metadata
|
||||
processors:
|
||||
- add_docker_metadata:
|
||||
host: "unix:///var/run/docker.sock"
|
||||
scrape_configs:
|
||||
# Docker container logs via Docker API
|
||||
- job_name: docker
|
||||
docker_sd_configs:
|
||||
- host: unix:///var/run/docker.sock
|
||||
refresh_interval: 5s
|
||||
relabel_configs:
|
||||
- source_labels: ['__meta_docker_container_name']
|
||||
regex: '/(.*)'
|
||||
target_label: 'container_name'
|
||||
- source_labels: ['__meta_docker_container_id']
|
||||
target_label: 'container_id'
|
||||
- source_labels: ['__meta_docker_container_image']
|
||||
target_label: 'image'
|
||||
- target_label: 'hostname'
|
||||
replacement: '${ACTUAL_HOSTNAME}'
|
||||
- target_label: 'job'
|
||||
replacement: 'docker'
|
||||
|
||||
# ======================== System Logs Configuration ==========================
|
||||
- type: log
|
||||
enabled: true
|
||||
paths:
|
||||
- /var/log/syslog
|
||||
- /var/log/messages
|
||||
exclude_lines: ['^#']
|
||||
fields:
|
||||
log_type: syslog
|
||||
# System logs
|
||||
- job_name: syslog
|
||||
static_configs:
|
||||
- targets:
|
||||
- localhost
|
||||
labels:
|
||||
job: syslog
|
||||
hostname: ${ACTUAL_HOSTNAME}
|
||||
__path__: /var/log/syslog
|
||||
|
||||
- type: log
|
||||
enabled: true
|
||||
paths:
|
||||
- /var/log/auth.log
|
||||
- /var/log/secure
|
||||
exclude_lines: ['^#']
|
||||
fields:
|
||||
log_type: auth
|
||||
- job_name: messages
|
||||
static_configs:
|
||||
- targets:
|
||||
- localhost
|
||||
labels:
|
||||
job: messages
|
||||
hostname: ${ACTUAL_HOSTNAME}
|
||||
__path__: /var/log/messages
|
||||
|
||||
# ======================== Processors Configuration ===========================
|
||||
processors:
|
||||
- add_host_metadata:
|
||||
when.not.contains:
|
||||
tags: forwarded
|
||||
# Override hostname with actual host's hostname
|
||||
- add_fields:
|
||||
target: agent
|
||||
fields:
|
||||
hostname: __ACTUAL_HOSTNAME__
|
||||
- add_fields:
|
||||
target: host
|
||||
fields:
|
||||
name: __ACTUAL_HOSTNAME__
|
||||
- job_name: auth
|
||||
static_configs:
|
||||
- targets:
|
||||
- localhost
|
||||
labels:
|
||||
job: auth
|
||||
hostname: ${ACTUAL_HOSTNAME}
|
||||
__path__: /var/log/auth.log
|
||||
|
||||
# ======================== Output Configuration ===============================
|
||||
output.logstash:
|
||||
hosts: ["__LOGSERVER_HOST__:__LOGSERVER_PORT__"]
|
||||
# SSL/TLS configuration
|
||||
ssl.enabled: false # Set to true when using TLS
|
||||
ssl.verification_mode: none # Set to full in production with proper certs
|
||||
# Docker container JSON logs (backup method)
|
||||
- job_name: containers
|
||||
static_configs:
|
||||
- targets:
|
||||
- localhost
|
||||
labels:
|
||||
job: containers
|
||||
hostname: ${ACTUAL_HOSTNAME}
|
||||
__path__: /var/lib/docker/containers/*/*-json.log
|
||||
pipeline_stages:
|
||||
- json:
|
||||
expressions:
|
||||
output: log
|
||||
stream: stream
|
||||
time: time
|
||||
- timestamp:
|
||||
source: time
|
||||
format: RFC3339Nano
|
||||
- labels:
|
||||
stream:
|
||||
- output:
|
||||
source: output
|
||||
EOF
|
||||
|
||||
# Performance settings
|
||||
bulk_max_size: __BULK_MAX_SIZE__
|
||||
worker: __WORKER_THREADS__ # Must be >= 1
|
||||
compression_level: 3
|
||||
|
||||
# Retry configuration
|
||||
max_retries: 3
|
||||
backoff.init: 1s
|
||||
backoff.max: __MAX_BACKOFF__
|
||||
|
||||
# ======================== Global Fields =======================================
|
||||
# Add API key as a field to all events
|
||||
fields:
|
||||
api_key: "__API_KEY__"
|
||||
fields_under_root: false
|
||||
|
||||
# ======================== Queue Configuration ================================
|
||||
queue.mem:
|
||||
events: __QUEUE_SIZE__
|
||||
flush.min_events: 512
|
||||
flush.timeout: 5s
|
||||
|
||||
# ======================== Logging Configuration ==============================
|
||||
logging.level: info
|
||||
logging.to_files: true
|
||||
logging.files:
|
||||
path: /usr/share/filebeat/data/logs
|
||||
name: filebeat
|
||||
keepfiles: 3
|
||||
permissions: 0600
|
||||
|
||||
# ======================== Monitoring ==========================================
|
||||
monitoring.enabled: false
|
||||
http.enabled: true
|
||||
http.host: 0.0.0.0
|
||||
http.port: 5066
|
||||
|
||||
# ======================== File Permissions ====================================
|
||||
# Set strict permissions (disabled for Docker)
|
||||
# filebeat.config.modules.path: ${path.config}/modules.d/*.yml
|
||||
TEMPLATE_EOF
|
||||
) | sed -e "s|__LOGSERVER_HOST__|${LOGSERVER_HOST}|g" \
|
||||
-e "s|__LOGSERVER_PORT__|${LOGSERVER_PORT}|g" \
|
||||
-e "s|__API_KEY__|${API_KEY}|g" \
|
||||
-e "s|__BULK_MAX_SIZE__|${BULK_MAX_SIZE}|g" \
|
||||
-e "s|__WORKER_THREADS__|${WORKER_THREADS}|g" \
|
||||
-e "s|__QUEUE_SIZE__|${QUEUE_SIZE}|g" \
|
||||
-e "s|__MAX_BACKOFF__|${MAX_BACKOFF}|g" \
|
||||
-e "s|__ACTUAL_HOSTNAME__|${ACTUAL_HOSTNAME}|g" > "$CONFIG_DIR/filebeat.yml"
|
||||
|
||||
echo "Filebeat configuration generated at: $CONFIG_DIR/filebeat.yml"
|
||||
echo "Promtail configuration generated at: $CONFIG_DIR/promtail.yaml"
|
||||
echo "Configuration:"
|
||||
echo " LOGSERVER_HOST: ${LOGSERVER_HOST}"
|
||||
echo " LOGSERVER_PORT: ${LOGSERVER_PORT}"
|
||||
echo " API_KEY: ${API_KEY:+[SET]}"
|
||||
echo " WORKER_THREADS: ${WORKER_THREADS}"
|
||||
|
||||
# Additional warning if API_KEY is not set
|
||||
if [ -z "$API_KEY" ]; then
|
||||
echo ""
|
||||
echo "WARNING: API_KEY is not set - logs may be rejected by the server"
|
||||
echo "Get an API key from the LogServer admin using generate-api-key.sh"
|
||||
fi
|
||||
echo " HOSTNAME: ${ACTUAL_HOSTNAME}"
|
||||
@@ -1,36 +1,6 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME" "IMAGE_REGISTRY" "IMAGE_REPO" "IMAGE_TAG"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
# Check that config file exists
|
||||
if [ ! -f "${CONFIG_PATH}/filebeat.yml" ]; then
|
||||
_die "filebeat.yml not found in ${CONFIG_PATH}/filebeat.yml"
|
||||
fi
|
||||
|
||||
# Create Docker command
|
||||
DOCKER_RUN_CMD="docker run -d \
|
||||
--name $CONTAINER_NAME \
|
||||
--restart unless-stopped \
|
||||
--user root \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock:ro \
|
||||
-v /var/log:/var/log:ro \
|
||||
-v ${CONFIG_PATH}:/usr/share/filebeat/config:ro \
|
||||
-v ${DATA_VOLUME}:/usr/share/filebeat/data \
|
||||
-e LOGSERVER_HOST=${LOGSERVER_HOST} \
|
||||
-e LOGSERVER_PORT=${LOGSERVER_PORT} \
|
||||
-e API_KEY=${API_KEY} \
|
||||
-e HOSTNAME=$(hostname) \
|
||||
$IMAGE_REGISTRY/$IMAGE_REPO:$IMAGE_TAG \
|
||||
filebeat -e -strict.perms=false \
|
||||
-c /usr/share/filebeat/config/filebeat.yml"
|
||||
|
||||
if ! _create_and_start_container "$DOCKER_RUN_CMD" "$CONTAINER_NAME"; then
|
||||
_die "Failed to start container ${CONTAINER_NAME}"
|
||||
fi
|
||||
|
||||
# Check if the container is running
|
||||
if ! _is_container_running "$CONTAINER_NAME"; then
|
||||
_die "Container ${CONTAINER_NAME} is not running"
|
||||
fi
|
||||
|
||||
echo "Container ${CONTAINER_NAME} started"
|
||||
docker compose up -d || _die "Failed to start"
|
||||
echo "Log Client started"
|
||||
@@ -2,30 +2,10 @@
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
# Check if container exists
|
||||
if ! docker ps -a --format "{{.Names}}" | grep -q "^${CONTAINER_NAME}$"; then
|
||||
echo "Unknown"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Check container state
|
||||
STATE=$(docker inspect -f '{{.State.Status}}' "$CONTAINER_NAME" 2>/dev/null)
|
||||
case "$STATE" in
|
||||
running)
|
||||
# Additional check: verify connection to logserver
|
||||
if docker logs "$CONTAINER_NAME" 2>&1 | tail -20 | grep -q "ERROR"; then
|
||||
echo "Error"
|
||||
else
|
||||
echo "Running"
|
||||
fi
|
||||
;;
|
||||
exited|stopped)
|
||||
echo "Stopped"
|
||||
;;
|
||||
restarting|paused)
|
||||
echo "Error"
|
||||
;;
|
||||
*)
|
||||
echo "Unknown"
|
||||
;;
|
||||
esac
|
||||
if docker ps | grep -q "${CONTAINER_NAME}"; then
|
||||
echo "Running"
|
||||
echo " Shipping logs to: ${LOGSERVER_HOST}:${LOGSERVER_PORT}"
|
||||
docker ps --format "table {{.Names}}\t{{.Status}}" | grep "${CONTAINER_NAME}"
|
||||
else
|
||||
echo "Stopped"
|
||||
fi
|
||||
@@ -2,6 +2,5 @@
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
docker stop "$CONTAINER_NAME" 2>/dev/null || true
|
||||
|
||||
echo "Filebeat stopped"
|
||||
docker compose stop || _die "Failed to stop"
|
||||
echo "Log Client stopped"
|
||||
@@ -2,16 +2,5 @@
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
# Stop the container
|
||||
bash ./stop.sh || _die "Failed to stop container"
|
||||
|
||||
# Remove the container
|
||||
_remove_container "$CONTAINER_NAME" || _die "Failed to remove container"
|
||||
|
||||
# CRITICAL: Never remove data volumes in uninstall.sh!
|
||||
# Data volumes must be preserved for potential reinstallation
|
||||
# Configuration and certificates must be preserved
|
||||
|
||||
echo "Uninstallation of ${CONTAINER_NAME} complete"
|
||||
echo "Note: Configuration and certificates have been preserved."
|
||||
echo "To remove all data, use destroy.sh"
|
||||
docker compose down || true
|
||||
echo "Log Client uninstalled"
|
||||
Reference in New Issue
Block a user