swtich from ELK to Loki!
All checks were successful
Test and Publish Templates / test-and-publish (push) Successful in 40s
All checks were successful
Test and Publish Templates / test-and-publish (push) Successful in 40s
This commit is contained in:
Your Name
@@ -1,389 +0,0 @@
|
||||
# Dropshell LogClient Template
|
||||
|
||||
An auto-configuring Filebeat agent that collects Docker container logs via the Docker API and system logs, shipping them to a centralized logging server with minimal configuration.
|
||||
|
||||
## Overview
|
||||
|
||||
This template deploys a lightweight Filebeat agent that:
|
||||
- Uses Docker API to collect logs from all containers (regardless of logging driver)
|
||||
- Allows containers to use any Docker logging driver (json-file, local, journald, etc.)
|
||||
- Collects system logs (syslog, auth logs, kernel logs)
|
||||
- Ships logs to a centralized ELK stack (logserver)
|
||||
- Requires minimal configuration - just the server address
|
||||
- Handles connection failures with local buffering
|
||||
- Auto-reconnects when the server becomes available
|
||||
|
||||
## Features
|
||||
|
||||
### Docker API Log Collection
|
||||
- **Direct API access**: Reads logs via Docker API, not from files
|
||||
- **Driver independent**: Works with any Docker logging driver (local, json-file, journald)
|
||||
- **Automatic discovery**: Finds all running containers dynamically
|
||||
- **Container metadata**: Enriches logs with container names, images, labels
|
||||
- **Real-time streaming**: Gets logs as they're generated
|
||||
- **Multi-line handling**: Properly handles stack traces and multi-line logs
|
||||
- **JSON parsing**: Automatically parses JSON-formatted logs
|
||||
- **Label-based filtering**: Can include/exclude containers based on labels
|
||||
|
||||
### System Log Collection
|
||||
- **/var/log/syslog** or **/var/log/messages**: System events
|
||||
- **/var/log/auth.log** or **/var/log/secure**: Authentication events
|
||||
- **/var/log/kern.log**: Kernel messages
|
||||
- **journald**: SystemD journal (if available)
|
||||
- **Custom paths**: Configurable additional log paths
|
||||
|
||||
### Reliability Features
|
||||
- **Local buffering**: Stores logs locally when server is unreachable
|
||||
- **Automatic retry**: Reconnects automatically with exponential backoff
|
||||
- **Compression**: Compresses logs before sending to save bandwidth
|
||||
- **Secure transmission**: Optional TLS/SSL encryption
|
||||
- **Backpressure handling**: Slows down when server is overwhelmed
|
||||
|
||||
## Architecture
|
||||
|
||||
### How It Works
|
||||
1. Filebeat runs as a container with Docker socket access
|
||||
2. Uses Docker API to stream logs from all containers
|
||||
3. Monitors Docker API for container lifecycle events
|
||||
4. Automatically starts collecting logs from new containers
|
||||
5. Reads host system logs from mounted volumes
|
||||
6. Ships all logs to configured Logstash/Elasticsearch endpoint
|
||||
7. Maintains connection state and buffering information
|
||||
|
||||
### Log Flow
|
||||
```
|
||||
Docker Containers → Docker API →
|
||||
↘
|
||||
Filebeat → Logstash → Elasticsearch → Kibana
|
||||
↗
|
||||
System Logs (mounted volumes) →
|
||||
```
|
||||
|
||||
### Why Docker API Instead of Log Files?
|
||||
- **Logging driver flexibility**: Containers can use `local`, `json-file`, `journald`, or any driver
|
||||
- **No log file management**: Don't need to worry about log rotation or file paths
|
||||
- **Better performance**: Direct streaming without file I/O overhead
|
||||
- **Consistent access**: Same method regardless of storage backend
|
||||
- **Real-time streaming**: Get logs immediately as they're generated
|
||||
- **Simplified permissions**: Only need Docker socket access
|
||||
|
||||
## Minimum Configuration
|
||||
|
||||
The template requires minimal configuration - server address and authentication:
|
||||
|
||||
```bash
|
||||
# Required - Server connection
|
||||
LOGSERVER_HOST=192.168.1.100
|
||||
LOGSERVER_PORT=5044
|
||||
|
||||
# Required - Authentication (choose one method)
|
||||
AUTH_MODE=mtls # Options: mtls, apikey, basic
|
||||
|
||||
# For mTLS authentication
|
||||
CLIENT_CERT_PATH=/certs/client.crt
|
||||
CLIENT_KEY_PATH=/certs/client.key
|
||||
CA_CERT_PATH=/certs/ca.crt
|
||||
|
||||
# For API key authentication
|
||||
API_KEY=your-api-key-here
|
||||
|
||||
# For basic auth (not recommended)
|
||||
USERNAME=filebeat
|
||||
PASSWORD=changeme
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
### Environment Variables (service.env)
|
||||
|
||||
```bash
|
||||
# REQUIRED: Log server connection
|
||||
LOGSERVER_HOST=logserver.example.com
|
||||
LOGSERVER_PORT=5044
|
||||
|
||||
# REQUIRED: Authentication method
|
||||
AUTH_MODE=mtls # mtls, apikey, or basic
|
||||
|
||||
# mTLS Authentication (if AUTH_MODE=mtls)
|
||||
CLIENT_CERT_PATH=/certs/${HOSTNAME}.crt
|
||||
CLIENT_KEY_PATH=/certs/${HOSTNAME}.key
|
||||
CA_CERT_PATH=/certs/ca.crt
|
||||
SSL_VERIFICATION_MODE=full
|
||||
|
||||
# API Key Authentication (if AUTH_MODE=apikey)
|
||||
API_KEY="" # Will be provided by logserver admin
|
||||
|
||||
# Basic Authentication (if AUTH_MODE=basic)
|
||||
USERNAME=filebeat
|
||||
PASSWORD=changeme
|
||||
|
||||
# Optional: Performance tuning
|
||||
BULK_MAX_SIZE=2048 # Maximum batch size
|
||||
WORKER_THREADS=1 # Number of worker threads
|
||||
QUEUE_SIZE=4096 # Internal queue size
|
||||
MAX_BACKOFF=60s # Maximum retry backoff
|
||||
|
||||
# Optional: Filtering
|
||||
EXCLUDE_CONTAINERS="" # Comma-separated container names to exclude
|
||||
INCLUDE_CONTAINERS="" # Only include these containers (if set)
|
||||
EXCLUDE_LABELS="" # Exclude containers with these labels
|
||||
INCLUDE_LABELS="" # Only include containers with these labels
|
||||
|
||||
# Optional: Additional log paths
|
||||
CUSTOM_LOG_PATHS="" # Comma-separated additional paths to monitor
|
||||
|
||||
# Optional: Resource limits
|
||||
MAX_CPU=50 # Maximum CPU usage percentage
|
||||
MAX_MEMORY=200MB # Maximum memory usage
|
||||
```
|
||||
|
||||
## Collected Log Types
|
||||
|
||||
### Docker Container Logs (via Docker API)
|
||||
- **stdout/stderr**: All container output regardless of logging driver
|
||||
- **Container metadata**: Name, ID, image, labels
|
||||
- **Docker events**: Start, stop, die, kill events
|
||||
- **Health check results**: If configured
|
||||
- **Works with all logging drivers**: local, json-file, journald, syslog, etc.
|
||||
|
||||
### System Logs
|
||||
- **System messages**: Service starts/stops, errors
|
||||
- **Authentication**: SSH logins, sudo usage
|
||||
- **Kernel messages**: Hardware events, driver messages
|
||||
- **Package management**: apt/yum operations
|
||||
- **Cron jobs**: Scheduled task execution
|
||||
|
||||
## Log Enrichment
|
||||
|
||||
Logs are automatically enriched with:
|
||||
- **Hostname**: Source host identification
|
||||
- **Timestamp**: Precise event time with timezone
|
||||
- **Log level**: Parsed from log content when possible
|
||||
- **Container info**: For Docker logs
|
||||
- **Process info**: PID, command for system logs
|
||||
- **File path**: Source log file
|
||||
|
||||
## Resource Requirements
|
||||
|
||||
### Minimum
|
||||
- CPU: 0.5 cores
|
||||
- RAM: 128MB
|
||||
- Storage: 1GB (for buffer)
|
||||
|
||||
### Typical Usage
|
||||
- CPU: 1-5% of one core
|
||||
- RAM: 150-200MB
|
||||
- Network: Varies with log volume
|
||||
- Storage: Depends on buffer size
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
1. A running logserver (ELK stack)
|
||||
2. Network connectivity to logserver
|
||||
3. Docker installed on host
|
||||
4. Authentication credentials from logserver admin
|
||||
|
||||
### Setup Authentication
|
||||
|
||||
#### For mTLS (Recommended):
|
||||
```bash
|
||||
# Get client certificate from logserver admin
|
||||
# They will run: dropshell exec logserver /scripts/generate-client-cert.sh $(hostname)
|
||||
# Copy the generated certificate files to this client
|
||||
mkdir -p /etc/dropshell/certs
|
||||
# Copy ca.crt, client.crt, and client.key to /etc/dropshell/certs/
|
||||
```
|
||||
|
||||
#### For API Key:
|
||||
```bash
|
||||
# Get API key from logserver admin
|
||||
# They will run: dropshell exec logserver /scripts/generate-api-key.sh $(hostname)
|
||||
# Add the API key to service.env
|
||||
```
|
||||
|
||||
### Deploy
|
||||
```bash
|
||||
# Configure authentication in service.env
|
||||
dropshell install logclient
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Check Status
|
||||
```bash
|
||||
dropshell status logclient
|
||||
```
|
||||
|
||||
### View Filebeat Logs
|
||||
```bash
|
||||
dropshell logs logclient
|
||||
```
|
||||
|
||||
### Verify Connectivity
|
||||
```bash
|
||||
# Check if logs are being shipped
|
||||
docker exec logclient-filebeat filebeat test output
|
||||
```
|
||||
|
||||
### Monitor Metrics
|
||||
```bash
|
||||
# View Filebeat statistics
|
||||
docker exec logclient-filebeat curl -s http://localhost:5066/stats
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### No Logs Appearing on Server
|
||||
|
||||
1. **Check connectivity**
|
||||
```bash
|
||||
telnet $LOGSERVER_HOST $LOGSERVER_PORT
|
||||
```
|
||||
|
||||
2. **Verify Filebeat is running**
|
||||
```bash
|
||||
dropshell status logclient
|
||||
```
|
||||
|
||||
3. **Check Filebeat logs**
|
||||
```bash
|
||||
dropshell logs logclient | tail -50
|
||||
```
|
||||
|
||||
4. **Test configuration**
|
||||
```bash
|
||||
docker exec logclient-filebeat filebeat test config
|
||||
```
|
||||
|
||||
### High CPU Usage
|
||||
|
||||
1. Reduce worker threads in service.env
|
||||
2. Increase bulk_max_size to send larger batches
|
||||
3. Add exclude filters for noisy containers
|
||||
|
||||
### Missing Container Logs
|
||||
|
||||
1. Verify Docker socket is mounted
|
||||
2. Check container isn't in exclude list
|
||||
3. Ensure Filebeat has permissions to Docker socket
|
||||
4. Verify container is actually producing output
|
||||
5. Check if container uses a supported logging driver
|
||||
|
||||
### Buffer Full Errors
|
||||
|
||||
1. Increase queue_size in service.env
|
||||
2. Check network connectivity to server
|
||||
3. Verify server isn't overwhelmed
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Authentication**:
|
||||
- Always use mTLS or API keys in production
|
||||
- Never use basic auth except for testing
|
||||
- Store credentials securely
|
||||
- Rotate certificates/keys regularly
|
||||
|
||||
2. **Docker Socket Access**:
|
||||
- Requires Docker socket access to read logs via API
|
||||
- Understand security implications of socket access
|
||||
- Consider read-only socket access if possible
|
||||
|
||||
3. **Network Security**:
|
||||
- All connections are TLS encrypted
|
||||
- Verify server certificates
|
||||
- Configure firewall rules appropriately
|
||||
- Use private networks when possible
|
||||
|
||||
4. **Data Protection**:
|
||||
- Logs may contain sensitive data
|
||||
- Filter sensitive information before shipping
|
||||
- Exclude containers with sensitive data if needed
|
||||
|
||||
5. **Resource Limits**:
|
||||
- Set CPU and memory limits
|
||||
- Monitor resource usage
|
||||
- Prevent resource exhaustion attacks
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### For High-Volume Environments
|
||||
```bash
|
||||
# Increase workers and batch size
|
||||
WORKER_THREADS=4
|
||||
BULK_MAX_SIZE=4096
|
||||
QUEUE_SIZE=8192
|
||||
```
|
||||
|
||||
### For Low-Resource Hosts
|
||||
```bash
|
||||
# Reduce resource usage
|
||||
WORKER_THREADS=1
|
||||
BULK_MAX_SIZE=512
|
||||
MAX_MEMORY=100MB
|
||||
MAX_CPU=25
|
||||
```
|
||||
|
||||
### Network Optimization
|
||||
```bash
|
||||
# Enable compression (CPU vs bandwidth tradeoff)
|
||||
COMPRESSION_LEVEL=3 # 0-9, higher = more compression
|
||||
```
|
||||
|
||||
## Integration with LogServer
|
||||
|
||||
This template is designed to work seamlessly with the `logserver` template:
|
||||
|
||||
1. Deploy logserver first
|
||||
2. Note the logserver's IP/hostname
|
||||
3. Configure logclient with server address
|
||||
4. Logs automatically start flowing
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Regular Tasks
|
||||
- Monitor buffer usage
|
||||
- Check for connection errors
|
||||
- Review excluded/included containers
|
||||
- Update Filebeat version
|
||||
|
||||
### Logs Rotation
|
||||
Filebeat handles log rotation automatically:
|
||||
- Detects renamed/rotated files
|
||||
- Continues reading from correct position
|
||||
- Cleans up old file handles
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
### Custom Filebeat Configuration
|
||||
Create a custom `filebeat.yml` in the config directory for advanced scenarios:
|
||||
- Custom processors
|
||||
- Additional inputs
|
||||
- Complex filtering rules
|
||||
- Multiple outputs
|
||||
|
||||
### Docker Labels for Control
|
||||
Control log collection per container:
|
||||
```yaml
|
||||
# In docker-compose.yml
|
||||
services:
|
||||
myapp:
|
||||
logging:
|
||||
driver: local # Can use any driver - Filebeat reads via API
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
labels:
|
||||
- "filebeat.enable=false" # Exclude this container
|
||||
- "filebeat.multiline.pattern=^\\[" # Custom multiline pattern
|
||||
```
|
||||
|
||||
### Logging Driver Compatibility
|
||||
The Docker API input works with all Docker logging drivers:
|
||||
- **local**: Recommended for production (efficient, no file access needed)
|
||||
- **json-file**: Traditional default driver
|
||||
- **journald**: SystemD journal integration
|
||||
- **syslog**: Forward to syslog
|
||||
- **none**: Disables logging (Filebeat won't collect)
|
||||
|
||||
You can use the `local` driver for better performance since Filebeat doesn't need to read files.
|
||||
@@ -1,34 +1,56 @@
|
||||
# LogClient
|
||||
# Log Client (Promtail)
|
||||
|
||||
Ships Docker container and system logs to LogServer using Filebeat.
|
||||
Ships Docker and system logs to Log Server. Zero configuration!
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. **Get API Key**
|
||||
- Ask LogServer admin to run `./generate-api-key.sh`
|
||||
- They'll provide your API key
|
||||
|
||||
2. **Configure**
|
||||
Edit `config/service.env`:
|
||||
1. **Configure** (edit `config/service.env`):
|
||||
```bash
|
||||
LOGSERVER_HOST=<server-ip>
|
||||
LOGSERVER_PORT=5044
|
||||
API_KEY=<your-api-key>
|
||||
LOGSERVER_HOST=<server-ip> # Your log server IP
|
||||
LOGSERVER_PORT=3100 # Default Loki port
|
||||
LOKI_USER=logclient # Default username
|
||||
LOKI_PASSWORD=<password> # Get from server admin
|
||||
```
|
||||
|
||||
3. **Install**
|
||||
2. **Install**:
|
||||
```bash
|
||||
dropshell install logclient
|
||||
```
|
||||
|
||||
## What It Does
|
||||
- Collects all Docker container logs via API
|
||||
- Collects system logs (/var/log)
|
||||
- Ships to central LogServer
|
||||
- Works with any Docker logging driver
|
||||
That's it! Logs are now shipping to your central server.
|
||||
|
||||
## Requirements
|
||||
- Docker socket access
|
||||
- Network connection to LogServer port 5044
|
||||
## What Gets Sent
|
||||
- All Docker container logs (via Docker API)
|
||||
- System logs (/var/log/syslog)
|
||||
- Auth logs (/var/log/auth.log)
|
||||
- Hostname is automatically included
|
||||
|
||||
See [DOCUMENTATION.md](DOCUMENTATION.md) for full details.
|
||||
## View Your Logs
|
||||
Go to: `http://<logserver-ip>:3000`
|
||||
- Click Dashboards → Central Logs
|
||||
- Your server's logs appear with its hostname
|
||||
|
||||
## Features
|
||||
- **Zero maintenance** - Just runs
|
||||
- **Lightweight** - ~20MB RAM
|
||||
- **Automatic** - Finds all containers
|
||||
- **Labeled** - Hostname included automatically
|
||||
|
||||
## Simple Authentication
|
||||
Just a username/password - much simpler than ELK's API key system!
|
||||
|
||||
## Troubleshooting
|
||||
Check if client is running:
|
||||
```bash
|
||||
./status.sh
|
||||
```
|
||||
|
||||
View client logs:
|
||||
```bash
|
||||
./logs.sh
|
||||
```
|
||||
|
||||
Test connection to server:
|
||||
```bash
|
||||
nc -zv <logserver-ip> 3100
|
||||
```
|
||||
@@ -1,294 +0,0 @@
|
||||
# LogClient Template - Implementation TODO
|
||||
|
||||
## Phase 1: Core Infrastructure (Priority 1)
|
||||
|
||||
### Configuration Files
|
||||
- [ ] Create `config/.template_info.env` with template metadata
|
||||
- [ ] Create `config/service.env` with minimal required settings
|
||||
- [ ] Define LOGSERVER_HOST and LOGSERVER_PORT variables
|
||||
- [ ] Add AUTH_MODE variable (mtls, apikey, basic)
|
||||
- [ ] Add certificate/key path variables for mTLS
|
||||
- [ ] Add API_KEY variable for API key auth
|
||||
- [ ] Add USERNAME/PASSWORD for basic auth
|
||||
- [ ] Add optional performance and filtering variables
|
||||
- [ ] Set sensible defaults where possible
|
||||
|
||||
### Filebeat Configuration
|
||||
- [ ] Create base `filebeat.yml` configuration template
|
||||
- [ ] Configure Docker input using Docker API (not autodiscover with hints)
|
||||
- [ ] Set containers.ids: ["*"] to collect from all containers
|
||||
- [ ] Set up system log inputs for host logs
|
||||
- [ ] Configure output to Logstash
|
||||
- [ ] Add error handling and retry logic
|
||||
- [ ] Set up local disk buffering
|
||||
- [ ] Configure stream: "all" to get both stdout and stderr
|
||||
|
||||
### Required Scripts
|
||||
- [ ] Implement `install.sh` - Pull Filebeat image, configure auth, start
|
||||
- [ ] Implement `uninstall.sh` - Stop and remove container (preserve config and certs)
|
||||
- [ ] Implement `start.sh` - Start Filebeat with auth config and proper mounts
|
||||
- [ ] Implement `stop.sh` - Gracefully stop Filebeat
|
||||
- [ ] Implement `status.sh` - Check Filebeat health and auth status
|
||||
- [ ] Create `setup-auth.sh` - Helper script to configure authentication
|
||||
|
||||
## Phase 2: Docker API Log Collection (Priority 1)
|
||||
|
||||
### Docker API Input Configuration
|
||||
- [ ] Configure Docker input type (NOT autodiscover, use direct Docker input)
|
||||
- [ ] Mount Docker socket (/var/run/docker.sock) with proper permissions
|
||||
- [ ] Configure Docker API endpoint (unix:///var/run/docker.sock)
|
||||
- [ ] Set up real-time log streaming from Docker daemon
|
||||
- [ ] Enable collection from ALL logging drivers (local, json-file, journald, etc.)
|
||||
- [ ] Configure since_time to get recent logs on startup
|
||||
|
||||
### Container Metadata Extraction
|
||||
- [ ] Extract container name, ID, image name, and image tag
|
||||
- [ ] Map container labels to fields
|
||||
- [ ] Handle docker-compose project names and service names
|
||||
- [ ] Add container state information
|
||||
- [ ] Include container environment variables (filtered)
|
||||
- [ ] Handle container lifecycle events (start, stop, restart)
|
||||
|
||||
### Container Filtering
|
||||
- [ ] Implement include/exclude by container name patterns
|
||||
- [ ] Add label-based filtering (containers.labels)
|
||||
- [ ] Create ignore patterns for system containers
|
||||
- [ ] Add support for custom filter expressions
|
||||
- [ ] Configure combine_partial to handle partial log lines
|
||||
- [ ] Document filtering examples with Docker API syntax
|
||||
|
||||
## Phase 3: System Log Collection (Priority 1)
|
||||
|
||||
### Log File Inputs
|
||||
- [ ] Configure /var/log/syslog or /var/log/messages
|
||||
- [ ] Add /var/log/auth.log or /var/log/secure
|
||||
- [ ] Include /var/log/kern.log
|
||||
- [ ] Monitor /var/log/dpkg.log or /var/log/yum.log
|
||||
- [ ] Add custom log path support via environment variable
|
||||
|
||||
### Journald Integration
|
||||
- [ ] Detect if systemd/journald is available
|
||||
- [ ] Configure journald input if present
|
||||
- [ ] Set up unit filtering
|
||||
- [ ] Extract systemd metadata
|
||||
- [ ] Handle binary journal format
|
||||
|
||||
### Log Parsing
|
||||
- [ ] Configure syslog parsing
|
||||
- [ ] Extract severity levels
|
||||
- [ ] Parse timestamps correctly
|
||||
- [ ] Handle different syslog formats
|
||||
- [ ] Add timezone handling
|
||||
|
||||
## Phase 4: Output Configuration (Priority 1)
|
||||
|
||||
### Logstash Output
|
||||
- [ ] Configure primary Logstash endpoint
|
||||
- [ ] Set up connection parameters (timeout, retry)
|
||||
- [ ] Configure bulk operations settings
|
||||
- [ ] Add compression support
|
||||
- [ ] Implement backpressure handling
|
||||
|
||||
### Connection Management
|
||||
- [ ] Configure automatic reconnection
|
||||
- [ ] Set exponential backoff for retries
|
||||
- [ ] Add connection pooling
|
||||
- [ ] Configure keepalive settings
|
||||
- [ ] Handle DNS resolution failures
|
||||
|
||||
### Authentication Configuration (Priority 1 - CRITICAL)
|
||||
- [ ] Implement mTLS authentication support
|
||||
- [ ] Configure client certificate and key loading
|
||||
- [ ] Add CA certificate validation
|
||||
- [ ] Implement API key authentication
|
||||
- [ ] Add basic auth as fallback option
|
||||
- [ ] Create authentication mode selection logic
|
||||
- [ ] Handle authentication failures gracefully
|
||||
- [ ] Add certificate expiry checking
|
||||
- [ ] Implement secure credential storage
|
||||
- [ ] Document authentication setup process
|
||||
|
||||
## Phase 5: Reliability Features (Priority 2)
|
||||
|
||||
### Local Buffering
|
||||
- [ ] Configure disk queue for reliability
|
||||
- [ ] Set queue size limits
|
||||
- [ ] Configure memory queue settings
|
||||
- [ ] Add overflow handling
|
||||
- [ ] Set up automatic cleanup of old events
|
||||
|
||||
### Error Handling
|
||||
- [ ] Add retry logic for failed sends
|
||||
- [ ] Configure dead letter queue
|
||||
- [ ] Add circuit breaker pattern
|
||||
- [ ] Log transmission errors appropriately
|
||||
- [ ] Add metrics for monitoring failures
|
||||
|
||||
### Performance Optimization
|
||||
- [ ] Configure worker count
|
||||
- [ ] Set batch size for sending
|
||||
- [ ] Add compression level setting
|
||||
- [ ] Configure CPU and memory limits
|
||||
- [ ] Optimize for high-volume scenarios
|
||||
|
||||
## Phase 6: Optional Scripts (Priority 2)
|
||||
|
||||
### Operational Scripts
|
||||
- [ ] Implement `logs.sh` - Show Filebeat logs
|
||||
- [ ] Implement `destroy.sh` - Complete removal
|
||||
- [ ] Implement `ssh.sh` - Shell into Filebeat container
|
||||
- [ ] Create `test.sh` - Test connectivity to server
|
||||
- [ ] Add `metrics.sh` - Show Filebeat statistics
|
||||
|
||||
### Diagnostic Scripts
|
||||
- [ ] Create connectivity test script
|
||||
- [ ] Add configuration validation script
|
||||
- [ ] Create debug mode enabler
|
||||
- [ ] Add log sampling script
|
||||
- [ ] Create performance benchmark script
|
||||
|
||||
## Phase 7: Monitoring & Health (Priority 2)
|
||||
|
||||
### Health Checks
|
||||
- [ ] Configure Filebeat HTTP endpoint
|
||||
- [ ] Add Docker health check
|
||||
- [ ] Monitor queue status
|
||||
- [ ] Check connection to Logstash
|
||||
- [ ] Track dropped events
|
||||
|
||||
### Metrics Collection
|
||||
- [ ] Enable Filebeat monitoring
|
||||
- [ ] Export metrics endpoint
|
||||
- [ ] Track events sent/failed
|
||||
- [ ] Monitor resource usage
|
||||
- [ ] Add performance counters
|
||||
|
||||
### Status Reporting
|
||||
- [ ] Implement detailed status in status.sh
|
||||
- [ ] Show connection state
|
||||
- [ ] Display queue status
|
||||
- [ ] Report recent errors
|
||||
- [ ] Show throughput metrics
|
||||
|
||||
## Phase 8: Advanced Features (Priority 3)
|
||||
|
||||
### Processors
|
||||
- [ ] Add field renaming processor
|
||||
- [ ] Configure drop_event conditions
|
||||
- [ ] Add rate limiting processor
|
||||
- [ ] Include fingerprinting for deduplication
|
||||
- [ ] Add custom field enrichment
|
||||
|
||||
### Multiline Handling
|
||||
- [ ] Configure patterns for common languages
|
||||
- [ ] Java stack trace handling
|
||||
- [ ] Python traceback handling
|
||||
- [ ] Go panic handling
|
||||
- [ ] Custom pattern support via environment
|
||||
|
||||
### Field Management
|
||||
- [ ] Configure field inclusion/exclusion
|
||||
- [ ] Add custom fields via environment
|
||||
- [ ] Set up field type conversions
|
||||
- [ ] Add timestamp parsing
|
||||
- [ ] Configure field aliasing
|
||||
|
||||
## Phase 9: Testing (Priority 3)
|
||||
|
||||
### Unit Testing
|
||||
- [ ] Test configuration generation
|
||||
- [ ] Verify volume mounts
|
||||
- [ ] Test environment variable substitution
|
||||
- [ ] Validate filtering logic
|
||||
- [ ] Test error conditions
|
||||
|
||||
### Integration Testing
|
||||
- [ ] Test with logserver template
|
||||
- [ ] Verify Docker log collection
|
||||
- [ ] Test system log collection
|
||||
- [ ] Validate SSL connectivity
|
||||
- [ ] Test reconnection scenarios
|
||||
- [ ] Verify buffering during outages
|
||||
|
||||
### Load Testing
|
||||
- [ ] Test with high log volume
|
||||
- [ ] Measure resource usage
|
||||
- [ ] Test queue overflow handling
|
||||
- [ ] Verify rate limiting
|
||||
- [ ] Benchmark throughput
|
||||
|
||||
## Phase 10: Documentation (Priority 3)
|
||||
|
||||
### User Documentation
|
||||
- [ ] Create README.txt for dropshell
|
||||
- [ ] Document all configuration options
|
||||
- [ ] Add troubleshooting guide
|
||||
- [ ] Create quick start guide
|
||||
- [ ] Add FAQ section
|
||||
|
||||
### Configuration Examples
|
||||
- [ ] Minimal configuration example
|
||||
- [ ] High-volume configuration
|
||||
- [ ] Secure SSL configuration
|
||||
- [ ] Filtered configuration
|
||||
- [ ] Custom paths configuration
|
||||
|
||||
### Integration Guides
|
||||
- [ ] Integration with logserver
|
||||
- [ ] Docker Compose examples
|
||||
- [ ] Kubernetes DaemonSet example
|
||||
- [ ] Swarm mode configuration
|
||||
- [ ] Custom application integration
|
||||
|
||||
## Phase 11: Production Readiness (Priority 4)
|
||||
|
||||
### Security Hardening
|
||||
- [ ] Run as non-root user where possible
|
||||
- [ ] Minimize container capabilities
|
||||
- [ ] Add secrets management
|
||||
- [ ] Configure log sanitization
|
||||
- [ ] Add audit logging
|
||||
|
||||
### Updates & Maintenance
|
||||
- [ ] Add update notification
|
||||
- [ ] Create upgrade script
|
||||
- [ ] Add configuration migration
|
||||
- [ ] Document breaking changes
|
||||
- [ ] Create rollback procedure
|
||||
|
||||
### Compatibility
|
||||
- [ ] Test with different Filebeat versions
|
||||
- [ ] Verify Docker API compatibility
|
||||
- [ ] Test on different Linux distributions
|
||||
- [ ] Validate with various log formats
|
||||
- [ ] Ensure Logstash version compatibility
|
||||
|
||||
## Notes
|
||||
|
||||
### Design Principles
|
||||
1. **Minimal configuration**: Just needs LOGSERVER_HOST to work
|
||||
2. **Docker API access**: Use Docker API for driver-independent log collection
|
||||
3. **Automatic discovery**: Find all container logs without manual configuration
|
||||
4. **Reliability first**: Never lose logs, buffer locally if needed
|
||||
5. **Low overhead**: Minimal resource usage on host
|
||||
6. **Non-intrusive**: No changes to existing containers needed
|
||||
7. **Driver flexibility**: Allow containers to use any logging driver (especially `local`)
|
||||
|
||||
### Key Requirements
|
||||
- Must work with zero configuration beyond server address
|
||||
- Must use Docker API input, not file-based collection
|
||||
- Must support all Docker logging drivers (local, json-file, etc.)
|
||||
- Must handle Docker socket permissions properly
|
||||
- Must be resilient to network failures
|
||||
- Must not impact host performance significantly
|
||||
- Must preserve configuration on uninstall
|
||||
|
||||
### Testing Checklist
|
||||
- [ ] Validates with dropshell test-template
|
||||
- [ ] Connects to logserver successfully
|
||||
- [ ] Collects Docker logs automatically
|
||||
- [ ] Collects system logs properly
|
||||
- [ ] Handles server downtime gracefully
|
||||
- [ ] Reconnects automatically
|
||||
- [ ] Resource usage stays within limits
|
||||
- [ ] Uninstall preserves configuration
|
||||
@@ -1,7 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Define volume items for logclient container
|
||||
# These are used across backup, restore, create, and destroy operations
|
||||
|
||||
get_logclient_volumes() {
|
||||
echo "volume:datavolume:${DATA_VOLUME}" "volume:configvolume:${CONFIG_VOLUME}"
|
||||
}
|
||||
@@ -1,32 +0,0 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
||||
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
echo "Backing up ${CONTAINER_NAME} volumes..."
|
||||
|
||||
# Stop the container to ensure data consistency
|
||||
bash ./stop.sh || true
|
||||
|
||||
# Backup volumes
|
||||
BACKUP_DIR="${CONFIG_PATH}/backups/$(date +%Y%m%d_%H%M%S)"
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
# Export volumes
|
||||
for volume in $(get_logclient_volumes); do
|
||||
volume_name=$(echo $volume | cut -d: -f3)
|
||||
echo "Backing up volume: $volume_name"
|
||||
docker run --rm -v "$volume_name:/source:ro" -v "$BACKUP_DIR:/backup" alpine \
|
||||
tar -czf "/backup/${volume_name}.tar.gz" -C /source .
|
||||
done
|
||||
|
||||
# Backup configuration
|
||||
cp -r "${CONFIG_PATH}" "$BACKUP_DIR/config_backup"
|
||||
|
||||
echo "Backup completed to: $BACKUP_DIR"
|
||||
|
||||
# Restart the container
|
||||
bash ./start.sh
|
||||
|
||||
echo "Container restarted"
|
||||
@@ -1,16 +0,0 @@
|
||||
# Template identifier - MUST match the directory name
|
||||
TEMPLATE=logclient
|
||||
|
||||
# Requirements
|
||||
REQUIRES_HOST_ROOT=false # No root access on host needed
|
||||
REQUIRES_DOCKER=true # Docker is required
|
||||
REQUIRES_DOCKER_ROOT=false # Docker root privileges not specifically needed
|
||||
|
||||
# Docker image settings
|
||||
IMAGE_REGISTRY="docker.elastic.co"
|
||||
IMAGE_REPO="beats/filebeat"
|
||||
IMAGE_TAG="7.17.23"
|
||||
|
||||
# Volume definitions
|
||||
DATA_VOLUME=logclient_data
|
||||
CONFIG_VOLUME=logclient_config
|
||||
@@ -1,34 +1,16 @@
|
||||
# Service identification
|
||||
CONTAINER_NAME=logclient-filebeat
|
||||
# Log Client Configuration (Promtail)
|
||||
CONTAINER_NAME=logclient
|
||||
|
||||
# Server settings (REQUIRED by dropshell)
|
||||
SSH_USER="root"
|
||||
|
||||
# Docker image tag override (optional)
|
||||
IMAGE_TAG="7.17.23"
|
||||
|
||||
# REQUIRED: Log server connection
|
||||
LOGSERVER_HOST=
|
||||
LOGSERVER_PORT=5044
|
||||
LOGSERVER_PORT=3100
|
||||
|
||||
# REQUIRED: API Key Authentication
|
||||
API_KEY="" # Get from logserver admin using generate-api-key.sh
|
||||
# REQUIRED: Authentication (get from log server admin)
|
||||
LOKI_USER=logclient
|
||||
LOKI_PASSWORD=
|
||||
|
||||
# Performance tuning
|
||||
BULK_MAX_SIZE=2048 # Maximum batch size
|
||||
WORKER_THREADS=1 # Number of worker threads
|
||||
QUEUE_SIZE=4096 # Internal queue size
|
||||
MAX_BACKOFF=60s # Maximum retry backoff
|
||||
|
||||
# Filtering
|
||||
EXCLUDE_CONTAINERS="" # Comma-separated container names to exclude
|
||||
INCLUDE_CONTAINERS="" # Only include these containers (if set)
|
||||
EXCLUDE_LABELS="" # Exclude containers with these labels
|
||||
INCLUDE_LABELS="" # Only include containers with these labels
|
||||
|
||||
# Additional log paths
|
||||
CUSTOM_LOG_PATHS="" # Comma-separated additional paths to monitor
|
||||
|
||||
# Resource limits
|
||||
MAX_CPU=50 # Maximum CPU usage percentage
|
||||
MAX_MEMORY=200MB # Maximum memory usage
|
||||
# Optional: Set a custom hostname label (defaults to actual hostname)
|
||||
# HOSTNAME_LABEL=
|
||||
@@ -1,22 +0,0 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
||||
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
echo "WARNING: This will permanently delete all data for ${CONTAINER_NAME}"
|
||||
echo "This action cannot be undone!"
|
||||
echo ""
|
||||
|
||||
# Since scripts must be non-interactive, only proceed if explicitly called
|
||||
bash ./stop.sh || true
|
||||
_remove_container "$CONTAINER_NAME" || true
|
||||
|
||||
# Remove all volumes
|
||||
for volume in $(get_logclient_volumes); do
|
||||
volume_name=$(echo $volume | cut -d: -f3)
|
||||
echo "Removing volume: $volume_name"
|
||||
docker volume rm "$volume_name" 2>/dev/null || true
|
||||
done
|
||||
|
||||
echo "Service and all data destroyed"
|
||||
19
logclient/docker-compose.yml
Normal file
19
logclient/docker-compose.yml
Normal file
@@ -0,0 +1,19 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
promtail:
|
||||
image: grafana/promtail:latest
|
||||
container_name: ${CONTAINER_NAME}
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- /var/log:/var/log:ro
|
||||
- /var/lib/docker/containers:/var/lib/docker/containers:ro
|
||||
- ${CONFIG_PATH}/promtail.yaml:/etc/promtail/config.yml:ro
|
||||
- promtail_positions:/tmp/positions
|
||||
command: -config.file=/etc/promtail/config.yml
|
||||
environment:
|
||||
- HOSTNAME=${HOSTNAME_LABEL:-$(hostname)}
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
promtail_positions:
|
||||
@@ -2,52 +2,39 @@
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# Check required environment variables
|
||||
_check_required_env_vars "CONTAINER_NAME" "IMAGE_REGISTRY" "IMAGE_REPO" "IMAGE_TAG" "LOGSERVER_HOST" "LOGSERVER_PORT" "API_KEY"
|
||||
_check_required_env_vars "CONTAINER_NAME" "LOGSERVER_HOST" "LOGSERVER_PORT" "LOKI_PASSWORD"
|
||||
|
||||
# Validate API key
|
||||
if [ -z "$API_KEY" ]; then
|
||||
echo ""
|
||||
echo "ERROR: API_KEY is not configured"
|
||||
echo ""
|
||||
echo "To get an API key:"
|
||||
echo "1. On the logserver, run: ./generate-api-key.sh"
|
||||
echo "2. Enter this client's hostname when prompted"
|
||||
echo "3. Copy the generated API_KEY to this client's service.env"
|
||||
echo ""
|
||||
_die "Missing API_KEY configuration"
|
||||
fi
|
||||
|
||||
# Check Docker is available
|
||||
# Check Docker
|
||||
_check_docker_installed || _die "Docker test failed"
|
||||
|
||||
# Test connectivity to logserver
|
||||
echo "Testing connectivity to logserver at ${LOGSERVER_HOST}:${LOGSERVER_PORT}..."
|
||||
nc -zv "$LOGSERVER_HOST" "$LOGSERVER_PORT" 2>/dev/null || echo "WARNING: Cannot connect to logserver. Will retry when container starts."
|
||||
|
||||
# Pull the Docker image
|
||||
docker pull "$IMAGE_REGISTRY/$IMAGE_REPO:$IMAGE_TAG" || _die "Failed to pull Filebeat image"
|
||||
echo "Testing connectivity to log server at ${LOGSERVER_HOST}:${LOGSERVER_PORT}..."
|
||||
nc -zv "$LOGSERVER_HOST" "$LOGSERVER_PORT" 2>/dev/null || echo "WARNING: Cannot connect to log server. Will retry when container starts."
|
||||
|
||||
# Stop any existing container
|
||||
bash ./stop.sh || true
|
||||
bash ./stop.sh 2>/dev/null || true
|
||||
|
||||
# Remove old container
|
||||
_remove_container "$CONTAINER_NAME" || true
|
||||
|
||||
# Generate Filebeat configuration (with actual hostname)
|
||||
echo "Generating Filebeat configuration..."
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# Generate configuration
|
||||
echo "Generating configuration..."
|
||||
export HOSTNAME=$(hostname)
|
||||
bash "$SCRIPT_DIR/scripts/generate-config.sh" || _die "Failed to generate configuration"
|
||||
|
||||
# Create volumes using common function
|
||||
source "$SCRIPT_DIR/_volumes.sh"
|
||||
echo "Creating volumes..."
|
||||
create_items $(get_logclient_volumes)
|
||||
# Start the client
|
||||
echo "Starting Log Client..."
|
||||
docker compose up -d || _die "Failed to start"
|
||||
|
||||
# Start the new container
|
||||
bash ./start.sh || _die "Failed to start Filebeat"
|
||||
|
||||
echo "Installation of ${CONTAINER_NAME} complete"
|
||||
echo "Collecting logs from Docker API and shipping to ${LOGSERVER_HOST}:${LOGSERVER_PORT}"
|
||||
echo "Using API key authentication"
|
||||
echo ""
|
||||
echo "========================================="
|
||||
echo "Log Client Installed!"
|
||||
echo "========================================="
|
||||
echo ""
|
||||
echo "Shipping logs to: ${LOGSERVER_HOST}:${LOGSERVER_PORT}"
|
||||
echo "Using authentication: ${LOKI_USER:-logclient}"
|
||||
echo "Hostname label: $(hostname)"
|
||||
echo ""
|
||||
echo "Collecting:"
|
||||
echo " - All Docker container logs"
|
||||
echo " - System logs (/var/log)"
|
||||
echo ""
|
||||
echo "View logs at: http://${LOGSERVER_HOST}:3000"
|
||||
echo "========================================="
|
||||
5
logclient/logs.sh
Executable file
5
logclient/logs.sh
Executable file
@@ -0,0 +1,5 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
docker logs "${CONTAINER_NAME}" --tail 50
|
||||
@@ -1,47 +0,0 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
||||
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
if [ -z "$1" ]; then
|
||||
echo "Usage: $0 <backup_directory>"
|
||||
echo "Available backups:"
|
||||
ls -la "${CONFIG_PATH}/backups/" 2>/dev/null || echo "No backups found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
BACKUP_DIR="$1"
|
||||
|
||||
if [ ! -d "$BACKUP_DIR" ]; then
|
||||
_die "Backup directory not found: $BACKUP_DIR"
|
||||
fi
|
||||
|
||||
echo "Restoring from backup: $BACKUP_DIR"
|
||||
|
||||
# Stop the container
|
||||
bash ./stop.sh || true
|
||||
|
||||
# Restore volumes
|
||||
for volume in $(get_logclient_volumes); do
|
||||
volume_name=$(echo $volume | cut -d: -f3)
|
||||
backup_file="$BACKUP_DIR/${volume_name}.tar.gz"
|
||||
|
||||
if [ -f "$backup_file" ]; then
|
||||
echo "Restoring volume: $volume_name"
|
||||
docker run --rm -v "$volume_name:/target" -v "$BACKUP_DIR:/backup:ro" alpine \
|
||||
sh -c "rm -rf /target/* && tar -xzf /backup/${volume_name}.tar.gz -C /target"
|
||||
fi
|
||||
done
|
||||
|
||||
# Restore configuration if exists
|
||||
if [ -d "$BACKUP_DIR/config_backup" ]; then
|
||||
cp -r "$BACKUP_DIR/config_backup/"* "${CONFIG_PATH}/"
|
||||
fi
|
||||
|
||||
echo "Restore completed"
|
||||
|
||||
# Start the container
|
||||
bash ./start.sh
|
||||
|
||||
echo "Container restarted"
|
||||
@@ -1,163 +1,111 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Generate Filebeat configuration from template
|
||||
# This script creates a filebeat.yml configuration file with proper authentication
|
||||
# Generate Promtail configuration for Log Client
|
||||
|
||||
# Check required variables
|
||||
if [ -z "$LOGSERVER_HOST" ] || [ -z "$LOGSERVER_PORT" ]; then
|
||||
if [ -z "$LOGSERVER_HOST" ] || [ -z "$LOGSERVER_PORT" ] || [ -z "$LOKI_PASSWORD" ]; then
|
||||
echo "ERROR: Required environment variables not set"
|
||||
echo " LOGSERVER_HOST: ${LOGSERVER_HOST:-NOT SET}"
|
||||
echo " LOGSERVER_PORT: ${LOGSERVER_PORT:-NOT SET}"
|
||||
echo " LOKI_PASSWORD: ${LOKI_PASSWORD:-NOT SET}"
|
||||
echo ""
|
||||
echo "Please set these in config/service.env before running install"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Determine config directory - use CONFIG_PATH from dropshell or fallback
|
||||
if [ -n "$CONFIG_PATH" ]; then
|
||||
CONFIG_DIR="$CONFIG_PATH"
|
||||
elif [ -d "./config" ]; then
|
||||
CONFIG_DIR="./config"
|
||||
else
|
||||
CONFIG_DIR="."
|
||||
fi
|
||||
# Get actual hostname
|
||||
ACTUAL_HOSTNAME=${HOSTNAME_LABEL:-${HOSTNAME:-$(hostname 2>/dev/null || echo "unknown")}}
|
||||
|
||||
# Ensure config directory exists
|
||||
# Determine config directory
|
||||
CONFIG_DIR="${CONFIG_PATH:-./config}"
|
||||
mkdir -p "$CONFIG_DIR"
|
||||
|
||||
# Set defaults for variables if not set
|
||||
BULK_MAX_SIZE=${BULK_MAX_SIZE:-2048}
|
||||
WORKER_THREADS=${WORKER_THREADS:-1}
|
||||
QUEUE_SIZE=${QUEUE_SIZE:-4096}
|
||||
MAX_BACKOFF=${MAX_BACKOFF:-60s}
|
||||
# Generate promtail.yaml configuration
|
||||
cat > "$CONFIG_DIR/promtail.yaml" << EOF
|
||||
server:
|
||||
http_listen_port: 9080
|
||||
grpc_listen_port: 0
|
||||
|
||||
# Get actual hostname from the host system
|
||||
ACTUAL_HOSTNAME=${HOSTNAME:-$(hostname 2>/dev/null || echo "unknown")}
|
||||
positions:
|
||||
filename: /tmp/positions/positions.yaml
|
||||
|
||||
# Generate filebeat.yml configuration with variable substitution
|
||||
(
|
||||
cat << 'TEMPLATE_EOF'
|
||||
# Filebeat Configuration for LogClient
|
||||
# Generated by generate-config.sh
|
||||
clients:
|
||||
- url: http://${LOKI_USER:-logclient}:${LOKI_PASSWORD}@${LOGSERVER_HOST}:${LOGSERVER_PORT}/loki/api/v1/push
|
||||
# Authentication via URL (HTTP Basic Auth)
|
||||
|
||||
# ======================== Docker Input Configuration =========================
|
||||
# Use Docker input to collect logs via Docker API
|
||||
filebeat.inputs:
|
||||
- type: docker
|
||||
enabled: true
|
||||
# Collect from all containers
|
||||
containers.ids:
|
||||
- '*'
|
||||
# Collect both stdout and stderr
|
||||
containers.stream: all
|
||||
# Combine partial log lines
|
||||
combine_partial: true
|
||||
# Add Docker metadata
|
||||
processors:
|
||||
- add_docker_metadata:
|
||||
host: "unix:///var/run/docker.sock"
|
||||
scrape_configs:
|
||||
# Docker container logs via Docker API
|
||||
- job_name: docker
|
||||
docker_sd_configs:
|
||||
- host: unix:///var/run/docker.sock
|
||||
refresh_interval: 5s
|
||||
relabel_configs:
|
||||
- source_labels: ['__meta_docker_container_name']
|
||||
regex: '/(.*)'
|
||||
target_label: 'container_name'
|
||||
- source_labels: ['__meta_docker_container_id']
|
||||
target_label: 'container_id'
|
||||
- source_labels: ['__meta_docker_container_image']
|
||||
target_label: 'image'
|
||||
- target_label: 'hostname'
|
||||
replacement: '${ACTUAL_HOSTNAME}'
|
||||
- target_label: 'job'
|
||||
replacement: 'docker'
|
||||
|
||||
# ======================== System Logs Configuration ==========================
|
||||
- type: log
|
||||
enabled: true
|
||||
paths:
|
||||
- /var/log/syslog
|
||||
- /var/log/messages
|
||||
exclude_lines: ['^#']
|
||||
fields:
|
||||
log_type: syslog
|
||||
# System logs
|
||||
- job_name: syslog
|
||||
static_configs:
|
||||
- targets:
|
||||
- localhost
|
||||
labels:
|
||||
job: syslog
|
||||
hostname: ${ACTUAL_HOSTNAME}
|
||||
__path__: /var/log/syslog
|
||||
|
||||
- type: log
|
||||
enabled: true
|
||||
paths:
|
||||
- /var/log/auth.log
|
||||
- /var/log/secure
|
||||
exclude_lines: ['^#']
|
||||
fields:
|
||||
log_type: auth
|
||||
- job_name: messages
|
||||
static_configs:
|
||||
- targets:
|
||||
- localhost
|
||||
labels:
|
||||
job: messages
|
||||
hostname: ${ACTUAL_HOSTNAME}
|
||||
__path__: /var/log/messages
|
||||
|
||||
# ======================== Processors Configuration ===========================
|
||||
processors:
|
||||
- add_host_metadata:
|
||||
when.not.contains:
|
||||
tags: forwarded
|
||||
# Override hostname with actual host's hostname
|
||||
- add_fields:
|
||||
target: agent
|
||||
fields:
|
||||
hostname: __ACTUAL_HOSTNAME__
|
||||
- add_fields:
|
||||
target: host
|
||||
fields:
|
||||
name: __ACTUAL_HOSTNAME__
|
||||
- job_name: auth
|
||||
static_configs:
|
||||
- targets:
|
||||
- localhost
|
||||
labels:
|
||||
job: auth
|
||||
hostname: ${ACTUAL_HOSTNAME}
|
||||
__path__: /var/log/auth.log
|
||||
|
||||
# ======================== Output Configuration ===============================
|
||||
output.logstash:
|
||||
hosts: ["__LOGSERVER_HOST__:__LOGSERVER_PORT__"]
|
||||
# SSL/TLS configuration
|
||||
ssl.enabled: false # Set to true when using TLS
|
||||
ssl.verification_mode: none # Set to full in production with proper certs
|
||||
# Docker container JSON logs (backup method)
|
||||
- job_name: containers
|
||||
static_configs:
|
||||
- targets:
|
||||
- localhost
|
||||
labels:
|
||||
job: containers
|
||||
hostname: ${ACTUAL_HOSTNAME}
|
||||
__path__: /var/lib/docker/containers/*/*-json.log
|
||||
pipeline_stages:
|
||||
- json:
|
||||
expressions:
|
||||
output: log
|
||||
stream: stream
|
||||
time: time
|
||||
- timestamp:
|
||||
source: time
|
||||
format: RFC3339Nano
|
||||
- labels:
|
||||
stream:
|
||||
- output:
|
||||
source: output
|
||||
EOF
|
||||
|
||||
# Performance settings
|
||||
bulk_max_size: __BULK_MAX_SIZE__
|
||||
worker: __WORKER_THREADS__ # Must be >= 1
|
||||
compression_level: 3
|
||||
|
||||
# Retry configuration
|
||||
max_retries: 3
|
||||
backoff.init: 1s
|
||||
backoff.max: __MAX_BACKOFF__
|
||||
|
||||
# ======================== Global Fields =======================================
|
||||
# Add API key as a field to all events
|
||||
fields:
|
||||
api_key: "__API_KEY__"
|
||||
fields_under_root: false
|
||||
|
||||
# ======================== Queue Configuration ================================
|
||||
queue.mem:
|
||||
events: __QUEUE_SIZE__
|
||||
flush.min_events: 512
|
||||
flush.timeout: 5s
|
||||
|
||||
# ======================== Logging Configuration ==============================
|
||||
logging.level: info
|
||||
logging.to_files: true
|
||||
logging.files:
|
||||
path: /usr/share/filebeat/data/logs
|
||||
name: filebeat
|
||||
keepfiles: 3
|
||||
permissions: 0600
|
||||
|
||||
# ======================== Monitoring ==========================================
|
||||
monitoring.enabled: false
|
||||
http.enabled: true
|
||||
http.host: 0.0.0.0
|
||||
http.port: 5066
|
||||
|
||||
# ======================== File Permissions ====================================
|
||||
# Set strict permissions (disabled for Docker)
|
||||
# filebeat.config.modules.path: ${path.config}/modules.d/*.yml
|
||||
TEMPLATE_EOF
|
||||
) | sed -e "s|__LOGSERVER_HOST__|${LOGSERVER_HOST}|g" \
|
||||
-e "s|__LOGSERVER_PORT__|${LOGSERVER_PORT}|g" \
|
||||
-e "s|__API_KEY__|${API_KEY}|g" \
|
||||
-e "s|__BULK_MAX_SIZE__|${BULK_MAX_SIZE}|g" \
|
||||
-e "s|__WORKER_THREADS__|${WORKER_THREADS}|g" \
|
||||
-e "s|__QUEUE_SIZE__|${QUEUE_SIZE}|g" \
|
||||
-e "s|__MAX_BACKOFF__|${MAX_BACKOFF}|g" \
|
||||
-e "s|__ACTUAL_HOSTNAME__|${ACTUAL_HOSTNAME}|g" > "$CONFIG_DIR/filebeat.yml"
|
||||
|
||||
echo "Filebeat configuration generated at: $CONFIG_DIR/filebeat.yml"
|
||||
echo "Promtail configuration generated at: $CONFIG_DIR/promtail.yaml"
|
||||
echo "Configuration:"
|
||||
echo " LOGSERVER_HOST: ${LOGSERVER_HOST}"
|
||||
echo " LOGSERVER_PORT: ${LOGSERVER_PORT}"
|
||||
echo " API_KEY: ${API_KEY:+[SET]}"
|
||||
echo " WORKER_THREADS: ${WORKER_THREADS}"
|
||||
|
||||
# Additional warning if API_KEY is not set
|
||||
if [ -z "$API_KEY" ]; then
|
||||
echo ""
|
||||
echo "WARNING: API_KEY is not set - logs may be rejected by the server"
|
||||
echo "Get an API key from the LogServer admin using generate-api-key.sh"
|
||||
fi
|
||||
echo " HOSTNAME: ${ACTUAL_HOSTNAME}"
|
||||
@@ -1,36 +1,6 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME" "IMAGE_REGISTRY" "IMAGE_REPO" "IMAGE_TAG"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
# Check that config file exists
|
||||
if [ ! -f "${CONFIG_PATH}/filebeat.yml" ]; then
|
||||
_die "filebeat.yml not found in ${CONFIG_PATH}/filebeat.yml"
|
||||
fi
|
||||
|
||||
# Create Docker command
|
||||
DOCKER_RUN_CMD="docker run -d \
|
||||
--name $CONTAINER_NAME \
|
||||
--restart unless-stopped \
|
||||
--user root \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock:ro \
|
||||
-v /var/log:/var/log:ro \
|
||||
-v ${CONFIG_PATH}:/usr/share/filebeat/config:ro \
|
||||
-v ${DATA_VOLUME}:/usr/share/filebeat/data \
|
||||
-e LOGSERVER_HOST=${LOGSERVER_HOST} \
|
||||
-e LOGSERVER_PORT=${LOGSERVER_PORT} \
|
||||
-e API_KEY=${API_KEY} \
|
||||
-e HOSTNAME=$(hostname) \
|
||||
$IMAGE_REGISTRY/$IMAGE_REPO:$IMAGE_TAG \
|
||||
filebeat -e -strict.perms=false \
|
||||
-c /usr/share/filebeat/config/filebeat.yml"
|
||||
|
||||
if ! _create_and_start_container "$DOCKER_RUN_CMD" "$CONTAINER_NAME"; then
|
||||
_die "Failed to start container ${CONTAINER_NAME}"
|
||||
fi
|
||||
|
||||
# Check if the container is running
|
||||
if ! _is_container_running "$CONTAINER_NAME"; then
|
||||
_die "Container ${CONTAINER_NAME} is not running"
|
||||
fi
|
||||
|
||||
echo "Container ${CONTAINER_NAME} started"
|
||||
docker compose up -d || _die "Failed to start"
|
||||
echo "Log Client started"
|
||||
@@ -2,30 +2,10 @@
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
# Check if container exists
|
||||
if ! docker ps -a --format "{{.Names}}" | grep -q "^${CONTAINER_NAME}$"; then
|
||||
echo "Unknown"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Check container state
|
||||
STATE=$(docker inspect -f '{{.State.Status}}' "$CONTAINER_NAME" 2>/dev/null)
|
||||
case "$STATE" in
|
||||
running)
|
||||
# Additional check: verify connection to logserver
|
||||
if docker logs "$CONTAINER_NAME" 2>&1 | tail -20 | grep -q "ERROR"; then
|
||||
echo "Error"
|
||||
else
|
||||
echo "Running"
|
||||
fi
|
||||
;;
|
||||
exited|stopped)
|
||||
echo "Stopped"
|
||||
;;
|
||||
restarting|paused)
|
||||
echo "Error"
|
||||
;;
|
||||
*)
|
||||
echo "Unknown"
|
||||
;;
|
||||
esac
|
||||
if docker ps | grep -q "${CONTAINER_NAME}"; then
|
||||
echo "Running"
|
||||
echo " Shipping logs to: ${LOGSERVER_HOST}:${LOGSERVER_PORT}"
|
||||
docker ps --format "table {{.Names}}\t{{.Status}}" | grep "${CONTAINER_NAME}"
|
||||
else
|
||||
echo "Stopped"
|
||||
fi
|
||||
@@ -2,6 +2,5 @@
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
docker stop "$CONTAINER_NAME" 2>/dev/null || true
|
||||
|
||||
echo "Filebeat stopped"
|
||||
docker compose stop || _die "Failed to stop"
|
||||
echo "Log Client stopped"
|
||||
@@ -2,16 +2,5 @@
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
# Stop the container
|
||||
bash ./stop.sh || _die "Failed to stop container"
|
||||
|
||||
# Remove the container
|
||||
_remove_container "$CONTAINER_NAME" || _die "Failed to remove container"
|
||||
|
||||
# CRITICAL: Never remove data volumes in uninstall.sh!
|
||||
# Data volumes must be preserved for potential reinstallation
|
||||
# Configuration and certificates must be preserved
|
||||
|
||||
echo "Uninstallation of ${CONTAINER_NAME} complete"
|
||||
echo "Note: Configuration and certificates have been preserved."
|
||||
echo "To remove all data, use destroy.sh"
|
||||
docker compose down || true
|
||||
echo "Log Client uninstalled"
|
||||
@@ -1,279 +0,0 @@
|
||||
# Dropshell LogServer Template
|
||||
|
||||
A comprehensive centralized logging solution using the ELK Stack (Elasticsearch, Logstash, Kibana) for receiving, processing, and visualizing logs from multiple hosts.
|
||||
|
||||
## Overview
|
||||
|
||||
This template deploys a full-featured ELK stack that:
|
||||
- Receives logs from multiple sources via Beats protocol
|
||||
- Stores and indexes logs in Elasticsearch
|
||||
- Provides powerful search and visualization through Kibana
|
||||
- Supports automatic log parsing and enrichment
|
||||
- Handles Docker container logs and system logs from clients
|
||||
|
||||
## Architecture
|
||||
|
||||
### Components
|
||||
|
||||
1. **Elasticsearch** (7.17.x)
|
||||
- Distributed search and analytics engine
|
||||
- Stores and indexes all log data
|
||||
- Provides fast full-text search capabilities
|
||||
- Single-node configuration for simplicity (can be scaled)
|
||||
|
||||
2. **Logstash** (7.17.x)
|
||||
- Log processing pipeline
|
||||
- Receives logs from Filebeat clients
|
||||
- Parses and enriches log data
|
||||
- Routes logs to appropriate Elasticsearch indices
|
||||
|
||||
3. **Kibana** (7.17.x)
|
||||
- Web UI for log exploration and visualization
|
||||
- Create dashboards and alerts
|
||||
- Real-time log streaming
|
||||
- Advanced search queries
|
||||
|
||||
## Features
|
||||
|
||||
### Minimum Configuration Design
|
||||
- Auto-discovery of log formats
|
||||
- Pre-configured dashboards for common services
|
||||
- Automatic index lifecycle management
|
||||
- Built-in parsing for Docker and syslog formats
|
||||
- Zero-configuration client connectivity
|
||||
|
||||
### Log Processing
|
||||
- Automatic timestamp extraction
|
||||
- Docker metadata enrichment (container name, image, labels)
|
||||
- Syslog parsing with severity levels
|
||||
- JSON log support
|
||||
- Multi-line log handling (stacktraces, etc.)
|
||||
- Grok pattern matching for common formats
|
||||
|
||||
### Security & Performance
|
||||
- **Mutual TLS (mTLS)** authentication for client connections
|
||||
- **API key authentication** as an alternative to certificates
|
||||
- **Per-client authentication** with unique keys/certificates
|
||||
- **SSL/TLS encryption** for all client connections
|
||||
- **Basic authentication** for Kibana web access
|
||||
- **IP whitelisting** for additional security
|
||||
- Index lifecycle management for storage optimization
|
||||
- Automatic old log cleanup
|
||||
- Resource limits to prevent overconsumption
|
||||
|
||||
## Port Configuration
|
||||
|
||||
- **5601**: Kibana Web UI (HTTP/HTTPS with authentication)
|
||||
- **9200**: Elasticsearch REST API (HTTP) - internal only
|
||||
- **5044**: Logstash Beats input (TCP/TLS) - authenticated client connections
|
||||
- **514**: Syslog input (UDP/TCP) - optional, unauthenticated
|
||||
- **24224**: Fluentd forward input - optional Docker logging driver
|
||||
|
||||
## Storage Requirements
|
||||
|
||||
- **Minimum**: 10GB for basic operation
|
||||
- **Recommended**: 50GB+ depending on log volume
|
||||
- **Log Retention**: Default 30 days (configurable)
|
||||
|
||||
## Client Authentication
|
||||
|
||||
### Authentication Methods
|
||||
|
||||
1. **Mutual TLS (mTLS) - Recommended**
|
||||
- Each client gets a unique certificate signed by the server's CA
|
||||
- Strongest security with mutual authentication
|
||||
- Automatic certificate validation
|
||||
|
||||
2. **API Keys**
|
||||
- Each client gets a unique API key
|
||||
- Simpler to manage than certificates
|
||||
- Good for environments where certificate management is difficult
|
||||
|
||||
3. **Basic Auth (Not Recommended)**
|
||||
- Shared username/password
|
||||
- Least secure, only for testing
|
||||
|
||||
### Client Configuration
|
||||
|
||||
Clients using the `logclient` template will:
|
||||
1. Authenticate using provided credentials (cert/key or API key)
|
||||
2. Establish encrypted TLS connection
|
||||
3. Ship all Docker container logs
|
||||
4. Ship system logs (syslog, auth, kernel)
|
||||
5. Maintain connection with automatic reconnection
|
||||
6. Buffer logs locally during network outages
|
||||
|
||||
## Dashboard Features
|
||||
|
||||
### Pre-configured Dashboards
|
||||
- **System Overview**: Overall health and log volume metrics
|
||||
- **Docker Containers**: Container-specific logs and metrics
|
||||
- **Error Analysis**: Aggregated error logs from all sources
|
||||
- **Security Events**: Authentication and access logs
|
||||
- **Application Logs**: Parsed application-specific logs
|
||||
|
||||
### Search Capabilities
|
||||
- Full-text search across all logs
|
||||
- Filter by time range, host, container, severity
|
||||
- Save and share search queries
|
||||
- Export search results
|
||||
|
||||
## Resource Requirements
|
||||
|
||||
### Minimum
|
||||
- CPU: 2 cores
|
||||
- RAM: 4GB
|
||||
- Storage: 10GB
|
||||
|
||||
### Recommended
|
||||
- CPU: 4+ cores
|
||||
- RAM: 8GB+
|
||||
- Storage: 50GB+ SSD
|
||||
|
||||
## Configuration Options
|
||||
|
||||
### Environment Variables (service.env)
|
||||
|
||||
```bash
|
||||
# Elasticsearch settings
|
||||
ES_HEAP_SIZE=2g
|
||||
ES_MAX_MAP_COUNT=262144
|
||||
|
||||
# Logstash settings
|
||||
LS_HEAP_SIZE=1g
|
||||
LS_PIPELINE_WORKERS=2
|
||||
|
||||
# Kibana settings
|
||||
KIBANA_PASSWORD=changeme
|
||||
KIBANA_BASE_PATH=/
|
||||
|
||||
# Log retention
|
||||
LOG_RETENTION_DAYS=30
|
||||
LOG_MAX_SIZE_GB=50
|
||||
|
||||
# Authentication Mode
|
||||
AUTH_MODE=mtls # Options: mtls, apikey, basic
|
||||
ENABLE_TLS=true
|
||||
|
||||
# mTLS Settings (if AUTH_MODE=mtls)
|
||||
CA_CERT_PATH=/certs/ca.crt
|
||||
SERVER_CERT_PATH=/certs/server.crt
|
||||
SERVER_KEY_PATH=/certs/server.key
|
||||
CLIENT_CERT_REQUIRED=true
|
||||
|
||||
# API Key Settings (if AUTH_MODE=apikey)
|
||||
API_KEYS_PATH=/config/api-keys.yml
|
||||
|
||||
# Network Security
|
||||
ALLOWED_IPS="" # Comma-separated list, empty = all
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Installation
|
||||
```bash
|
||||
dropshell install logserver
|
||||
```
|
||||
|
||||
### Generate Client Credentials
|
||||
|
||||
#### For mTLS Authentication:
|
||||
```bash
|
||||
# Generate client certificate for a new host
|
||||
dropshell exec logserver /scripts/generate-client-cert.sh hostname
|
||||
# This creates hostname.crt and hostname.key files
|
||||
```
|
||||
|
||||
#### For API Key Authentication:
|
||||
```bash
|
||||
# Generate API key for a new client
|
||||
dropshell exec logserver /scripts/generate-api-key.sh hostname
|
||||
# Returns an API key to configure in the client
|
||||
```
|
||||
|
||||
### Access Kibana
|
||||
Navigate to `https://<server-ip>:5601` in your browser.
|
||||
|
||||
Default credentials:
|
||||
- Username: `elastic`
|
||||
- Password: `changeme` (change in service.env)
|
||||
|
||||
### View Logs
|
||||
```bash
|
||||
dropshell logs logserver
|
||||
```
|
||||
|
||||
### Backup
|
||||
```bash
|
||||
dropshell backup logserver
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Elasticsearch failing to start**
|
||||
- Check vm.max_map_count: `sysctl vm.max_map_count` (should be 262144+)
|
||||
- Verify sufficient memory available
|
||||
|
||||
2. **No logs appearing in Kibana**
|
||||
- Check Logstash is receiving data: port 5044 should be open
|
||||
- Verify client connectivity
|
||||
- Check index patterns in Kibana
|
||||
|
||||
3. **High memory usage**
|
||||
- Adjust heap sizes in service.env
|
||||
- Configure index lifecycle management
|
||||
- Reduce retention period
|
||||
|
||||
## Integration
|
||||
|
||||
This template is designed to work seamlessly with the `logclient` template. Simply:
|
||||
1. Deploy this logserver
|
||||
2. Deploy logclient on each host you want to monitor
|
||||
3. Configure logclient with the logserver address
|
||||
4. Logs will automatically start flowing
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Authentication Setup**
|
||||
- Use mTLS for production environments
|
||||
- Generate unique credentials for each client
|
||||
- Rotate certificates/keys regularly
|
||||
- Store credentials securely
|
||||
|
||||
2. **Network Security**
|
||||
- Always use TLS encryption for client connections
|
||||
- Configure IP whitelisting when possible
|
||||
- Use firewall rules to restrict access
|
||||
- Consider VPN or private networks
|
||||
|
||||
3. **Access Control**
|
||||
- Change default Kibana password immediately
|
||||
- Create read-only users for viewing logs
|
||||
- Implement role-based access control (RBAC)
|
||||
- Audit access logs regularly
|
||||
|
||||
4. **Data Protection**
|
||||
- Regular backups of Elasticsearch indices
|
||||
- Encrypt data at rest (optional)
|
||||
- Monitor disk usage to prevent data loss
|
||||
- Implement log retention policies
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Daily Tasks
|
||||
- Monitor disk usage
|
||||
- Check for failed log shipments
|
||||
- Review error dashboards
|
||||
|
||||
### Weekly Tasks
|
||||
- Verify all clients are reporting
|
||||
- Check index health
|
||||
- Review and optimize slow queries
|
||||
|
||||
### Monthly Tasks
|
||||
- Update ELK stack components
|
||||
- Archive old indices
|
||||
- Review retention policies
|
||||
- Performance tuning based on usage patterns
|
||||
@@ -1,50 +1,73 @@
|
||||
# LogServer
|
||||
# Log Server (Loki + Grafana)
|
||||
|
||||
Centralized logging with ELK Stack (Elasticsearch, Logstash, Kibana).
|
||||
Central log server using Grafana + Loki. MUCH simpler than ELK/Kibana!
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. **System Setup**
|
||||
```bash
|
||||
sudo sysctl -w vm.max_map_count=262144
|
||||
```
|
||||
|
||||
2. **Configure**
|
||||
Edit `config/service.env`:
|
||||
- Set `SERVER_PUBLICBASEURL` to your actual server URL
|
||||
- Set `KIBANA_USERNAME` to your preferred username
|
||||
- Change `KIBANA_USER_PASSWORD` from default
|
||||
|
||||
3. **Install**
|
||||
1. **Install**
|
||||
```bash
|
||||
dropshell install logserver
|
||||
```
|
||||
|
||||
4. **Generate Client Keys**
|
||||
2. **Access Web UI**
|
||||
- URL: `http://<server-ip>:3000`
|
||||
- Login: `admin` / `changeme` (set in service.env)
|
||||
|
||||
3. **View Logs**
|
||||
- Click **Dashboards** (4 squares icon)
|
||||
- Click **"Central Logs"**
|
||||
- All logs from all servers appear!
|
||||
|
||||
## For Client Servers
|
||||
|
||||
On each server that should send logs here:
|
||||
|
||||
1. Edit `logclient/config/service.env`:
|
||||
```bash
|
||||
./generate-api-key.sh
|
||||
# Enter hostname when prompted
|
||||
# Save the API key for client configuration
|
||||
LOGSERVER_HOST=<this-server-ip>
|
||||
LOGSERVER_PORT=3100
|
||||
LOKI_USER=logclient # Default username
|
||||
LOKI_PASSWORD=<your-password> # From server's service.env
|
||||
```
|
||||
|
||||
5. **Setup Kibana** (first time only)
|
||||
2. Install the client:
|
||||
```bash
|
||||
./setup-kibana.sh
|
||||
dropshell install logclient
|
||||
```
|
||||
|
||||
6. **Access Kibana**
|
||||
- URL: `http://<server-ip>:5601`
|
||||
- Username: Set in `service.env` (KIBANA_USERNAME, default: `admin`)
|
||||
- Password: Set in `service.env` (KIBANA_USER_PASSWORD)
|
||||
- Click "Discover" → View your logs!
|
||||
## Features
|
||||
- **Super simple interface** - Just one dashboard
|
||||
- **Lightweight** - Uses 10x less resources than ELK
|
||||
- **Real-time** - Logs appear instantly
|
||||
- **Multi-server** - See all servers in one place
|
||||
- **Automatic labels** - Filter by hostname, container, etc.
|
||||
|
||||
## Security
|
||||
- Loki endpoint requires authentication (HTTP Basic Auth)
|
||||
- Default: `logclient` / `changeme`
|
||||
- **IMPORTANT**: Change `LOKI_PASSWORD` in service.env!
|
||||
|
||||
## Ports
|
||||
- `5601` - Kibana Web UI
|
||||
- `5044` - Log ingestion (Filebeat)
|
||||
- `3000` - Grafana Web UI
|
||||
- `3100` - Loki API (authenticated)
|
||||
|
||||
## Files
|
||||
- `config/service.env` - Configuration
|
||||
- `config/api-keys.yml` - Client API keys
|
||||
- `generate-api-key.sh` - Add new clients
|
||||
## How to Use
|
||||
|
||||
See [DOCUMENTATION.md](DOCUMENTATION.md) for full details.
|
||||
### Filter by Server
|
||||
- Use the "Server" dropdown at the top
|
||||
- Or click any `hostname` label
|
||||
|
||||
### Filter by Container
|
||||
- Click any `container_name` label
|
||||
|
||||
### Search Text
|
||||
- Use the search box in each panel
|
||||
|
||||
### Time Range
|
||||
- Top-right corner - adjust as needed
|
||||
|
||||
## Why This Instead of ELK?
|
||||
- **10x simpler** - One clean dashboard vs Kibana complexity
|
||||
- **10x smaller** - ~200MB RAM vs 2-3GB for ELK
|
||||
- **Zero configuration** - Just works
|
||||
- **Fast setup** - No index patterns, mappings, or complexity
|
||||
@@ -1,100 +0,0 @@
|
||||
# LogServer Quick Setup Guide
|
||||
|
||||
## Prerequisites
|
||||
- Docker and Docker Compose installed
|
||||
- 4GB+ RAM, 10GB+ disk space
|
||||
- Port 5601 (Kibana) and 5044 (Logstash) available
|
||||
|
||||
## Initial Setup
|
||||
|
||||
### 1. System Configuration
|
||||
```bash
|
||||
# Required for Elasticsearch
|
||||
sudo sysctl -w vm.max_map_count=262144
|
||||
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
|
||||
```
|
||||
|
||||
### 2. Configure Server
|
||||
Edit `config/service.env`:
|
||||
```bash
|
||||
# Change default password
|
||||
KIBANA_PASSWORD=your-secure-password
|
||||
```
|
||||
|
||||
### 3. Install
|
||||
```bash
|
||||
dropshell install logserver
|
||||
```
|
||||
|
||||
## Generate Client API Keys
|
||||
|
||||
Run the interactive key generator:
|
||||
```bash
|
||||
./generate-api-key.sh
|
||||
```
|
||||
|
||||
Follow the prompts:
|
||||
1. Enter hostname for each client
|
||||
2. Script generates secure API key
|
||||
3. Shows configuration to copy to client
|
||||
4. Repeat for additional clients
|
||||
|
||||
## Access Kibana
|
||||
|
||||
1. Open browser: `http://your-server-ip:5601`
|
||||
2. Login: `elastic` / `your-secure-password`
|
||||
3. Create index pattern: `filebeat-*`
|
||||
4. View logs in Discover tab
|
||||
|
||||
## Add Log Clients
|
||||
|
||||
On each client machine:
|
||||
```bash
|
||||
# Get API key from server admin (they run ./generate-api-key.sh)
|
||||
|
||||
# Edit logclient/config/service.env:
|
||||
LOGSERVER_HOST=your-server-ip
|
||||
LOGSERVER_PORT=5044
|
||||
API_KEY=your-api-key-here
|
||||
|
||||
# Install and start
|
||||
dropshell install logclient
|
||||
```
|
||||
|
||||
## Verify Setup
|
||||
|
||||
```bash
|
||||
# Check server status
|
||||
dropshell status logserver
|
||||
|
||||
# View server logs
|
||||
dropshell logs logserver
|
||||
|
||||
# Test client connection (from client)
|
||||
docker logs logclient-filebeat | grep "connection"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Elasticsearch won't start**: Check `vm.max_map_count` is 262144+
|
||||
|
||||
**No logs in Kibana**:
|
||||
- Verify client can reach server port 5044
|
||||
- Check API key is correct in client's service.env
|
||||
- Verify API key exists in server's api-keys.yml
|
||||
- Refresh index pattern in Kibana
|
||||
|
||||
**High memory usage**: Adjust heap sizes in `service.env`:
|
||||
```bash
|
||||
ES_HEAP_SIZE=1g # Reduce from 2g
|
||||
LS_HEAP_SIZE=512m # Reduce from 1g
|
||||
```
|
||||
|
||||
## Security Checklist
|
||||
|
||||
- [ ] Changed default Kibana password
|
||||
- [ ] Generated unique API key per client
|
||||
- [ ] API keys stored securely
|
||||
- [ ] Firewall allows only necessary ports (5601, 5044)
|
||||
- [ ] Regular backup configured
|
||||
- [ ] Reviewed api-keys.yml for old/unused keys
|
||||
@@ -1,84 +0,0 @@
|
||||
# Simple Guide to Viewing Logs in Kibana
|
||||
|
||||
## First Time Setup
|
||||
Run this once after installing LogServer:
|
||||
```bash
|
||||
./setup-kibana.sh
|
||||
```
|
||||
|
||||
## Viewing Logs - The Easy Way
|
||||
|
||||
### 1. Open Kibana
|
||||
Go to: `http://<your-server>:5601`
|
||||
|
||||
### 2. Login
|
||||
Use the username and password from your service.env
|
||||
|
||||
### 3. Click "Discover"
|
||||
It's in the left menu (looks like a compass icon)
|
||||
|
||||
### 4. You're Done!
|
||||
Your logs are now visible. That's it!
|
||||
|
||||
## Simple Controls
|
||||
|
||||
### See Recent Logs Only
|
||||
- Top-right corner: Click the time picker
|
||||
- Choose "Last 15 minutes" or "Last 1 hour"
|
||||
|
||||
### Filter by Container
|
||||
- Find any log entry
|
||||
- Next to `container_name`: click the `+` button
|
||||
- Now you only see logs from that container
|
||||
|
||||
### Filter by Server
|
||||
- Next to `host.name`: click the `+` button
|
||||
- Now you only see logs from that host
|
||||
|
||||
### Search for Text
|
||||
- Top search bar: Type any word
|
||||
- Press Enter
|
||||
- Shows only logs containing that word
|
||||
|
||||
### Live Updates
|
||||
- Top-right: Click "Refresh"
|
||||
- Choose "Every 5 seconds"
|
||||
- Logs update automatically
|
||||
|
||||
### Remove Filters
|
||||
- Look for filter pills under the search bar
|
||||
- Click the `x` on any filter to remove it
|
||||
|
||||
## Common Searches
|
||||
|
||||
**Show errors only:**
|
||||
```
|
||||
error OR ERROR OR Error
|
||||
```
|
||||
|
||||
**Show warnings and errors:**
|
||||
```
|
||||
error OR ERROR OR warn OR WARN
|
||||
```
|
||||
|
||||
**Show specific container:**
|
||||
```
|
||||
container_name: "myapp"
|
||||
```
|
||||
|
||||
**Show multiple containers:**
|
||||
```
|
||||
container_name: ("app1" OR "app2")
|
||||
```
|
||||
|
||||
## Tips
|
||||
|
||||
1. **Too many columns?** Click "container_name" and "message" in the left sidebar to show just those
|
||||
|
||||
2. **Want raw logs?** Click the ">" arrow next to any log entry to expand it
|
||||
|
||||
3. **Export logs?** Click "Share" → "CSV Reports" → "Generate CSV"
|
||||
|
||||
4. **Time zone wrong?** Click your profile icon → "Advanced Settings" → search "timezone"
|
||||
|
||||
That's all you need to know! Kibana has many advanced features, but for basic log viewing and searching, these commands are sufficient.
|
||||
@@ -1,244 +0,0 @@
|
||||
# LogServer Template - Implementation TODO
|
||||
|
||||
## Phase 1: Core Infrastructure (Priority 1)
|
||||
|
||||
### Configuration Files
|
||||
- [ ] Create `config/.template_info.env` with template metadata
|
||||
- [ ] Create `config/service.env` with user-configurable settings
|
||||
- [ ] Define all required environment variables (ports, passwords, heap sizes)
|
||||
- [ ] Set appropriate default values for zero-config experience
|
||||
|
||||
### Docker Compose Setup
|
||||
- [ ] Create `docker-compose.yml` with ELK stack services
|
||||
- [ ] Configure Elasticsearch single-node setup
|
||||
- [ ] Configure Logstash with Beats input pipeline
|
||||
- [ ] Configure Kibana with Elasticsearch connection
|
||||
- [ ] Set up proper networking between services
|
||||
- [ ] Define named volumes for data persistence
|
||||
- [ ] Configure health checks for each service
|
||||
|
||||
### Required Scripts
|
||||
- [ ] Implement `install.sh` - Pull images, create volumes, start services
|
||||
- [ ] Implement `uninstall.sh` - Stop and remove containers (preserve volumes!)
|
||||
- [ ] Implement `start.sh` - Start all ELK services with docker-compose
|
||||
- [ ] Implement `stop.sh` - Gracefully stop all services
|
||||
- [ ] Implement `status.sh` - Check health of all three services
|
||||
|
||||
## Phase 2: Logstash Configuration (Priority 1)
|
||||
|
||||
### Input Configuration
|
||||
- [ ] Configure Beats input on port 5044 with TLS/SSL
|
||||
- [ ] Set up mutual TLS (mTLS) authentication
|
||||
- [ ] Configure client certificate validation
|
||||
- [ ] Add API key authentication option
|
||||
- [ ] Implement IP whitelisting
|
||||
- [ ] Add Syslog input on port 514 (UDP/TCP) - unauthenticated
|
||||
- [ ] Add Docker Fluentd input on port 24224 (optional)
|
||||
|
||||
### Filter Pipeline
|
||||
- [ ] Create Docker log parser (extract container metadata)
|
||||
- [ ] Create Syslog parser (RFC3164 and RFC5424)
|
||||
- [ ] Add JSON parser for structured logs
|
||||
- [ ] Implement multiline pattern for stack traces
|
||||
- [ ] Add timestamp extraction and normalization
|
||||
- [ ] Create field enrichment (add host metadata)
|
||||
- [ ] Implement conditional routing based on log type
|
||||
|
||||
### Output Configuration
|
||||
- [ ] Configure Elasticsearch output with index patterns
|
||||
- [ ] Set up index templates for different log types
|
||||
- [ ] Configure index lifecycle management (ILM)
|
||||
|
||||
## Phase 3: Elasticsearch Setup (Priority 1)
|
||||
|
||||
### System Configuration
|
||||
- [ ] Set appropriate heap size defaults (ES_HEAP_SIZE)
|
||||
- [ ] Configure vm.max_map_count requirement check
|
||||
- [ ] Set up single-node discovery settings
|
||||
- [ ] Configure data persistence volume
|
||||
- [ ] Set up index templates for:
|
||||
- [ ] Docker logs (docker-*)
|
||||
- [ ] System logs (syslog-*)
|
||||
- [ ] Application logs (app-*)
|
||||
- [ ] Error logs (errors-*)
|
||||
|
||||
### Index Management
|
||||
- [ ] Configure ILM policies for log rotation
|
||||
- [ ] Set retention period (default 30 days)
|
||||
- [ ] Configure max index size limits
|
||||
- [ ] Set up automatic cleanup of old indices
|
||||
- [ ] Create snapshot repository configuration
|
||||
|
||||
## Phase 4: Kibana Configuration (Priority 2)
|
||||
|
||||
### Initial Setup
|
||||
- [ ] Configure Kibana with Elasticsearch URL
|
||||
- [ ] Set up basic authentication
|
||||
- [ ] Configure server base path
|
||||
- [ ] Set appropriate memory limits
|
||||
|
||||
### Pre-built Dashboards
|
||||
- [ ] Create System Overview dashboard
|
||||
- [ ] Create Docker Containers dashboard
|
||||
- [ ] Create Error Analysis dashboard
|
||||
- [ ] Create Security Events dashboard
|
||||
- [ ] Create Host Metrics dashboard
|
||||
|
||||
### Saved Searches
|
||||
- [ ] Error logs across all sources
|
||||
- [ ] Authentication events
|
||||
- [ ] Container lifecycle events
|
||||
- [ ] Slow queries/performance issues
|
||||
- [ ] Critical system events
|
||||
|
||||
### Index Patterns
|
||||
- [ ] Configure docker-* pattern
|
||||
- [ ] Configure syslog-* pattern
|
||||
- [ ] Configure app-* pattern
|
||||
- [ ] Configure filebeat-* pattern
|
||||
|
||||
## Phase 5: Optional Scripts (Priority 2)
|
||||
|
||||
### Operational Scripts
|
||||
- [ ] Implement `logs.sh` - Show logs from all ELK services
|
||||
- [ ] Implement `backup.sh` - Snapshot Elasticsearch indices
|
||||
- [ ] Implement `restore.sh` - Restore from snapshots
|
||||
- [ ] Implement `destroy.sh` - Complete removal including volumes
|
||||
- [ ] Implement `ports.sh` - Display all exposed ports
|
||||
- [ ] Implement `ssh.sh` - Shell into specific container
|
||||
|
||||
### Helper Scripts
|
||||
- [ ] Create `_volumes.sh` for volume management helpers
|
||||
- [ ] Add health check script for all services
|
||||
- [ ] Create performance tuning script
|
||||
- [ ] Add certificate generation script for SSL
|
||||
|
||||
## Phase 6: Security Features (Priority 1 - CRITICAL)
|
||||
|
||||
### Certificate Authority Setup
|
||||
- [ ] Create CA certificate and key for signing client certs
|
||||
- [ ] Generate server certificate for Logstash
|
||||
- [ ] Create certificate generation script for clients
|
||||
- [ ] Set up certificate storage structure
|
||||
- [ ] Implement certificate rotation mechanism
|
||||
|
||||
### mTLS Authentication
|
||||
- [ ] Configure Logstash for mutual TLS
|
||||
- [ ] Set up client certificate validation
|
||||
- [ ] Create client certificate generation script
|
||||
- [ ] Implement certificate revocation list (CRL)
|
||||
- [ ] Add certificate expiry monitoring
|
||||
|
||||
### API Key Authentication
|
||||
- [ ] Create API key generation script
|
||||
- [ ] Configure Logstash to accept API keys
|
||||
- [ ] Implement API key storage (encrypted)
|
||||
- [ ] Add API key rotation mechanism
|
||||
- [ ] Create API key revocation process
|
||||
|
||||
### Network Security
|
||||
- [ ] Implement IP whitelisting in Logstash
|
||||
- [ ] Configure firewall rules
|
||||
- [ ] Set up rate limiting
|
||||
- [ ] Add connection throttling
|
||||
- [ ] Implement DDoS protection
|
||||
|
||||
### Kibana Security
|
||||
- [ ] Configure Kibana HTTPS
|
||||
- [ ] Set up basic authentication
|
||||
- [ ] Create user management scripts
|
||||
- [ ] Implement session management
|
||||
- [ ] Add audit logging
|
||||
|
||||
## Phase 7: Performance & Optimization (Priority 3)
|
||||
|
||||
### Resource Management
|
||||
- [ ] Configure CPU limits for each service
|
||||
- [ ] Set memory limits appropriately
|
||||
- [ ] Add swap handling configuration
|
||||
- [ ] Configure JVM options files
|
||||
- [ ] Add performance monitoring
|
||||
|
||||
### Optimization
|
||||
- [ ] Configure pipeline workers
|
||||
- [ ] Set batch sizes for optimal throughput
|
||||
- [ ] Configure queue sizes
|
||||
- [ ] Add caching configuration
|
||||
- [ ] Optimize index refresh intervals
|
||||
|
||||
## Phase 8: Testing & Documentation (Priority 3)
|
||||
|
||||
### Testing
|
||||
- [ ] Test installation process
|
||||
- [ ] Test uninstall (verify volume preservation)
|
||||
- [ ] Test log ingestion from sample client
|
||||
- [ ] Test all dashboard functionality
|
||||
- [ ] Test backup and restore procedures
|
||||
- [ ] Load test with high log volume
|
||||
- [ ] Test failover and recovery
|
||||
|
||||
### Documentation
|
||||
- [ ] Create README.txt for dropshell format
|
||||
- [ ] Document all configuration options
|
||||
- [ ] Add troubleshooting guide
|
||||
- [ ] Create quick start guide
|
||||
- [ ] Document upgrade procedures
|
||||
- [ ] Add performance tuning guide
|
||||
|
||||
## Phase 9: Integration Testing (Priority 3)
|
||||
|
||||
### With LogClient
|
||||
- [ ] Test automatic discovery
|
||||
- [ ] Verify log flow from client to server
|
||||
- [ ] Test reconnection scenarios
|
||||
- [ ] Verify all log types are parsed correctly
|
||||
- [ ] Test SSL communication
|
||||
- [ ] Measure end-to-end latency
|
||||
|
||||
### Compatibility Testing
|
||||
- [ ] Test with different Docker versions
|
||||
- [ ] Test on various Linux distributions
|
||||
- [ ] Verify with different log formats
|
||||
- [ ] Test with high-volume producers
|
||||
- [ ] Validate resource usage
|
||||
|
||||
## Phase 10: Production Readiness (Priority 4)
|
||||
|
||||
### Monitoring & Alerting
|
||||
- [ ] Add Elasticsearch monitoring
|
||||
- [ ] Configure disk space alerts
|
||||
- [ ] Set up index health monitoring
|
||||
- [ ] Add performance metrics collection
|
||||
- [ ] Create alert rules in Kibana
|
||||
|
||||
### Maintenance Features
|
||||
- [ ] Add automatic update check
|
||||
- [ ] Create maintenance mode
|
||||
- [ ] Add data export functionality
|
||||
- [ ] Create migration scripts
|
||||
- [ ] Add configuration validation
|
||||
|
||||
## Notes
|
||||
|
||||
### Design Principles
|
||||
1. **Minimum configuration**: Should work with just `dropshell install logserver`
|
||||
2. **Data safety**: Never delete volumes in uninstall.sh
|
||||
3. **Non-interactive**: All scripts must run without user input
|
||||
4. **Idempotent**: Scripts can be run multiple times safely
|
||||
5. **Clear feedback**: Provide clear status and error messages
|
||||
|
||||
### Dependencies
|
||||
- Docker and Docker Compose
|
||||
- Sufficient system resources (4GB+ RAM recommended)
|
||||
- Network connectivity for clients
|
||||
- Persistent storage for logs
|
||||
|
||||
### Testing Checklist
|
||||
- [ ] All required scripts present and executable
|
||||
- [ ] Template validates with dropshell test-template
|
||||
- [ ] Services start and connect properly
|
||||
- [ ] Logs flow from client to Kibana
|
||||
- [ ] Data persists across container restarts
|
||||
- [ ] Uninstall preserves data volumes
|
||||
- [ ] Resource limits are enforced
|
||||
- [ ] Error handling works correctly
|
||||
@@ -1,9 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Define volume items for logserver container
|
||||
# These are used across backup, restore, create, and destroy operations
|
||||
|
||||
get_logserver_volumes() {
|
||||
echo "volume:elasticsearch_data:${CONTAINER_NAME}_elasticsearch_data" \
|
||||
"volume:logstash_data:${CONTAINER_NAME}_logstash_data" \
|
||||
"volume:kibana_data:${CONTAINER_NAME}_kibana_data"
|
||||
}
|
||||
@@ -1,37 +0,0 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
||||
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
echo "Backing up ${CONTAINER_NAME} ELK stack volumes..."
|
||||
echo "Note: This may take a while for large log databases"
|
||||
|
||||
# Stop the containers to ensure data consistency
|
||||
bash ./stop.sh || true
|
||||
|
||||
# Backup volumes
|
||||
BACKUP_DIR="${CONFIG_PATH}/backups/$(date +%Y%m%d_%H%M%S)"
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
# Export volumes
|
||||
for volume in $(get_logserver_volumes); do
|
||||
volume_name=$(echo $volume | cut -d: -f3)
|
||||
echo "Backing up volume: $volume_name"
|
||||
docker run --rm -v "$volume_name:/source:ro" -v "$BACKUP_DIR:/backup" alpine \
|
||||
tar -czf "/backup/${volume_name}.tar.gz" -C /source .
|
||||
done
|
||||
|
||||
# Backup configuration
|
||||
cp -r "${CONFIG_PATH}" "$BACKUP_DIR/config_backup"
|
||||
|
||||
# Backup docker-compose.yml
|
||||
cp docker-compose.yml "$BACKUP_DIR/" 2>/dev/null || true
|
||||
|
||||
echo "Backup completed to: $BACKUP_DIR"
|
||||
echo "Size: $(du -sh $BACKUP_DIR | cut -f1)"
|
||||
|
||||
# Restart the containers
|
||||
bash ./start.sh
|
||||
|
||||
echo "ELK stack restarted"
|
||||
@@ -1,17 +0,0 @@
|
||||
# Template identifier - MUST match the directory name
|
||||
TEMPLATE=logserver
|
||||
|
||||
# Requirements
|
||||
REQUIRES_HOST_ROOT=false # No root access on host needed
|
||||
REQUIRES_DOCKER=true # Docker is required
|
||||
REQUIRES_DOCKER_ROOT=false # Docker root privileges not specifically needed
|
||||
|
||||
# Docker compose used for ELK stack
|
||||
USES_DOCKER_COMPOSE=true
|
||||
|
||||
# Volume definitions for persistence
|
||||
DATA_VOLUME="${CONTAINER_NAME}_elasticsearch_data"
|
||||
LOGSTASH_VOLUME="${CONTAINER_NAME}_logstash_data"
|
||||
KIBANA_VOLUME="${CONTAINER_NAME}_kibana_data"
|
||||
CERTS_VOLUME="${CONTAINER_NAME}_certs"
|
||||
CONFIG_VOLUME="${CONTAINER_NAME}_config"
|
||||
@@ -1,6 +0,0 @@
|
||||
# API Keys for LogServer Authentication
|
||||
# Format: hostname:api_key
|
||||
# Generated by generate-api-key.sh
|
||||
|
||||
api_keys:
|
||||
video: a7798c63c2ac439b5ba20f3bf8bf27b5361231cdcbdc4fc9d7af715308fdf707
|
||||
154
logserver/config/dashboards/central-logs.json
Normal file
154
logserver/config/dashboards/central-logs.json
Normal file
@@ -0,0 +1,154 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": "-- Grafana --",
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"gnetId": null,
|
||||
"graphTooltip": 0,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": "Loki",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 3,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"content": "## Central Log Viewer\n\n**Quick Filters:** Click any label (hostname, container_name, job) to filter | **Search:** Use the search box above each panel | **Time Range:** Top right corner",
|
||||
"mode": "markdown"
|
||||
},
|
||||
"pluginVersion": "7.5.7",
|
||||
"title": "",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"datasource": "Loki",
|
||||
"fieldConfig": {
|
||||
"defaults": {},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 3
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "{job=\"docker\"}",
|
||||
"legendFormat": "{{hostname}} - {{container_name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Docker Container Logs (All Servers)",
|
||||
"type": "logs"
|
||||
},
|
||||
{
|
||||
"datasource": "Loki",
|
||||
"fieldConfig": {
|
||||
"defaults": {},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 13
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "{job=~\"syslog|auth\"}",
|
||||
"legendFormat": "{{hostname}} - {{job}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "System Logs (All Servers)",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"refresh": "10s",
|
||||
"schemaVersion": 27,
|
||||
"style": "dark",
|
||||
"tags": ["logs", "central"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"allValue": ".*",
|
||||
"current": {
|
||||
"selected": true,
|
||||
"text": ["All"],
|
||||
"value": ["$__all"]
|
||||
},
|
||||
"datasource": "Loki",
|
||||
"definition": "label_values(hostname)",
|
||||
"description": "Filter by server",
|
||||
"error": null,
|
||||
"hide": 0,
|
||||
"includeAll": true,
|
||||
"label": "Server",
|
||||
"multi": true,
|
||||
"name": "hostname",
|
||||
"options": [],
|
||||
"query": "label_values(hostname)",
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"sort": 1,
|
||||
"type": "query"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-15m",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {
|
||||
"refresh_intervals": ["5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h"]
|
||||
},
|
||||
"timezone": "",
|
||||
"title": "Central Logs",
|
||||
"uid": "central-logs",
|
||||
"version": 0
|
||||
}
|
||||
69
logserver/config/dashboards/simple-logs.json
Normal file
69
logserver/config/dashboards/simple-logs.json
Normal file
@@ -0,0 +1,69 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": "-- Grafana --",
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"gnetId": null,
|
||||
"graphTooltip": 0,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": "Loki",
|
||||
"fieldConfig": {
|
||||
"defaults": {},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 20,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "{job=~\"docker|syslog|auth\"}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "All Logs",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"schemaVersion": 27,
|
||||
"style": "dark",
|
||||
"tags": ["logs"],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-15m",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "",
|
||||
"title": "Simple Logs",
|
||||
"uid": "simple-logs",
|
||||
"version": 0
|
||||
}
|
||||
12
logserver/config/grafana-dashboard.yml
Normal file
12
logserver/config/grafana-dashboard.yml
Normal file
@@ -0,0 +1,12 @@
|
||||
apiVersion: 1
|
||||
|
||||
providers:
|
||||
- name: 'default'
|
||||
orgId: 1
|
||||
folder: ''
|
||||
type: file
|
||||
disableDeletion: false
|
||||
updateIntervalSeconds: 10
|
||||
allowUiUpdates: true
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
9
logserver/config/grafana-datasources.yml
Normal file
9
logserver/config/grafana-datasources.yml
Normal file
@@ -0,0 +1,9 @@
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
isDefault: true
|
||||
editable: false
|
||||
@@ -1,142 +0,0 @@
|
||||
# Logstash Configuration for LogServer
|
||||
# Handles Beats input with API key authentication
|
||||
|
||||
input {
|
||||
# Beats input for Filebeat clients
|
||||
beats {
|
||||
port => 5044
|
||||
ssl => false # Set to true for production with proper certificates
|
||||
|
||||
# API key authentication handled via filter below
|
||||
}
|
||||
|
||||
# Optional: Syslog input for direct syslog shipping
|
||||
tcp {
|
||||
port => 514
|
||||
type => "syslog"
|
||||
}
|
||||
|
||||
udp {
|
||||
port => 514
|
||||
type => "syslog"
|
||||
}
|
||||
}
|
||||
|
||||
filter {
|
||||
# API Key validation - check if client provided a valid key
|
||||
# The API key should be in the [fields][api_key] field from Filebeat
|
||||
if [fields][api_key] {
|
||||
# Load and validate API key
|
||||
ruby {
|
||||
init => "
|
||||
require 'yaml'
|
||||
@api_keys = {}
|
||||
|
||||
# Load API keys from file
|
||||
begin
|
||||
if File.exist?('/usr/share/logstash/config/api-keys.yml')
|
||||
config = YAML.load_file('/usr/share/logstash/config/api-keys.yml')
|
||||
if config && config['api_keys']
|
||||
config['api_keys'].each do |hostname, key|
|
||||
@api_keys[key.to_s.strip] = hostname.to_s.strip
|
||||
end
|
||||
end
|
||||
end
|
||||
rescue => e
|
||||
@logger.error('Failed to load API keys', :error => e.message)
|
||||
end
|
||||
"
|
||||
code => "
|
||||
api_key = event.get('[fields][api_key]')
|
||||
|
||||
if api_key && @api_keys.has_key?(api_key)
|
||||
# Valid API key - add hostname to event
|
||||
event.set('[@metadata][client_hostname]', @api_keys[api_key])
|
||||
event.set('[@metadata][authenticated]', true)
|
||||
else
|
||||
# Invalid API key
|
||||
event.set('[@metadata][authenticated]', false)
|
||||
event.tag('_authfailure')
|
||||
end
|
||||
"
|
||||
}
|
||||
|
||||
# Drop unauthorized events
|
||||
if "_authfailure" in [tags] {
|
||||
drop { }
|
||||
}
|
||||
} else {
|
||||
# No API key provided - mark as unauthenticated
|
||||
# You can choose to drop these or allow them based on your security requirements
|
||||
mutate {
|
||||
add_tag => [ "no_api_key" ]
|
||||
}
|
||||
# Uncomment to require API keys for all connections:
|
||||
# drop { }
|
||||
}
|
||||
|
||||
# Parse Docker logs
|
||||
if [docker] {
|
||||
# Docker metadata is already parsed by Filebeat
|
||||
mutate {
|
||||
add_field => {
|
||||
"container_name" => "%{[docker][container][name]}"
|
||||
"container_id" => "%{[docker][container][id]}"
|
||||
"container_image" => "%{[docker][container][image]}"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Parse syslog
|
||||
if [type] == "syslog" {
|
||||
grok {
|
||||
match => {
|
||||
"message" => "%{SYSLOGLINE}"
|
||||
}
|
||||
}
|
||||
date {
|
||||
match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
|
||||
}
|
||||
}
|
||||
|
||||
# Parse JSON logs if they exist
|
||||
if [message] =~ /^\{.*\}$/ {
|
||||
json {
|
||||
source => "message"
|
||||
target => "json_message"
|
||||
}
|
||||
}
|
||||
|
||||
# Add timestamp if not present
|
||||
if ![timestamp] {
|
||||
mutate {
|
||||
add_field => { "timestamp" => "%{@timestamp}" }
|
||||
}
|
||||
}
|
||||
|
||||
# Clean up metadata
|
||||
mutate {
|
||||
remove_field => [ "@version", "beat", "offset", "prospector" ]
|
||||
}
|
||||
}
|
||||
|
||||
output {
|
||||
# Send to Elasticsearch with authentication
|
||||
elasticsearch {
|
||||
hosts => ["elasticsearch:9200"]
|
||||
user => "elastic"
|
||||
password => "${ELASTIC_PASSWORD:changeme}"
|
||||
|
||||
# Use different indices based on input type
|
||||
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
|
||||
|
||||
# Manage index templates
|
||||
manage_template => true
|
||||
template_overwrite => true
|
||||
}
|
||||
|
||||
# Optional: Debug output (comment out in production)
|
||||
# stdout {
|
||||
# codec => rubydebug
|
||||
# }
|
||||
}
|
||||
@@ -1,29 +0,0 @@
|
||||
# Logstash Configuration Settings
|
||||
# This file contains Logstash settings (not pipeline configuration)
|
||||
|
||||
# Node name
|
||||
node.name: "logstash"
|
||||
|
||||
# Pipeline settings
|
||||
pipeline.workers: 2
|
||||
pipeline.batch.size: 125
|
||||
pipeline.batch.delay: 50
|
||||
pipeline.ecs_compatibility: disabled
|
||||
|
||||
# HTTP API settings
|
||||
http.host: "0.0.0.0"
|
||||
http.port: 9600
|
||||
|
||||
# Monitoring
|
||||
monitoring.enabled: false
|
||||
|
||||
# Queue settings
|
||||
queue.type: memory
|
||||
queue.max_bytes: 1gb
|
||||
|
||||
# Path settings are handled by Docker volumes
|
||||
# path.data: /usr/share/logstash/data
|
||||
# path.logs: /usr/share/logstash/logs
|
||||
|
||||
# Log level
|
||||
log.level: info
|
||||
51
logserver/config/loki.yaml
Normal file
51
logserver/config/loki.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
auth_enabled: false
|
||||
|
||||
server:
|
||||
http_listen_port: 3100
|
||||
grpc_listen_port: 9096
|
||||
|
||||
ingester:
|
||||
wal:
|
||||
enabled: true
|
||||
dir: /loki/wal
|
||||
lifecycler:
|
||||
address: 127.0.0.1
|
||||
ring:
|
||||
kvstore:
|
||||
store: inmemory
|
||||
replication_factor: 1
|
||||
|
||||
schema_config:
|
||||
configs:
|
||||
- from: 2020-10-24
|
||||
store: boltdb-shipper
|
||||
object_store: filesystem
|
||||
schema: v11
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
|
||||
storage_config:
|
||||
boltdb_shipper:
|
||||
active_index_directory: /loki/boltdb-shipper-active
|
||||
cache_location: /loki/boltdb-shipper-cache
|
||||
cache_ttl: 24h
|
||||
filesystem:
|
||||
directory: /loki/chunks
|
||||
|
||||
compactor:
|
||||
working_directory: /loki/boltdb-shipper-compactor
|
||||
|
||||
limits_config:
|
||||
enforce_metric_name: false
|
||||
reject_old_samples: true
|
||||
reject_old_samples_max_age: 168h
|
||||
max_entries_limit_per_query: 5000
|
||||
retention_period: 168h # 7 days
|
||||
|
||||
chunk_store_config:
|
||||
max_look_back_period: 168h
|
||||
|
||||
table_manager:
|
||||
retention_deletes_enabled: true
|
||||
retention_period: 168h
|
||||
30
logserver/config/nginx.conf
Normal file
30
logserver/config/nginx.conf
Normal file
@@ -0,0 +1,30 @@
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
upstream loki {
|
||||
server loki:3100;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
|
||||
# Require authentication for all requests
|
||||
auth_basic "Loki Authentication";
|
||||
auth_basic_user_file /etc/nginx/.htpasswd;
|
||||
|
||||
# Proxy all requests to Loki
|
||||
location / {
|
||||
proxy_pass http://loki;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Important for Loki push endpoint
|
||||
client_max_body_size 100M;
|
||||
client_body_buffer_size 10M;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,46 +1,20 @@
|
||||
# Service identification
|
||||
# Log Server Configuration (Loki + Grafana)
|
||||
CONTAINER_NAME=logserver
|
||||
|
||||
# Server settings (REQUIRED by dropshell)
|
||||
SSH_USER="root"
|
||||
|
||||
# Elasticsearch settings
|
||||
ES_VERSION=7.17.23
|
||||
ES_HEAP_SIZE=2g
|
||||
ES_MAX_MAP_COUNT=262144
|
||||
|
||||
# Logstash settings
|
||||
LS_VERSION=7.17.23
|
||||
LS_HEAP_SIZE=1g
|
||||
LS_PIPELINE_WORKERS=2
|
||||
|
||||
# Kibana settings
|
||||
KIBANA_VERSION=7.17.23
|
||||
|
||||
# Authentication (IMPORTANT: Change these!)
|
||||
ELASTIC_PASSWORD=changeme # Password for 'elastic' superuser (internal use)
|
||||
KIBANA_USERNAME=admin # Your login username for Kibana
|
||||
KIBANA_USER_PASSWORD=changeme # Your login password for Kibana
|
||||
|
||||
# Ports
|
||||
KIBANA_PORT=5601
|
||||
LOGSTASH_BEATS_PORT=5044
|
||||
LOGSTASH_SYSLOG_PORT=514
|
||||
WEB_PORT=3000 # Grafana web UI
|
||||
LOKI_PORT=3100 # Loki API (for clients to send logs)
|
||||
|
||||
# Server configuration
|
||||
SERVER_PUBLICBASEURL=http://localhost:5601 # Change to your server's actual URL
|
||||
# Authentication for Grafana
|
||||
ADMIN_USER=admin
|
||||
ADMIN_PASSWORD=changeme
|
||||
|
||||
# Log retention
|
||||
LOG_RETENTION_DAYS=30
|
||||
LOG_MAX_SIZE_GB=50
|
||||
# Authentication for Loki (clients must use these)
|
||||
LOKI_USER=logclient
|
||||
LOKI_PASSWORD=changeme
|
||||
|
||||
# Authentication
|
||||
ENABLE_TLS=true
|
||||
API_KEYS_FILE=${CONFIG_PATH}/api-keys.yml
|
||||
|
||||
# Network Security
|
||||
ALLOWED_IPS="" # Comma-separated list, empty = all
|
||||
|
||||
# Resource limits
|
||||
MAX_CPU_PERCENT=80
|
||||
MAX_MEMORY=4GB
|
||||
# Log retention (days)
|
||||
LOG_RETENTION=7
|
||||
@@ -1,43 +0,0 @@
|
||||
# Ruby script for Logstash to validate API keys
|
||||
# This is a simplified validation - in production, use proper authentication
|
||||
|
||||
require 'yaml'
|
||||
|
||||
def register(params)
|
||||
@api_keys_file = params["api_keys_file"]
|
||||
end
|
||||
|
||||
def filter(event)
|
||||
# Get the API key from the event
|
||||
api_key = event.get("[api_key]") || event.get("[@metadata][api_key]")
|
||||
|
||||
# If no API key, pass through (for backwards compatibility)
|
||||
# In production, you should reject events without valid keys
|
||||
if api_key.nil? || api_key.empty?
|
||||
# For now, allow events without API keys
|
||||
# event.cancel # Uncomment to require API keys
|
||||
return [event]
|
||||
end
|
||||
|
||||
# Load API keys from file
|
||||
begin
|
||||
if File.exist?(@api_keys_file)
|
||||
config = YAML.load_file(@api_keys_file)
|
||||
valid_keys = config['api_keys'].values if config && config['api_keys']
|
||||
|
||||
# Check if the provided key is valid
|
||||
if valid_keys && valid_keys.include?(api_key)
|
||||
# Valid key - let the event through
|
||||
event.set("[@metadata][authenticated]", true)
|
||||
else
|
||||
# Invalid key - drop the event
|
||||
event.cancel
|
||||
end
|
||||
end
|
||||
rescue => e
|
||||
# Log error but don't crash
|
||||
event.set("[@metadata][auth_error]", e.message)
|
||||
end
|
||||
|
||||
return [event]
|
||||
end
|
||||
@@ -1,26 +0,0 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
||||
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
echo "WARNING: This will PERMANENTLY DELETE all ELK stack data!"
|
||||
echo "This includes all logs, dashboards, and configurations"
|
||||
echo "This action cannot be undone!"
|
||||
echo ""
|
||||
|
||||
# Since scripts must be non-interactive, only proceed if explicitly called
|
||||
# Stop all containers
|
||||
docker compose down || true
|
||||
|
||||
# Remove all volumes
|
||||
for volume in $(get_logserver_volumes); do
|
||||
volume_name=$(echo $volume | cut -d: -f3)
|
||||
echo "Removing volume: $volume_name"
|
||||
docker volume rm "$volume_name" 2>/dev/null || true
|
||||
done
|
||||
|
||||
# Also remove the Docker network if it exists
|
||||
docker network rm "${CONTAINER_NAME}_elk" 2>/dev/null || true
|
||||
|
||||
echo "ELK stack and all data destroyed"
|
||||
@@ -1,80 +1,57 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
elasticsearch:
|
||||
image: docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION:-7.17.23}
|
||||
container_name: ${CONTAINER_NAME}_elasticsearch
|
||||
# Grafana - Simple Web UI
|
||||
grafana:
|
||||
image: grafana/grafana:latest
|
||||
container_name: ${CONTAINER_NAME}_grafana
|
||||
environment:
|
||||
- discovery.type=single-node
|
||||
- "ES_JAVA_OPTS=-Xms${ES_HEAP_SIZE:-2g} -Xmx${ES_HEAP_SIZE:-2g}"
|
||||
- xpack.security.enabled=true
|
||||
- xpack.security.authc.api_key.enabled=true
|
||||
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD:-${KIBANA_PASSWORD:-changeme}}
|
||||
- xpack.monitoring.enabled=false
|
||||
- cluster.routing.allocation.disk.threshold_enabled=false
|
||||
- GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
|
||||
- GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-changeme}
|
||||
- GF_USERS_ALLOW_SIGN_UP=false
|
||||
volumes:
|
||||
- elasticsearch_data:/usr/share/elasticsearch/data
|
||||
- grafana_data:/var/lib/grafana
|
||||
- ${CONFIG_PATH}/grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yaml:ro
|
||||
- ${CONFIG_PATH}/grafana-dashboard.yml:/etc/grafana/provisioning/dashboards/dashboard.yaml:ro
|
||||
- ${CONFIG_PATH}/dashboards:/var/lib/grafana/dashboards:ro
|
||||
ports:
|
||||
- "127.0.0.1:9200:9200"
|
||||
networks:
|
||||
- elk
|
||||
- "${WEB_PORT:-3000}:3000"
|
||||
restart: unless-stopped
|
||||
ulimits:
|
||||
memlock:
|
||||
soft: -1
|
||||
hard: -1
|
||||
nofile:
|
||||
soft: 65536
|
||||
hard: 65536
|
||||
|
||||
logstash:
|
||||
image: docker.elastic.co/logstash/logstash:${LS_VERSION:-7.17.23}
|
||||
container_name: ${CONTAINER_NAME}_logstash
|
||||
environment:
|
||||
- "LS_JAVA_OPTS=-Xms${LS_HEAP_SIZE:-1g} -Xmx${LS_HEAP_SIZE:-1g}"
|
||||
- "xpack.monitoring.enabled=false"
|
||||
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD:-${KIBANA_PASSWORD:-changeme}}
|
||||
- CONTAINER_NAME=${CONTAINER_NAME}
|
||||
command: logstash -f /usr/share/logstash/config/logstash.conf
|
||||
volumes:
|
||||
- ${CONFIG_PATH}:/usr/share/logstash/config
|
||||
- logstash_data:/usr/share/logstash/data
|
||||
ports:
|
||||
- "${LOGSTASH_BEATS_PORT:-5044}:5044"
|
||||
- "${LOGSTASH_SYSLOG_PORT:-514}:514/tcp"
|
||||
- "${LOGSTASH_SYSLOG_PORT:-514}:514/udp"
|
||||
networks:
|
||||
- elk
|
||||
depends_on:
|
||||
- elasticsearch
|
||||
restart: unless-stopped
|
||||
|
||||
kibana:
|
||||
image: docker.elastic.co/kibana/kibana:${KIBANA_VERSION:-7.17.23}
|
||||
container_name: ${CONTAINER_NAME}_kibana
|
||||
environment:
|
||||
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
|
||||
- ELASTICSEARCH_USERNAME=elastic
|
||||
- ELASTICSEARCH_PASSWORD=${ELASTIC_PASSWORD:-${KIBANA_PASSWORD:-changeme}}
|
||||
- XPACK_SECURITY_ENABLED=true
|
||||
- NODE_OPTIONS=--openssl-legacy-provider
|
||||
- SERVER_PUBLICBASEURL=${SERVER_PUBLICBASEURL:-http://localhost:5601}
|
||||
volumes:
|
||||
- kibana_data:/usr/share/kibana/data
|
||||
ports:
|
||||
- "${KIBANA_PORT:-5601}:5601"
|
||||
- loki
|
||||
networks:
|
||||
- elk
|
||||
depends_on:
|
||||
- elasticsearch
|
||||
- loki-net
|
||||
|
||||
# Loki - Log storage (not exposed directly)
|
||||
loki:
|
||||
image: grafana/loki:latest
|
||||
container_name: ${CONTAINER_NAME}_loki
|
||||
volumes:
|
||||
- loki_data:/loki
|
||||
- ${CONFIG_PATH}/loki.yaml:/etc/loki/local-config.yaml:ro
|
||||
command: -config.file=/etc/loki/local-config.yaml
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- loki-net
|
||||
|
||||
# Nginx - Authentication proxy for Loki
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
container_name: ${CONTAINER_NAME}_nginx
|
||||
volumes:
|
||||
- ${CONFIG_PATH}/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ${CONFIG_PATH}/.htpasswd:/etc/nginx/.htpasswd:ro
|
||||
ports:
|
||||
- "${LOKI_PORT:-3100}:80"
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- loki
|
||||
networks:
|
||||
- loki-net
|
||||
|
||||
networks:
|
||||
elk:
|
||||
driver: bridge
|
||||
loki-net:
|
||||
|
||||
volumes:
|
||||
elasticsearch_data:
|
||||
name: ${CONTAINER_NAME}_elasticsearch_data
|
||||
logstash_data:
|
||||
name: ${CONTAINER_NAME}_logstash_data
|
||||
kibana_data:
|
||||
name: ${CONTAINER_NAME}_kibana_data
|
||||
grafana_data:
|
||||
loki_data:
|
||||
@@ -1,172 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Interactive API Key Generation Script for LogServer
|
||||
# This script generates secure API keys and adds them to api-keys.yml
|
||||
|
||||
# Determine where to put the api-keys.yml file
|
||||
determine_api_keys_location() {
|
||||
# 1. If api-keys.yml already exists in current folder, use it
|
||||
if [ -f "./api-keys.yml" ]; then
|
||||
echo "./api-keys.yml"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# 2. If service.env exists in current folder, put keys here
|
||||
if [ -f "./service.env" ]; then
|
||||
echo "./api-keys.yml"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# 3. If config folder exists, put keys there
|
||||
if [ -d "./config" ]; then
|
||||
echo "./config/api-keys.yml"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# No valid location found
|
||||
return 1
|
||||
}
|
||||
|
||||
# Try to determine location
|
||||
if API_KEYS_FILE=$(determine_api_keys_location); then
|
||||
: # Location found, continue
|
||||
else
|
||||
echo -e "${RED}Error: Cannot determine where to place api-keys.yml${NC}"
|
||||
echo ""
|
||||
echo "This script must be run from one of these locations:"
|
||||
echo " 1. A deployed service directory (contains service.env)"
|
||||
echo " 2. The logserver template directory (contains config/ folder)"
|
||||
echo " 3. A directory with existing api-keys.yml file"
|
||||
echo ""
|
||||
echo "Current directory: $(pwd)"
|
||||
echo "Contents: $(ls -la 2>/dev/null | head -5)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Generate a secure random API key
|
||||
generate_key() {
|
||||
openssl rand -hex 32 2>/dev/null || cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1
|
||||
}
|
||||
|
||||
# Initialize api-keys.yml if it doesn't exist
|
||||
init_api_keys_file() {
|
||||
if [ ! -f "$API_KEYS_FILE" ]; then
|
||||
# Create directory if needed
|
||||
local dir=$(dirname "$API_KEYS_FILE")
|
||||
if [ ! -d "$dir" ]; then
|
||||
mkdir -p "$dir"
|
||||
echo -e "${GREEN}Created directory: $dir${NC}"
|
||||
fi
|
||||
|
||||
echo "# API Keys for LogServer Authentication" > "$API_KEYS_FILE"
|
||||
echo "# Format: hostname:api_key" >> "$API_KEYS_FILE"
|
||||
echo "# Generated by generate-api-key.sh" >> "$API_KEYS_FILE"
|
||||
echo "" >> "$API_KEYS_FILE"
|
||||
echo "api_keys:" >> "$API_KEYS_FILE"
|
||||
echo -e "${GREEN}Created new api-keys.yml file at: $API_KEYS_FILE${NC}"
|
||||
else
|
||||
echo -e "${GREEN}Using existing api-keys.yml at: $API_KEYS_FILE${NC}"
|
||||
fi
|
||||
}
|
||||
|
||||
# Check if hostname already has a key
|
||||
check_existing_key() {
|
||||
local hostname=$1
|
||||
if grep -q "^ ${hostname}:" "$API_KEYS_FILE" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
# Add key to api-keys.yml
|
||||
add_key_to_file() {
|
||||
local hostname=$1
|
||||
local api_key=$2
|
||||
echo " ${hostname}: ${api_key}" >> "$API_KEYS_FILE"
|
||||
}
|
||||
|
||||
# Main script
|
||||
echo -e "${GREEN}=== LogServer API Key Generator ===${NC}"
|
||||
echo ""
|
||||
|
||||
# Initialize file if needed
|
||||
init_api_keys_file
|
||||
|
||||
# Interactive mode
|
||||
while true; do
|
||||
echo -e "${YELLOW}Enter hostname for the client (or 'done' to finish):${NC}"
|
||||
read -p "> " hostname
|
||||
|
||||
if [ "$hostname" = "done" ] || [ -z "$hostname" ]; then
|
||||
break
|
||||
fi
|
||||
|
||||
# Validate hostname - allow simple names, must start with alphanumeric
|
||||
if [[ ! "$hostname" =~ ^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$ ]]; then
|
||||
echo -e "${RED}Invalid hostname format. Use only letters, numbers, dots, dashes, and underscores.${NC}"
|
||||
echo -e "${RED}Hostname must start and end with a letter or number.${NC}"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Check if key already exists
|
||||
if check_existing_key "$hostname"; then
|
||||
echo -e "${YELLOW}Key already exists for ${hostname}${NC}"
|
||||
read -p "Generate new key? (y/n): " overwrite
|
||||
if [ "$overwrite" != "y" ]; then
|
||||
continue
|
||||
fi
|
||||
# Remove old key
|
||||
sed -i "/^ ${hostname}:/d" "$API_KEYS_FILE"
|
||||
fi
|
||||
|
||||
# Generate new key
|
||||
api_key=$(generate_key)
|
||||
|
||||
# Add to file
|
||||
add_key_to_file "$hostname" "$api_key"
|
||||
|
||||
echo -e "${GREEN}✓ Generated API key for ${hostname}${NC}"
|
||||
echo ""
|
||||
echo "Configuration for ${hostname}:"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "Add to client's service.env:"
|
||||
echo ""
|
||||
echo "LOGSERVER_HOST=$(hostname -I | awk '{print $1}')"
|
||||
echo "LOGSERVER_PORT=5044"
|
||||
echo "API_KEY=${api_key}"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
|
||||
# Option to add more
|
||||
read -p "Add another client? (y/n): " add_more
|
||||
if [ "$add_more" != "y" ]; then
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
# Show summary
|
||||
echo ""
|
||||
echo -e "${GREEN}=== Summary ===${NC}"
|
||||
echo "API keys file: $API_KEYS_FILE"
|
||||
echo "Total clients configured: $(grep -c "^ " "$API_KEYS_FILE" 2>/dev/null || echo 0)"
|
||||
echo ""
|
||||
echo "To view all keys: cat $API_KEYS_FILE"
|
||||
echo "To revoke a key: Edit $API_KEYS_FILE and remove the line"
|
||||
echo ""
|
||||
|
||||
# Show location-specific restart instructions
|
||||
if [[ "$API_KEYS_FILE" == "./api-keys.yml" ]] && [ -f "./service.env" ]; then
|
||||
# We're in a deployed service directory
|
||||
echo -e "${YELLOW}Remember to restart the service to apply changes:${NC}"
|
||||
echo " dropshell restart logserver"
|
||||
else
|
||||
# We're in the template directory
|
||||
echo -e "${YELLOW}Note: Deploy this template to use these keys:${NC}"
|
||||
echo " dropshell install logserver"
|
||||
fi
|
||||
@@ -2,128 +2,48 @@
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# Check required environment variables
|
||||
_check_required_env_vars "CONTAINER_NAME" "ES_VERSION" "LS_VERSION" "KIBANA_VERSION"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
# Check Docker and Docker Compose are available
|
||||
# Check Docker
|
||||
_check_docker_installed || _die "Docker test failed"
|
||||
docker compose version >/dev/null 2>&1 || _die "Docker Compose is not installed (requires Docker Compose V2)"
|
||||
|
||||
# Check vm.max_map_count for Elasticsearch
|
||||
current_max_map_count=$(sysctl -n vm.max_map_count 2>/dev/null || echo 0)
|
||||
if [ "$current_max_map_count" -lt 262144 ]; then
|
||||
echo "WARNING: vm.max_map_count is too low ($current_max_map_count)"
|
||||
echo "Elasticsearch requires at least 262144"
|
||||
echo "Please run: sudo sysctl -w vm.max_map_count=262144"
|
||||
echo "And add to /etc/sysctl.conf to persist"
|
||||
_die "System configuration needs adjustment"
|
||||
fi
|
||||
|
||||
# Check available memory
|
||||
available_mem=$(free -m | awk '/^Mem:/{print $7}')
|
||||
if [ "$available_mem" -lt 3000 ]; then
|
||||
echo "WARNING: Low available memory (${available_mem}MB)"
|
||||
echo "ELK stack requires at least 3-4GB free memory for proper operation"
|
||||
echo "Services may take longer to start or fail to start"
|
||||
echo ""
|
||||
fi
|
||||
docker compose version >/dev/null 2>&1 || _die "Docker Compose V2 is required"
|
||||
|
||||
# Stop any existing containers
|
||||
bash ./stop.sh || true
|
||||
bash ./stop.sh 2>/dev/null || true
|
||||
|
||||
# Remove old containers
|
||||
docker compose down --remove-orphans 2>/dev/null || true
|
||||
# Create config directory
|
||||
mkdir -p "${CONFIG_PATH}/dashboards"
|
||||
|
||||
# Pull the Docker images
|
||||
echo "Pulling ELK stack images..."
|
||||
docker pull docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION} || _die "Failed to pull Elasticsearch"
|
||||
docker pull docker.elastic.co/logstash/logstash:${LS_VERSION} || _die "Failed to pull Logstash"
|
||||
docker pull docker.elastic.co/kibana/kibana:${KIBANA_VERSION} || _die "Failed to pull Kibana"
|
||||
# Copy configuration files
|
||||
cp "$SCRIPT_DIR/config/"*.yaml "$SCRIPT_DIR/config/"*.yml "$SCRIPT_DIR/config/"*.conf "${CONFIG_PATH}/" 2>/dev/null || true
|
||||
cp "$SCRIPT_DIR/config/dashboards/"*.json "${CONFIG_PATH}/dashboards/" 2>/dev/null || true
|
||||
|
||||
# Create volumes using common function
|
||||
source "$SCRIPT_DIR/_volumes.sh"
|
||||
echo "Creating volumes..."
|
||||
create_items $(get_logserver_volumes)
|
||||
|
||||
# Ensure config directory exists
|
||||
mkdir -p "${CONFIG_PATH}"
|
||||
|
||||
# Initialize API keys file if it doesn't exist
|
||||
if [ ! -f "${CONFIG_PATH}/api-keys.yml" ]; then
|
||||
echo "No API keys configured yet."
|
||||
echo "Run ./generate-api-key.sh to add client keys"
|
||||
echo "api_keys:" > "${CONFIG_PATH}/api-keys.yml"
|
||||
# Generate htpasswd file for Loki authentication
|
||||
echo "Generating authentication file..."
|
||||
# Use openssl to generate htpasswd (available on most systems)
|
||||
if command -v openssl >/dev/null 2>&1; then
|
||||
# Generate password hash
|
||||
PASS_HASH=$(openssl passwd -apr1 "${LOKI_PASSWORD:-changeme}")
|
||||
echo "${LOKI_USER:-logclient}:$PASS_HASH" > "${CONFIG_PATH}/.htpasswd"
|
||||
elif command -v htpasswd >/dev/null 2>&1; then
|
||||
# Use htpasswd if available
|
||||
htpasswd -cb "${CONFIG_PATH}/.htpasswd" "${LOKI_USER:-logclient}" "${LOKI_PASSWORD:-changeme}"
|
||||
else
|
||||
echo "WARNING: Cannot generate password file - no openssl or htpasswd found"
|
||||
echo "Using basic auth with plain text (NOT SECURE):"
|
||||
echo "${LOKI_USER:-logclient}:${LOKI_PASSWORD:-changeme}" > "${CONFIG_PATH}/.htpasswd"
|
||||
fi
|
||||
|
||||
# Copy Logstash configurations if they don't exist
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# Start the stack
|
||||
echo "Starting Log Server..."
|
||||
docker compose up -d || _die "Failed to start"
|
||||
|
||||
if [ ! -f "${CONFIG_PATH}/logstash.conf" ]; then
|
||||
if [ -f "$SCRIPT_DIR/config/logstash.conf" ]; then
|
||||
cp "$SCRIPT_DIR/config/logstash.conf" "${CONFIG_PATH}/logstash.conf"
|
||||
echo "Copied Logstash pipeline configuration to ${CONFIG_PATH}"
|
||||
else
|
||||
echo "WARNING: logstash.conf not found in template"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ ! -f "${CONFIG_PATH}/logstash.yml" ]; then
|
||||
if [ -f "$SCRIPT_DIR/config/logstash.yml" ]; then
|
||||
cp "$SCRIPT_DIR/config/logstash.yml" "${CONFIG_PATH}/logstash.yml"
|
||||
echo "Copied Logstash settings to ${CONFIG_PATH}"
|
||||
else
|
||||
echo "WARNING: logstash.yml not found in template"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Start the ELK stack
|
||||
echo "Starting ELK stack..."
|
||||
docker compose up -d --build || _die "Failed to start ELK stack"
|
||||
|
||||
# Wait for services to be ready with polling
|
||||
echo "Waiting for services to start (this can take 2-3 minutes on first run)..."
|
||||
MAX_WAIT=240 # Maximum 4 minutes
|
||||
# Wait for services
|
||||
echo -n "Waiting for services to start..."
|
||||
MAX_WAIT=60
|
||||
WAITED=0
|
||||
while [ $WAITED -lt $MAX_WAIT ]; do
|
||||
# Check if all services are running
|
||||
if bash ./status.sh 2>/dev/null | grep -q "Running"; then
|
||||
echo " All services are up!"
|
||||
break
|
||||
fi
|
||||
|
||||
# Show progress with time elapsed
|
||||
if [ $((WAITED % 10)) -eq 0 ] && [ $WAITED -gt 0 ]; then
|
||||
echo -n " [${WAITED}s]"
|
||||
else
|
||||
echo -n "."
|
||||
fi
|
||||
|
||||
sleep 2
|
||||
WAITED=$((WAITED + 2))
|
||||
done
|
||||
echo ""
|
||||
|
||||
if [ $WAITED -ge $MAX_WAIT ]; then
|
||||
echo "Warning: Services are still starting. This is normal on first run."
|
||||
echo "Current status:"
|
||||
bash ./status.sh || true
|
||||
echo ""
|
||||
echo "You can check the logs with:"
|
||||
echo " docker logs ${CONTAINER_NAME}_elasticsearch"
|
||||
echo " docker logs ${CONTAINER_NAME}_logstash"
|
||||
echo " docker logs ${CONTAINER_NAME}_kibana"
|
||||
echo ""
|
||||
echo "The services will continue starting in the background."
|
||||
fi
|
||||
|
||||
# Create custom user
|
||||
echo "Setting up custom user '${KIBANA_USERNAME:-admin}'..."
|
||||
echo -n "Waiting for Elasticsearch API..."
|
||||
|
||||
# First wait for Elasticsearch to be ready
|
||||
WAITED=0
|
||||
while [ $WAITED -lt 60 ]; do
|
||||
if docker exec ${CONTAINER_NAME}_elasticsearch curl -s -u elastic:${ELASTIC_PASSWORD} http://localhost:9200/_cluster/health 2>/dev/null | grep -q '"status":"yellow"\|"status":"green"'; then
|
||||
if curl -s "http://localhost:${WEB_PORT:-3000}/api/health" 2>/dev/null | grep -q "ok"; then
|
||||
echo " Ready!"
|
||||
break
|
||||
fi
|
||||
@@ -131,56 +51,28 @@ while [ $WAITED -lt 60 ]; do
|
||||
sleep 2
|
||||
WAITED=$((WAITED + 2))
|
||||
done
|
||||
echo ""
|
||||
|
||||
if [ $WAITED -lt 60 ]; then
|
||||
# Now create the user
|
||||
docker exec ${CONTAINER_NAME}_elasticsearch bash -c "
|
||||
|
||||
result=\$(curl -s -X POST -u elastic:${ELASTIC_PASSWORD} \
|
||||
-H 'Content-Type: application/json' \
|
||||
http://localhost:9200/_security/user/${KIBANA_USERNAME:-admin} \
|
||||
-d '{
|
||||
\"password\" : \"${KIBANA_USER_PASSWORD:-changeme}\",
|
||||
\"roles\" : [ \"superuser\" ],
|
||||
\"full_name\" : \"Admin User\"
|
||||
}' 2>/dev/null)
|
||||
|
||||
if echo \"\$result\" | grep -q '\"created\":true'; then
|
||||
echo \"User created successfully\"
|
||||
else
|
||||
echo \"User already exists (this is fine)\"
|
||||
fi
|
||||
"
|
||||
else
|
||||
echo "Warning: Elasticsearch API not ready after 60 seconds"
|
||||
fi
|
||||
|
||||
echo "Installation of ${CONTAINER_NAME} complete"
|
||||
echo ""
|
||||
echo "========================================="
|
||||
echo "Kibana UI: ${SERVER_PUBLICBASEURL:-http://$(hostname -I | awk '{print $1}'):${KIBANA_PORT}}"
|
||||
echo ""
|
||||
echo "Login with your custom user:"
|
||||
echo " Username: ${KIBANA_USERNAME:-admin}"
|
||||
echo " Password: ${KIBANA_USER_PASSWORD:-changeme}"
|
||||
echo ""
|
||||
echo "Or the superuser:"
|
||||
echo " Username: elastic"
|
||||
echo " Password: ${ELASTIC_PASSWORD:-changeme}"
|
||||
echo "Log Server Installed!"
|
||||
echo "========================================="
|
||||
echo ""
|
||||
# Only show reminder if using default values
|
||||
if [ "${ELASTIC_PASSWORD}" = "changeme" ] || [ -z "${SERVER_PUBLICBASEURL}" ] || [ "${SERVER_PUBLICBASEURL}" = "http://localhost:5601" ]; then
|
||||
echo "REMINDER: Update service.env with:"
|
||||
if [ "${ELASTIC_PASSWORD}" = "changeme" ]; then
|
||||
echo " - A secure password in ELASTIC_PASSWORD"
|
||||
fi
|
||||
if [ -z "${SERVER_PUBLICBASEURL}" ] || [ "${SERVER_PUBLICBASEURL}" = "http://localhost:5601" ]; then
|
||||
echo " - Your actual server IP/domain in SERVER_PUBLICBASEURL"
|
||||
fi
|
||||
echo ""
|
||||
fi
|
||||
echo "Logstash listening on port ${LOGSTASH_BEATS_PORT} for Filebeat clients"
|
||||
echo "Web UI: http://$(hostname -I | awk '{print $1}'):${WEB_PORT:-3000}"
|
||||
echo "Login: ${ADMIN_USER:-admin} / ${ADMIN_PASSWORD:-changeme}"
|
||||
echo ""
|
||||
echo "To add client authentication:"
|
||||
echo " ./generate-api-key.sh"
|
||||
echo "TO VIEW LOGS:"
|
||||
echo "1. Click 'Dashboards' (4 squares icon)"
|
||||
echo "2. Click 'Central Logs'"
|
||||
echo "3. See all logs from all servers!"
|
||||
echo ""
|
||||
echo "FOR CLIENTS TO SEND LOGS HERE:"
|
||||
echo "Server: $(hostname -I | awk '{print $1}')"
|
||||
echo "Port: ${LOKI_PORT:-3100}"
|
||||
echo "Username: ${LOKI_USER:-logclient}"
|
||||
echo "Password: ${LOKI_PASSWORD:-changeme}"
|
||||
echo ""
|
||||
echo "IMPORTANT: Change LOKI_PASSWORD in service.env!"
|
||||
echo ""
|
||||
echo "Install 'logclient' on other servers"
|
||||
echo "========================================="
|
||||
@@ -1,56 +0,0 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
||||
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
if [ -z "$1" ]; then
|
||||
echo "Usage: $0 <backup_directory>"
|
||||
echo "Available backups:"
|
||||
ls -la "${CONFIG_PATH}/backups/" 2>/dev/null || echo "No backups found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
BACKUP_DIR="$1"
|
||||
|
||||
if [ ! -d "$BACKUP_DIR" ]; then
|
||||
_die "Backup directory not found: $BACKUP_DIR"
|
||||
fi
|
||||
|
||||
echo "Restoring ELK stack from backup: $BACKUP_DIR"
|
||||
echo "WARNING: This will overwrite all current data!"
|
||||
|
||||
# Stop the containers
|
||||
bash ./stop.sh || true
|
||||
|
||||
# Restore volumes
|
||||
for volume in $(get_logserver_volumes); do
|
||||
volume_name=$(echo $volume | cut -d: -f3)
|
||||
backup_file="$BACKUP_DIR/${volume_name}.tar.gz"
|
||||
|
||||
if [ -f "$backup_file" ]; then
|
||||
echo "Restoring volume: $volume_name"
|
||||
# Clear existing data and restore
|
||||
docker run --rm -v "$volume_name:/target" -v "$BACKUP_DIR:/backup:ro" alpine \
|
||||
sh -c "rm -rf /target/* && tar -xzf /backup/${volume_name}.tar.gz -C /target"
|
||||
else
|
||||
echo "Warning: Backup file not found for $volume_name"
|
||||
fi
|
||||
done
|
||||
|
||||
# Restore configuration if exists
|
||||
if [ -d "$BACKUP_DIR/config_backup" ]; then
|
||||
cp -r "$BACKUP_DIR/config_backup/"* "${CONFIG_PATH}/"
|
||||
fi
|
||||
|
||||
# Restore docker-compose.yml if exists
|
||||
if [ -f "$BACKUP_DIR/docker-compose.yml" ]; then
|
||||
cp "$BACKUP_DIR/docker-compose.yml" .
|
||||
fi
|
||||
|
||||
echo "Restore completed"
|
||||
|
||||
# Start the containers
|
||||
bash ./start.sh
|
||||
|
||||
echo "ELK stack restarted with restored data"
|
||||
@@ -1,33 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Script to create a custom user for Kibana login
|
||||
# This runs after Elasticsearch is up and creates the user defined in service.env
|
||||
|
||||
# Wait for Elasticsearch to be ready
|
||||
echo "Waiting for Elasticsearch to be ready..."
|
||||
until curl -s -u elastic:${ELASTIC_PASSWORD} http://localhost:9200/_cluster/health | grep -q '"status":"yellow"\|"status":"green"'; do
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo "Creating user '${KIBANA_USERNAME}'..."
|
||||
|
||||
# Create the custom user with superuser role
|
||||
curl -X POST -u elastic:${ELASTIC_PASSWORD} \
|
||||
-H "Content-Type: application/json" \
|
||||
http://localhost:9200/_security/user/${KIBANA_USERNAME} \
|
||||
-d '{
|
||||
"password" : "'${KIBANA_USER_PASSWORD}'",
|
||||
"roles" : [ "superuser" ],
|
||||
"full_name" : "Admin User",
|
||||
"email" : "admin@example.com"
|
||||
}'
|
||||
|
||||
if [ $? -eq 0 ]; then
|
||||
echo ""
|
||||
echo "User '${KIBANA_USERNAME}' created successfully!"
|
||||
echo "You can now log in to Kibana with:"
|
||||
echo " Username: ${KIBANA_USERNAME}"
|
||||
echo " Password: ${KIBANA_USER_PASSWORD}"
|
||||
else
|
||||
echo "Note: User might already exist or there was an error"
|
||||
fi
|
||||
@@ -1,108 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Setup Kibana for simple log viewing
|
||||
# This creates index patterns and saved searches for easy log access
|
||||
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME" "ELASTIC_PASSWORD" "KIBANA_PORT"
|
||||
|
||||
KIBANA_URL="http://localhost:${KIBANA_PORT}"
|
||||
AUTH="elastic:${ELASTIC_PASSWORD}"
|
||||
|
||||
echo "Setting up Kibana for simple log viewing..."
|
||||
echo ""
|
||||
|
||||
# Wait for Kibana to be ready
|
||||
echo -n "Waiting for Kibana to be ready..."
|
||||
MAX_WAIT=60
|
||||
WAITED=0
|
||||
while [ $WAITED -lt $MAX_WAIT ]; do
|
||||
if docker exec ${CONTAINER_NAME}_kibana curl -s -u "$AUTH" "${KIBANA_URL}/api/status" 2>/dev/null | grep -q '"level":"available"'; then
|
||||
echo " Ready!"
|
||||
break
|
||||
fi
|
||||
echo -n "."
|
||||
sleep 2
|
||||
WAITED=$((WAITED + 2))
|
||||
done
|
||||
|
||||
if [ $WAITED -ge $MAX_WAIT ]; then
|
||||
echo ""
|
||||
echo "ERROR: Kibana is not ready after ${MAX_WAIT} seconds"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Create index pattern for Filebeat
|
||||
echo "Creating Filebeat index pattern..."
|
||||
docker exec ${CONTAINER_NAME}_kibana curl -s -X POST \
|
||||
-u "$AUTH" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "kbn-xsrf: true" \
|
||||
"${KIBANA_URL}/api/saved_objects/index-pattern/filebeat-*" \
|
||||
-d '{
|
||||
"attributes": {
|
||||
"title": "filebeat-*",
|
||||
"timeFieldName": "@timestamp",
|
||||
"fields": "[]"
|
||||
}
|
||||
}' > /dev/null 2>&1
|
||||
|
||||
# Set as default index pattern
|
||||
docker exec ${CONTAINER_NAME}_kibana curl -s -X POST \
|
||||
-u "$AUTH" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "kbn-xsrf: true" \
|
||||
"${KIBANA_URL}/api/kibana/settings" \
|
||||
-d '{
|
||||
"changes": {
|
||||
"defaultIndex": "filebeat-*"
|
||||
}
|
||||
}' > /dev/null 2>&1
|
||||
|
||||
# Create a simple saved search for Docker logs
|
||||
echo "Creating saved searches..."
|
||||
docker exec ${CONTAINER_NAME}_kibana curl -s -X POST \
|
||||
-u "$AUTH" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "kbn-xsrf: true" \
|
||||
"${KIBANA_URL}/api/saved_objects/search" \
|
||||
-d '{
|
||||
"attributes": {
|
||||
"title": "Docker Container Logs",
|
||||
"description": "View all Docker container logs",
|
||||
"columns": ["container_name", "message"],
|
||||
"sort": ["@timestamp", "desc"],
|
||||
"kibanaSavedObjectMeta": {
|
||||
"searchSourceJSON": "{\"index\":\"filebeat-*\",\"query\":{\"match_all\":{}},\"filter\":[{\"meta\":{\"index\":\"filebeat-*\",\"negate\":false,\"disabled\":false,\"alias\":null,\"type\":\"exists\",\"key\":\"container_name\",\"value\":\"exists\"},\"exists\":{\"field\":\"container_name\"}}]}"
|
||||
}
|
||||
}
|
||||
}' > /dev/null 2>&1
|
||||
|
||||
echo ""
|
||||
echo "========================================="
|
||||
echo "Kibana Setup Complete!"
|
||||
echo "========================================="
|
||||
echo ""
|
||||
echo "QUICK START GUIDE:"
|
||||
echo ""
|
||||
echo "1. Open Kibana: ${SERVER_PUBLICBASEURL:-http://$(hostname -I | awk '{print $1}'):${KIBANA_PORT}}"
|
||||
echo ""
|
||||
echo "2. Login with:"
|
||||
echo " Username: ${KIBANA_USERNAME:-elastic}"
|
||||
echo " Password: [your password]"
|
||||
echo ""
|
||||
echo "3. TO VIEW LOGS SIMPLY:"
|
||||
echo " a) Click 'Discover' in the left menu"
|
||||
echo " b) Time range is in top-right (set to 'Last 15 minutes' or 'Today')"
|
||||
echo " c) Your logs will appear below"
|
||||
echo ""
|
||||
echo "4. TO FILTER LOGS:"
|
||||
echo " - By container: Click '+' next to any 'container_name' value"
|
||||
echo " - By host: Click '+' next to any 'host.name' value"
|
||||
echo " - Search box: Type keywords to search all logs"
|
||||
echo ""
|
||||
echo "5. TO VIEW LIVE LOGS:"
|
||||
echo " - Click the 'Refresh' button in top-right"
|
||||
echo " - Set it to refresh every 5 seconds"
|
||||
echo ""
|
||||
echo "========================================="
|
||||
@@ -2,16 +2,5 @@
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
echo "Starting ELK stack..."
|
||||
docker compose up -d || _die "Failed to start ELK stack"
|
||||
|
||||
# Wait for services to be ready
|
||||
echo "Waiting for services to start..."
|
||||
sleep 5
|
||||
|
||||
# Check if services are running
|
||||
if docker compose ps | grep -q "Up"; then
|
||||
echo "ELK stack started successfully"
|
||||
else
|
||||
_die "Failed to start ELK stack services"
|
||||
fi
|
||||
docker compose up -d || _die "Failed to start"
|
||||
echo "Log Server started"
|
||||
@@ -2,38 +2,11 @@
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
# For verbose mode (when called directly, not from install script)
|
||||
if [ "$1" = "-v" ] || [ "$1" = "--verbose" ]; then
|
||||
VERBOSE=true
|
||||
else
|
||||
VERBOSE=false
|
||||
fi
|
||||
|
||||
# Check if containers exist
|
||||
if ! docker ps -a --format "{{.Names}}" | grep -q "^${CONTAINER_NAME}_"; then
|
||||
echo "Unknown"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Check individual service status using docker directly (more reliable)
|
||||
es_running=$(docker ps --format "{{.Names}}" | grep -c "^${CONTAINER_NAME}_elasticsearch$" || echo 0)
|
||||
ls_running=$(docker ps --format "{{.Names}}" | grep -c "^${CONTAINER_NAME}_logstash$" || echo 0)
|
||||
kb_running=$(docker ps --format "{{.Names}}" | grep -c "^${CONTAINER_NAME}_kibana$" || echo 0)
|
||||
|
||||
if [ "$VERBOSE" = "true" ]; then
|
||||
echo "Elasticsearch: $([ $es_running -eq 1 ] && echo "Running" || echo "Not running")"
|
||||
echo "Logstash: $([ $ls_running -eq 1 ] && echo "Running" || echo "Not running")"
|
||||
echo "Kibana: $([ $kb_running -eq 1 ] && echo "Running" || echo "Not running")"
|
||||
fi
|
||||
|
||||
# Return overall status
|
||||
if [ "$es_running" -eq 1 ] && [ "$ls_running" -eq 1 ] && [ "$kb_running" -eq 1 ]; then
|
||||
if docker ps | grep -q "${CONTAINER_NAME}_grafana"; then
|
||||
echo "Running"
|
||||
elif [ "$es_running" -eq 0 ] && [ "$ls_running" -eq 0 ] && [ "$kb_running" -eq 0 ]; then
|
||||
echo "Stopped"
|
||||
echo " Web UI: http://$(hostname -I | awk '{print $1}'):${WEB_PORT:-3000}"
|
||||
echo " Loki API: http://$(hostname -I | awk '{print $1}'):${LOKI_PORT:-3100}"
|
||||
docker ps --format "table {{.Names}}\t{{.Status}}" | grep "${CONTAINER_NAME}_"
|
||||
else
|
||||
# At least one service is having issues
|
||||
if [ "$VERBOSE" = "false" ]; then
|
||||
echo "Starting" # More accurate than "Error" during startup
|
||||
fi
|
||||
echo "Stopped"
|
||||
fi
|
||||
@@ -2,7 +2,5 @@
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
echo "Stopping ELK stack..."
|
||||
docker compose stop || true
|
||||
|
||||
echo "ELK stack stopped"
|
||||
docker compose stop || _die "Failed to stop"
|
||||
echo "Log Server stopped"
|
||||
@@ -1,54 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Test script to verify API key authentication is working
|
||||
# This sends test log messages with and without API keys to verify authentication
|
||||
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME" "LOGSTASH_BEATS_PORT"
|
||||
|
||||
echo "=== Testing Logstash API Key Authentication ==="
|
||||
echo ""
|
||||
|
||||
# Check if service is running
|
||||
if ! bash ./status.sh | grep -q "Running"; then
|
||||
echo "Error: LogServer is not running. Please start it first."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Get server IP
|
||||
SERVER_IP=$(hostname -I | awk '{print $1}')
|
||||
|
||||
echo "Testing authentication on ${SERVER_IP}:${LOGSTASH_BEATS_PORT}"
|
||||
echo ""
|
||||
|
||||
# Test 1: Send without API key (should be tagged or dropped based on config)
|
||||
echo "Test 1: Sending log without API key..."
|
||||
echo '{"message":"Test log without API key","timestamp":"'$(date -Iseconds)'"}' | \
|
||||
nc -w 2 ${SERVER_IP} ${LOGSTASH_BEATS_PORT} 2>/dev/null || true
|
||||
|
||||
# Test 2: Send with invalid API key (should be dropped)
|
||||
echo "Test 2: Sending log with invalid API key..."
|
||||
echo '{"message":"Test log with invalid key","fields":{"api_key":"invalid-key-12345"},"timestamp":"'$(date -Iseconds)'"}' | \
|
||||
nc -w 2 ${SERVER_IP} ${LOGSTASH_BEATS_PORT} 2>/dev/null || true
|
||||
|
||||
# Test 3: Send with valid API key (if one exists)
|
||||
if [ -f "${CONFIG_PATH}/api-keys.yml" ]; then
|
||||
# Extract first API key from the file
|
||||
API_KEY=$(grep -E "^ [^:]+: " "${CONFIG_PATH}/api-keys.yml" | head -1 | awk '{print $2}')
|
||||
if [ -n "$API_KEY" ]; then
|
||||
echo "Test 3: Sending log with valid API key..."
|
||||
echo '{"message":"Test log with valid key","fields":{"api_key":"'${API_KEY}'"},"timestamp":"'$(date -Iseconds)'"}' | \
|
||||
nc -w 2 ${SERVER_IP} ${LOGSTASH_BEATS_PORT} 2>/dev/null || true
|
||||
else
|
||||
echo "Test 3: Skipped - no API keys found"
|
||||
fi
|
||||
else
|
||||
echo "Test 3: Skipped - api-keys.yml not found"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Tests sent. Check Logstash logs to verify authentication:"
|
||||
echo " docker logs ${CONTAINER_NAME}_logstash --tail 20"
|
||||
echo ""
|
||||
echo "Valid logs should appear in Elasticsearch."
|
||||
echo "Invalid logs should be dropped."
|
||||
@@ -2,15 +2,5 @@
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
# Stop the containers
|
||||
bash ./stop.sh || _die "Failed to stop containers"
|
||||
|
||||
# Remove the containers
|
||||
docker compose down --remove-orphans || _die "Failed to remove containers"
|
||||
|
||||
# CRITICAL: Never remove data volumes in uninstall.sh!
|
||||
# Data volumes must be preserved for potential reinstallation
|
||||
# Only destroy.sh should remove volumes, and it must be explicit
|
||||
|
||||
echo "Uninstallation of ${CONTAINER_NAME} complete"
|
||||
echo "Note: Data volumes have been preserved. To remove all data, use destroy.sh"
|
||||
docker compose down || true
|
||||
echo "Log Server uninstalled (data preserved)"
|
||||
61
simple-logs/README.md
Normal file
61
simple-logs/README.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# Simple Logs
|
||||
|
||||
A dead-simple log viewer using Grafana + Loki. Much easier than ELK/Kibana!
|
||||
|
||||
## Features
|
||||
- **ALL logs in one place** - Docker containers + system logs
|
||||
- **Simple interface** - Just one dashboard, no complex setup
|
||||
- **Fast & lightweight** - Uses 10x less resources than ELK
|
||||
- **Real-time** - Logs appear instantly
|
||||
- **No configuration** - Works out of the box
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. **Install**
|
||||
```bash
|
||||
dropshell install simple-logs
|
||||
```
|
||||
|
||||
2. **Access**
|
||||
- Go to: `http://<server-ip>:3000`
|
||||
- Login: `admin` / `admin` (change on first login)
|
||||
|
||||
3. **View Logs**
|
||||
- Click the **Dashboards** icon (4 squares) on left
|
||||
- Click **"Simple Logs"**
|
||||
- That's it! All your logs are there
|
||||
|
||||
## How to Use
|
||||
|
||||
### Filter by Container
|
||||
Click any `container_name` label to see only that container's logs
|
||||
|
||||
### Search Text
|
||||
Use the search box at the top to find specific text
|
||||
|
||||
### Change Time Range
|
||||
Top-right corner - set to "Last 5 minutes" for recent logs
|
||||
|
||||
### View Details
|
||||
Click any log line to expand and see full details
|
||||
|
||||
## What Gets Logged
|
||||
- All Docker container logs
|
||||
- System logs (/var/log/syslog)
|
||||
- Authentication logs (/var/log/auth.log)
|
||||
|
||||
## Ports
|
||||
- `3000` - Grafana Web UI
|
||||
- `3100` - Loki API (internal)
|
||||
|
||||
## Why This Instead of ELK?
|
||||
- **10x simpler** - One dashboard vs Kibana's complexity
|
||||
- **10x smaller** - ~200MB RAM vs 2-3GB for ELK
|
||||
- **Instant setup** - No index patterns, no field mappings
|
||||
- **Just works** - No configuration needed
|
||||
|
||||
## Advanced: Custom Queries
|
||||
In the logs panel, you can use LogQL queries:
|
||||
- `{container_name="myapp"}` - Specific container
|
||||
- `{job="syslog"} |= "error"` - System errors
|
||||
- `{job="docker"} |~ "(?i)error|warn"` - All Docker warnings/errors
|
||||
69
simple-logs/config/dashboards/simple-logs.json
Normal file
69
simple-logs/config/dashboards/simple-logs.json
Normal file
@@ -0,0 +1,69 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": "-- Grafana --",
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"gnetId": null,
|
||||
"graphTooltip": 0,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": "Loki",
|
||||
"fieldConfig": {
|
||||
"defaults": {},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 20,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "{job=~\"docker|syslog|auth\"}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "All Logs",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"schemaVersion": 27,
|
||||
"style": "dark",
|
||||
"tags": ["logs"],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-15m",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "",
|
||||
"title": "Simple Logs",
|
||||
"uid": "simple-logs",
|
||||
"version": 0
|
||||
}
|
||||
12
simple-logs/config/grafana-dashboard.yml
Normal file
12
simple-logs/config/grafana-dashboard.yml
Normal file
@@ -0,0 +1,12 @@
|
||||
apiVersion: 1
|
||||
|
||||
providers:
|
||||
- name: 'default'
|
||||
orgId: 1
|
||||
folder: ''
|
||||
type: file
|
||||
disableDeletion: false
|
||||
updateIntervalSeconds: 10
|
||||
allowUiUpdates: true
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
9
simple-logs/config/grafana-datasources.yml
Normal file
9
simple-logs/config/grafana-datasources.yml
Normal file
@@ -0,0 +1,9 @@
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
isDefault: true
|
||||
editable: false
|
||||
36
simple-logs/config/loki.yaml
Normal file
36
simple-logs/config/loki.yaml
Normal file
@@ -0,0 +1,36 @@
|
||||
auth_enabled: false
|
||||
|
||||
server:
|
||||
http_listen_port: 3100
|
||||
grpc_listen_port: 9096
|
||||
|
||||
common:
|
||||
path_prefix: /loki
|
||||
storage:
|
||||
filesystem:
|
||||
chunks_directory: /loki/chunks
|
||||
rules_directory: /loki/rules
|
||||
replication_factor: 1
|
||||
ring:
|
||||
instance_addr: 127.0.0.1
|
||||
kvstore:
|
||||
store: inmemory
|
||||
|
||||
schema_config:
|
||||
configs:
|
||||
- from: 2020-10-24
|
||||
store: boltdb-shipper
|
||||
object_store: filesystem
|
||||
schema: v11
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
|
||||
ruler:
|
||||
alertmanager_url: http://localhost:9093
|
||||
|
||||
limits_config:
|
||||
retention_period: 168h # 7 days
|
||||
enforce_metric_name: false
|
||||
reject_old_samples: true
|
||||
reject_old_samples_max_age: 168h
|
||||
67
simple-logs/config/promtail.yaml
Normal file
67
simple-logs/config/promtail.yaml
Normal file
@@ -0,0 +1,67 @@
|
||||
server:
|
||||
http_listen_port: 9080
|
||||
grpc_listen_port: 0
|
||||
|
||||
positions:
|
||||
filename: /tmp/positions/positions.yaml
|
||||
|
||||
clients:
|
||||
- url: http://loki:3100/loki/api/v1/push
|
||||
|
||||
scrape_configs:
|
||||
# Docker container logs
|
||||
- job_name: docker
|
||||
static_configs:
|
||||
- targets:
|
||||
- localhost
|
||||
labels:
|
||||
job: docker
|
||||
__path__: /var/lib/docker/containers/*/*-json.log
|
||||
pipeline_stages:
|
||||
- json:
|
||||
expressions:
|
||||
stream: stream
|
||||
time: time
|
||||
log: log
|
||||
- timestamp:
|
||||
source: time
|
||||
format: RFC3339Nano
|
||||
- labels:
|
||||
stream:
|
||||
- regex:
|
||||
expression: '(?P<container_id>[0-9a-f]{12})'
|
||||
source: filename
|
||||
- labels:
|
||||
container_id:
|
||||
|
||||
# System logs
|
||||
- job_name: syslog
|
||||
static_configs:
|
||||
- targets:
|
||||
- localhost
|
||||
labels:
|
||||
job: syslog
|
||||
__path__: /var/log/syslog
|
||||
host: ${HOSTNAME}
|
||||
|
||||
- job_name: auth
|
||||
static_configs:
|
||||
- targets:
|
||||
- localhost
|
||||
labels:
|
||||
job: auth
|
||||
__path__: /var/log/auth.log
|
||||
host: ${HOSTNAME}
|
||||
|
||||
# Docker events via Docker socket
|
||||
- job_name: docker_events
|
||||
docker_sd_configs:
|
||||
- host: unix:///var/run/docker.sock
|
||||
refresh_interval: 5s
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_docker_container_name]
|
||||
target_label: container_name
|
||||
- source_labels: [__meta_docker_container_id]
|
||||
target_label: container_id
|
||||
- source_labels: [__meta_docker_container_image]
|
||||
target_label: image
|
||||
15
simple-logs/config/service.env
Normal file
15
simple-logs/config/service.env
Normal file
@@ -0,0 +1,15 @@
|
||||
# Simple Log Viewer Configuration
|
||||
CONTAINER_NAME=simple-logs
|
||||
|
||||
# Server settings (REQUIRED by dropshell)
|
||||
SSH_USER="root"
|
||||
|
||||
# Web UI Port
|
||||
WEB_PORT=3000
|
||||
|
||||
# Log retention (days)
|
||||
LOG_RETENTION=7
|
||||
|
||||
# Authentication (optional - leave empty for no auth)
|
||||
AUTH_USERNAME=""
|
||||
AUTH_PASSWORD=""
|
||||
54
simple-logs/docker-compose.yml
Normal file
54
simple-logs/docker-compose.yml
Normal file
@@ -0,0 +1,54 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
# Grafana - Simple Web UI for viewing logs
|
||||
grafana:
|
||||
image: grafana/grafana:latest
|
||||
container_name: ${CONTAINER_NAME}_grafana
|
||||
environment:
|
||||
- GF_SECURITY_ADMIN_USER=${AUTH_USERNAME:-admin}
|
||||
- GF_SECURITY_ADMIN_PASSWORD=${AUTH_PASSWORD:-admin}
|
||||
- GF_AUTH_ANONYMOUS_ENABLED=${AUTH_USERNAME:+false}
|
||||
- GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer
|
||||
- GF_INSTALL_PLUGINS=
|
||||
volumes:
|
||||
- grafana_data:/var/lib/grafana
|
||||
- ${CONFIG_PATH}/grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yaml:ro
|
||||
- ${CONFIG_PATH}/grafana-dashboard.yml:/etc/grafana/provisioning/dashboards/dashboard.yaml:ro
|
||||
- ${CONFIG_PATH}/dashboards:/var/lib/grafana/dashboards:ro
|
||||
ports:
|
||||
- "${WEB_PORT:-3000}:3000"
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- loki
|
||||
|
||||
# Loki - Lightweight log storage (100x simpler than Elasticsearch)
|
||||
loki:
|
||||
image: grafana/loki:2.9.0
|
||||
container_name: ${CONTAINER_NAME}_loki
|
||||
volumes:
|
||||
- loki_data:/loki
|
||||
- ${CONFIG_PATH}/loki.yaml:/etc/loki/local-config.yaml:ro
|
||||
ports:
|
||||
- "3100:3100"
|
||||
command: -config.file=/etc/loki/local-config.yaml
|
||||
restart: unless-stopped
|
||||
|
||||
# Promtail - Collects all logs (Docker + System)
|
||||
promtail:
|
||||
image: grafana/promtail:2.9.0
|
||||
container_name: ${CONTAINER_NAME}_promtail
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- /var/log:/var/log:ro
|
||||
- ${CONFIG_PATH}/promtail.yaml:/etc/promtail/config.yml:ro
|
||||
- promtail_positions:/tmp/positions
|
||||
command: -config.file=/etc/promtail/config.yml
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- loki
|
||||
|
||||
volumes:
|
||||
grafana_data:
|
||||
loki_data:
|
||||
promtail_positions:
|
||||
62
simple-logs/install.sh
Executable file
62
simple-logs/install.sh
Executable file
@@ -0,0 +1,62 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
# Check Docker
|
||||
_check_docker_installed || _die "Docker test failed"
|
||||
docker compose version >/dev/null 2>&1 || _die "Docker Compose V2 is required"
|
||||
|
||||
# Stop any existing containers
|
||||
bash ./stop.sh 2>/dev/null || true
|
||||
|
||||
# Create config directory
|
||||
mkdir -p "${CONFIG_PATH}/dashboards"
|
||||
|
||||
# Copy configuration files
|
||||
cp "$SCRIPT_DIR/config/"*.yaml "$SCRIPT_DIR/config/"*.yml "${CONFIG_PATH}/" 2>/dev/null || true
|
||||
cp "$SCRIPT_DIR/config/dashboards/"*.json "${CONFIG_PATH}/dashboards/" 2>/dev/null || true
|
||||
|
||||
# Start the stack
|
||||
echo "Starting Simple Logs stack..."
|
||||
docker compose up -d || _die "Failed to start"
|
||||
|
||||
# Wait for Grafana to be ready
|
||||
echo -n "Waiting for Grafana to start..."
|
||||
MAX_WAIT=60
|
||||
WAITED=0
|
||||
while [ $WAITED -lt $MAX_WAIT ]; do
|
||||
if curl -s "http://localhost:${WEB_PORT:-3000}/api/health" 2>/dev/null | grep -q "ok"; then
|
||||
echo " Ready!"
|
||||
break
|
||||
fi
|
||||
echo -n "."
|
||||
sleep 2
|
||||
WAITED=$((WAITED + 2))
|
||||
done
|
||||
echo ""
|
||||
|
||||
echo "========================================="
|
||||
echo "Simple Logs Installed!"
|
||||
echo "========================================="
|
||||
echo ""
|
||||
echo "Access at: http://$(hostname -I | awk '{print $1}'):${WEB_PORT:-3000}"
|
||||
echo ""
|
||||
if [ -n "${AUTH_USERNAME}" ]; then
|
||||
echo "Login: ${AUTH_USERNAME} / ${AUTH_PASSWORD}"
|
||||
else
|
||||
echo "Login: admin / admin"
|
||||
echo "(You'll be asked to change password on first login)"
|
||||
fi
|
||||
echo ""
|
||||
echo "TO VIEW LOGS:"
|
||||
echo "1. Click 'Dashboards' icon (4 squares) on left"
|
||||
echo "2. Click 'Simple Logs'"
|
||||
echo "3. Your logs appear immediately!"
|
||||
echo ""
|
||||
echo "TO FILTER:"
|
||||
echo "- Click any label to filter by it"
|
||||
echo "- Use the search box for text search"
|
||||
echo "- Time range selector is in top-right"
|
||||
echo "========================================="
|
||||
12
simple-logs/logs.sh
Executable file
12
simple-logs/logs.sh
Executable file
@@ -0,0 +1,12 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
echo "=== Grafana Logs ==="
|
||||
docker logs "${CONTAINER_NAME}_grafana" --tail 20
|
||||
echo ""
|
||||
echo "=== Loki Logs ==="
|
||||
docker logs "${CONTAINER_NAME}_loki" --tail 20
|
||||
echo ""
|
||||
echo "=== Promtail Logs ==="
|
||||
docker logs "${CONTAINER_NAME}_promtail" --tail 20
|
||||
6
simple-logs/start.sh
Executable file
6
simple-logs/start.sh
Executable file
@@ -0,0 +1,6 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
docker compose up -d || _die "Failed to start Simple Logs"
|
||||
echo "Simple Logs started"
|
||||
11
simple-logs/status.sh
Executable file
11
simple-logs/status.sh
Executable file
@@ -0,0 +1,11 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
if docker ps | grep -q "${CONTAINER_NAME}_grafana"; then
|
||||
echo "Running"
|
||||
echo " Grafana: http://$(hostname -I | awk '{print $1}'):${WEB_PORT:-3000}"
|
||||
docker ps --format "table {{.Names}}\t{{.Status}}" | grep "${CONTAINER_NAME}_"
|
||||
else
|
||||
echo "Stopped"
|
||||
fi
|
||||
6
simple-logs/stop.sh
Executable file
6
simple-logs/stop.sh
Executable file
@@ -0,0 +1,6 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
docker compose stop || _die "Failed to stop Simple Logs"
|
||||
echo "Simple Logs stopped"
|
||||
6
simple-logs/uninstall.sh
Executable file
6
simple-logs/uninstall.sh
Executable file
@@ -0,0 +1,6 @@
|
||||
#!/bin/bash
|
||||
source "${AGENT_PATH}/common.sh"
|
||||
_check_required_env_vars "CONTAINER_NAME"
|
||||
|
||||
docker compose down || true
|
||||
echo "Simple Logs uninstalled (data preserved)"
|
||||
Reference in New Issue
Block a user