swtich from ELK to Loki!
All checks were successful
Test and Publish Templates / test-and-publish (push) Successful in 40s
All checks were successful
Test and Publish Templates / test-and-publish (push) Successful in 40s
This commit is contained in:
Your Name
@@ -1,389 +0,0 @@
|
|||||||
# Dropshell LogClient Template
|
|
||||||
|
|
||||||
An auto-configuring Filebeat agent that collects Docker container logs via the Docker API and system logs, shipping them to a centralized logging server with minimal configuration.
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
This template deploys a lightweight Filebeat agent that:
|
|
||||||
- Uses Docker API to collect logs from all containers (regardless of logging driver)
|
|
||||||
- Allows containers to use any Docker logging driver (json-file, local, journald, etc.)
|
|
||||||
- Collects system logs (syslog, auth logs, kernel logs)
|
|
||||||
- Ships logs to a centralized ELK stack (logserver)
|
|
||||||
- Requires minimal configuration - just the server address
|
|
||||||
- Handles connection failures with local buffering
|
|
||||||
- Auto-reconnects when the server becomes available
|
|
||||||
|
|
||||||
## Features
|
|
||||||
|
|
||||||
### Docker API Log Collection
|
|
||||||
- **Direct API access**: Reads logs via Docker API, not from files
|
|
||||||
- **Driver independent**: Works with any Docker logging driver (local, json-file, journald)
|
|
||||||
- **Automatic discovery**: Finds all running containers dynamically
|
|
||||||
- **Container metadata**: Enriches logs with container names, images, labels
|
|
||||||
- **Real-time streaming**: Gets logs as they're generated
|
|
||||||
- **Multi-line handling**: Properly handles stack traces and multi-line logs
|
|
||||||
- **JSON parsing**: Automatically parses JSON-formatted logs
|
|
||||||
- **Label-based filtering**: Can include/exclude containers based on labels
|
|
||||||
|
|
||||||
### System Log Collection
|
|
||||||
- **/var/log/syslog** or **/var/log/messages**: System events
|
|
||||||
- **/var/log/auth.log** or **/var/log/secure**: Authentication events
|
|
||||||
- **/var/log/kern.log**: Kernel messages
|
|
||||||
- **journald**: SystemD journal (if available)
|
|
||||||
- **Custom paths**: Configurable additional log paths
|
|
||||||
|
|
||||||
### Reliability Features
|
|
||||||
- **Local buffering**: Stores logs locally when server is unreachable
|
|
||||||
- **Automatic retry**: Reconnects automatically with exponential backoff
|
|
||||||
- **Compression**: Compresses logs before sending to save bandwidth
|
|
||||||
- **Secure transmission**: Optional TLS/SSL encryption
|
|
||||||
- **Backpressure handling**: Slows down when server is overwhelmed
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
### How It Works
|
|
||||||
1. Filebeat runs as a container with Docker socket access
|
|
||||||
2. Uses Docker API to stream logs from all containers
|
|
||||||
3. Monitors Docker API for container lifecycle events
|
|
||||||
4. Automatically starts collecting logs from new containers
|
|
||||||
5. Reads host system logs from mounted volumes
|
|
||||||
6. Ships all logs to configured Logstash/Elasticsearch endpoint
|
|
||||||
7. Maintains connection state and buffering information
|
|
||||||
|
|
||||||
### Log Flow
|
|
||||||
```
|
|
||||||
Docker Containers → Docker API →
|
|
||||||
↘
|
|
||||||
Filebeat → Logstash → Elasticsearch → Kibana
|
|
||||||
↗
|
|
||||||
System Logs (mounted volumes) →
|
|
||||||
```
|
|
||||||
|
|
||||||
### Why Docker API Instead of Log Files?
|
|
||||||
- **Logging driver flexibility**: Containers can use `local`, `json-file`, `journald`, or any driver
|
|
||||||
- **No log file management**: Don't need to worry about log rotation or file paths
|
|
||||||
- **Better performance**: Direct streaming without file I/O overhead
|
|
||||||
- **Consistent access**: Same method regardless of storage backend
|
|
||||||
- **Real-time streaming**: Get logs immediately as they're generated
|
|
||||||
- **Simplified permissions**: Only need Docker socket access
|
|
||||||
|
|
||||||
## Minimum Configuration
|
|
||||||
|
|
||||||
The template requires minimal configuration - server address and authentication:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Required - Server connection
|
|
||||||
LOGSERVER_HOST=192.168.1.100
|
|
||||||
LOGSERVER_PORT=5044
|
|
||||||
|
|
||||||
# Required - Authentication (choose one method)
|
|
||||||
AUTH_MODE=mtls # Options: mtls, apikey, basic
|
|
||||||
|
|
||||||
# For mTLS authentication
|
|
||||||
CLIENT_CERT_PATH=/certs/client.crt
|
|
||||||
CLIENT_KEY_PATH=/certs/client.key
|
|
||||||
CA_CERT_PATH=/certs/ca.crt
|
|
||||||
|
|
||||||
# For API key authentication
|
|
||||||
API_KEY=your-api-key-here
|
|
||||||
|
|
||||||
# For basic auth (not recommended)
|
|
||||||
USERNAME=filebeat
|
|
||||||
PASSWORD=changeme
|
|
||||||
```
|
|
||||||
|
|
||||||
## Configuration Options
|
|
||||||
|
|
||||||
### Environment Variables (service.env)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# REQUIRED: Log server connection
|
|
||||||
LOGSERVER_HOST=logserver.example.com
|
|
||||||
LOGSERVER_PORT=5044
|
|
||||||
|
|
||||||
# REQUIRED: Authentication method
|
|
||||||
AUTH_MODE=mtls # mtls, apikey, or basic
|
|
||||||
|
|
||||||
# mTLS Authentication (if AUTH_MODE=mtls)
|
|
||||||
CLIENT_CERT_PATH=/certs/${HOSTNAME}.crt
|
|
||||||
CLIENT_KEY_PATH=/certs/${HOSTNAME}.key
|
|
||||||
CA_CERT_PATH=/certs/ca.crt
|
|
||||||
SSL_VERIFICATION_MODE=full
|
|
||||||
|
|
||||||
# API Key Authentication (if AUTH_MODE=apikey)
|
|
||||||
API_KEY="" # Will be provided by logserver admin
|
|
||||||
|
|
||||||
# Basic Authentication (if AUTH_MODE=basic)
|
|
||||||
USERNAME=filebeat
|
|
||||||
PASSWORD=changeme
|
|
||||||
|
|
||||||
# Optional: Performance tuning
|
|
||||||
BULK_MAX_SIZE=2048 # Maximum batch size
|
|
||||||
WORKER_THREADS=1 # Number of worker threads
|
|
||||||
QUEUE_SIZE=4096 # Internal queue size
|
|
||||||
MAX_BACKOFF=60s # Maximum retry backoff
|
|
||||||
|
|
||||||
# Optional: Filtering
|
|
||||||
EXCLUDE_CONTAINERS="" # Comma-separated container names to exclude
|
|
||||||
INCLUDE_CONTAINERS="" # Only include these containers (if set)
|
|
||||||
EXCLUDE_LABELS="" # Exclude containers with these labels
|
|
||||||
INCLUDE_LABELS="" # Only include containers with these labels
|
|
||||||
|
|
||||||
# Optional: Additional log paths
|
|
||||||
CUSTOM_LOG_PATHS="" # Comma-separated additional paths to monitor
|
|
||||||
|
|
||||||
# Optional: Resource limits
|
|
||||||
MAX_CPU=50 # Maximum CPU usage percentage
|
|
||||||
MAX_MEMORY=200MB # Maximum memory usage
|
|
||||||
```
|
|
||||||
|
|
||||||
## Collected Log Types
|
|
||||||
|
|
||||||
### Docker Container Logs (via Docker API)
|
|
||||||
- **stdout/stderr**: All container output regardless of logging driver
|
|
||||||
- **Container metadata**: Name, ID, image, labels
|
|
||||||
- **Docker events**: Start, stop, die, kill events
|
|
||||||
- **Health check results**: If configured
|
|
||||||
- **Works with all logging drivers**: local, json-file, journald, syslog, etc.
|
|
||||||
|
|
||||||
### System Logs
|
|
||||||
- **System messages**: Service starts/stops, errors
|
|
||||||
- **Authentication**: SSH logins, sudo usage
|
|
||||||
- **Kernel messages**: Hardware events, driver messages
|
|
||||||
- **Package management**: apt/yum operations
|
|
||||||
- **Cron jobs**: Scheduled task execution
|
|
||||||
|
|
||||||
## Log Enrichment
|
|
||||||
|
|
||||||
Logs are automatically enriched with:
|
|
||||||
- **Hostname**: Source host identification
|
|
||||||
- **Timestamp**: Precise event time with timezone
|
|
||||||
- **Log level**: Parsed from log content when possible
|
|
||||||
- **Container info**: For Docker logs
|
|
||||||
- **Process info**: PID, command for system logs
|
|
||||||
- **File path**: Source log file
|
|
||||||
|
|
||||||
## Resource Requirements
|
|
||||||
|
|
||||||
### Minimum
|
|
||||||
- CPU: 0.5 cores
|
|
||||||
- RAM: 128MB
|
|
||||||
- Storage: 1GB (for buffer)
|
|
||||||
|
|
||||||
### Typical Usage
|
|
||||||
- CPU: 1-5% of one core
|
|
||||||
- RAM: 150-200MB
|
|
||||||
- Network: Varies with log volume
|
|
||||||
- Storage: Depends on buffer size
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
### Prerequisites
|
|
||||||
1. A running logserver (ELK stack)
|
|
||||||
2. Network connectivity to logserver
|
|
||||||
3. Docker installed on host
|
|
||||||
4. Authentication credentials from logserver admin
|
|
||||||
|
|
||||||
### Setup Authentication
|
|
||||||
|
|
||||||
#### For mTLS (Recommended):
|
|
||||||
```bash
|
|
||||||
# Get client certificate from logserver admin
|
|
||||||
# They will run: dropshell exec logserver /scripts/generate-client-cert.sh $(hostname)
|
|
||||||
# Copy the generated certificate files to this client
|
|
||||||
mkdir -p /etc/dropshell/certs
|
|
||||||
# Copy ca.crt, client.crt, and client.key to /etc/dropshell/certs/
|
|
||||||
```
|
|
||||||
|
|
||||||
#### For API Key:
|
|
||||||
```bash
|
|
||||||
# Get API key from logserver admin
|
|
||||||
# They will run: dropshell exec logserver /scripts/generate-api-key.sh $(hostname)
|
|
||||||
# Add the API key to service.env
|
|
||||||
```
|
|
||||||
|
|
||||||
### Deploy
|
|
||||||
```bash
|
|
||||||
# Configure authentication in service.env
|
|
||||||
dropshell install logclient
|
|
||||||
```
|
|
||||||
|
|
||||||
## Monitoring
|
|
||||||
|
|
||||||
### Check Status
|
|
||||||
```bash
|
|
||||||
dropshell status logclient
|
|
||||||
```
|
|
||||||
|
|
||||||
### View Filebeat Logs
|
|
||||||
```bash
|
|
||||||
dropshell logs logclient
|
|
||||||
```
|
|
||||||
|
|
||||||
### Verify Connectivity
|
|
||||||
```bash
|
|
||||||
# Check if logs are being shipped
|
|
||||||
docker exec logclient-filebeat filebeat test output
|
|
||||||
```
|
|
||||||
|
|
||||||
### Monitor Metrics
|
|
||||||
```bash
|
|
||||||
# View Filebeat statistics
|
|
||||||
docker exec logclient-filebeat curl -s http://localhost:5066/stats
|
|
||||||
```
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### No Logs Appearing on Server
|
|
||||||
|
|
||||||
1. **Check connectivity**
|
|
||||||
```bash
|
|
||||||
telnet $LOGSERVER_HOST $LOGSERVER_PORT
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Verify Filebeat is running**
|
|
||||||
```bash
|
|
||||||
dropshell status logclient
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Check Filebeat logs**
|
|
||||||
```bash
|
|
||||||
dropshell logs logclient | tail -50
|
|
||||||
```
|
|
||||||
|
|
||||||
4. **Test configuration**
|
|
||||||
```bash
|
|
||||||
docker exec logclient-filebeat filebeat test config
|
|
||||||
```
|
|
||||||
|
|
||||||
### High CPU Usage
|
|
||||||
|
|
||||||
1. Reduce worker threads in service.env
|
|
||||||
2. Increase bulk_max_size to send larger batches
|
|
||||||
3. Add exclude filters for noisy containers
|
|
||||||
|
|
||||||
### Missing Container Logs
|
|
||||||
|
|
||||||
1. Verify Docker socket is mounted
|
|
||||||
2. Check container isn't in exclude list
|
|
||||||
3. Ensure Filebeat has permissions to Docker socket
|
|
||||||
4. Verify container is actually producing output
|
|
||||||
5. Check if container uses a supported logging driver
|
|
||||||
|
|
||||||
### Buffer Full Errors
|
|
||||||
|
|
||||||
1. Increase queue_size in service.env
|
|
||||||
2. Check network connectivity to server
|
|
||||||
3. Verify server isn't overwhelmed
|
|
||||||
|
|
||||||
## Security Considerations
|
|
||||||
|
|
||||||
1. **Authentication**:
|
|
||||||
- Always use mTLS or API keys in production
|
|
||||||
- Never use basic auth except for testing
|
|
||||||
- Store credentials securely
|
|
||||||
- Rotate certificates/keys regularly
|
|
||||||
|
|
||||||
2. **Docker Socket Access**:
|
|
||||||
- Requires Docker socket access to read logs via API
|
|
||||||
- Understand security implications of socket access
|
|
||||||
- Consider read-only socket access if possible
|
|
||||||
|
|
||||||
3. **Network Security**:
|
|
||||||
- All connections are TLS encrypted
|
|
||||||
- Verify server certificates
|
|
||||||
- Configure firewall rules appropriately
|
|
||||||
- Use private networks when possible
|
|
||||||
|
|
||||||
4. **Data Protection**:
|
|
||||||
- Logs may contain sensitive data
|
|
||||||
- Filter sensitive information before shipping
|
|
||||||
- Exclude containers with sensitive data if needed
|
|
||||||
|
|
||||||
5. **Resource Limits**:
|
|
||||||
- Set CPU and memory limits
|
|
||||||
- Monitor resource usage
|
|
||||||
- Prevent resource exhaustion attacks
|
|
||||||
|
|
||||||
## Performance Tuning
|
|
||||||
|
|
||||||
### For High-Volume Environments
|
|
||||||
```bash
|
|
||||||
# Increase workers and batch size
|
|
||||||
WORKER_THREADS=4
|
|
||||||
BULK_MAX_SIZE=4096
|
|
||||||
QUEUE_SIZE=8192
|
|
||||||
```
|
|
||||||
|
|
||||||
### For Low-Resource Hosts
|
|
||||||
```bash
|
|
||||||
# Reduce resource usage
|
|
||||||
WORKER_THREADS=1
|
|
||||||
BULK_MAX_SIZE=512
|
|
||||||
MAX_MEMORY=100MB
|
|
||||||
MAX_CPU=25
|
|
||||||
```
|
|
||||||
|
|
||||||
### Network Optimization
|
|
||||||
```bash
|
|
||||||
# Enable compression (CPU vs bandwidth tradeoff)
|
|
||||||
COMPRESSION_LEVEL=3 # 0-9, higher = more compression
|
|
||||||
```
|
|
||||||
|
|
||||||
## Integration with LogServer
|
|
||||||
|
|
||||||
This template is designed to work seamlessly with the `logserver` template:
|
|
||||||
|
|
||||||
1. Deploy logserver first
|
|
||||||
2. Note the logserver's IP/hostname
|
|
||||||
3. Configure logclient with server address
|
|
||||||
4. Logs automatically start flowing
|
|
||||||
|
|
||||||
## Maintenance
|
|
||||||
|
|
||||||
### Regular Tasks
|
|
||||||
- Monitor buffer usage
|
|
||||||
- Check for connection errors
|
|
||||||
- Review excluded/included containers
|
|
||||||
- Update Filebeat version
|
|
||||||
|
|
||||||
### Logs Rotation
|
|
||||||
Filebeat handles log rotation automatically:
|
|
||||||
- Detects renamed/rotated files
|
|
||||||
- Continues reading from correct position
|
|
||||||
- Cleans up old file handles
|
|
||||||
|
|
||||||
## Advanced Configuration
|
|
||||||
|
|
||||||
### Custom Filebeat Configuration
|
|
||||||
Create a custom `filebeat.yml` in the config directory for advanced scenarios:
|
|
||||||
- Custom processors
|
|
||||||
- Additional inputs
|
|
||||||
- Complex filtering rules
|
|
||||||
- Multiple outputs
|
|
||||||
|
|
||||||
### Docker Labels for Control
|
|
||||||
Control log collection per container:
|
|
||||||
```yaml
|
|
||||||
# In docker-compose.yml
|
|
||||||
services:
|
|
||||||
myapp:
|
|
||||||
logging:
|
|
||||||
driver: local # Can use any driver - Filebeat reads via API
|
|
||||||
options:
|
|
||||||
max-size: "10m"
|
|
||||||
max-file: "3"
|
|
||||||
labels:
|
|
||||||
- "filebeat.enable=false" # Exclude this container
|
|
||||||
- "filebeat.multiline.pattern=^\\[" # Custom multiline pattern
|
|
||||||
```
|
|
||||||
|
|
||||||
### Logging Driver Compatibility
|
|
||||||
The Docker API input works with all Docker logging drivers:
|
|
||||||
- **local**: Recommended for production (efficient, no file access needed)
|
|
||||||
- **json-file**: Traditional default driver
|
|
||||||
- **journald**: SystemD journal integration
|
|
||||||
- **syslog**: Forward to syslog
|
|
||||||
- **none**: Disables logging (Filebeat won't collect)
|
|
||||||
|
|
||||||
You can use the `local` driver for better performance since Filebeat doesn't need to read files.
|
|
||||||
@@ -1,34 +1,56 @@
|
|||||||
# LogClient
|
# Log Client (Promtail)
|
||||||
|
|
||||||
Ships Docker container and system logs to LogServer using Filebeat.
|
Ships Docker and system logs to Log Server. Zero configuration!
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
1. **Get API Key**
|
1. **Configure** (edit `config/service.env`):
|
||||||
- Ask LogServer admin to run `./generate-api-key.sh`
|
|
||||||
- They'll provide your API key
|
|
||||||
|
|
||||||
2. **Configure**
|
|
||||||
Edit `config/service.env`:
|
|
||||||
```bash
|
```bash
|
||||||
LOGSERVER_HOST=<server-ip>
|
LOGSERVER_HOST=<server-ip> # Your log server IP
|
||||||
LOGSERVER_PORT=5044
|
LOGSERVER_PORT=3100 # Default Loki port
|
||||||
API_KEY=<your-api-key>
|
LOKI_USER=logclient # Default username
|
||||||
|
LOKI_PASSWORD=<password> # Get from server admin
|
||||||
```
|
```
|
||||||
|
|
||||||
3. **Install**
|
2. **Install**:
|
||||||
```bash
|
```bash
|
||||||
dropshell install logclient
|
dropshell install logclient
|
||||||
```
|
```
|
||||||
|
|
||||||
## What It Does
|
That's it! Logs are now shipping to your central server.
|
||||||
- Collects all Docker container logs via API
|
|
||||||
- Collects system logs (/var/log)
|
|
||||||
- Ships to central LogServer
|
|
||||||
- Works with any Docker logging driver
|
|
||||||
|
|
||||||
## Requirements
|
## What Gets Sent
|
||||||
- Docker socket access
|
- All Docker container logs (via Docker API)
|
||||||
- Network connection to LogServer port 5044
|
- System logs (/var/log/syslog)
|
||||||
|
- Auth logs (/var/log/auth.log)
|
||||||
|
- Hostname is automatically included
|
||||||
|
|
||||||
See [DOCUMENTATION.md](DOCUMENTATION.md) for full details.
|
## View Your Logs
|
||||||
|
Go to: `http://<logserver-ip>:3000`
|
||||||
|
- Click Dashboards → Central Logs
|
||||||
|
- Your server's logs appear with its hostname
|
||||||
|
|
||||||
|
## Features
|
||||||
|
- **Zero maintenance** - Just runs
|
||||||
|
- **Lightweight** - ~20MB RAM
|
||||||
|
- **Automatic** - Finds all containers
|
||||||
|
- **Labeled** - Hostname included automatically
|
||||||
|
|
||||||
|
## Simple Authentication
|
||||||
|
Just a username/password - much simpler than ELK's API key system!
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
Check if client is running:
|
||||||
|
```bash
|
||||||
|
./status.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
View client logs:
|
||||||
|
```bash
|
||||||
|
./logs.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Test connection to server:
|
||||||
|
```bash
|
||||||
|
nc -zv <logserver-ip> 3100
|
||||||
|
```
|
||||||
@@ -1,294 +0,0 @@
|
|||||||
# LogClient Template - Implementation TODO
|
|
||||||
|
|
||||||
## Phase 1: Core Infrastructure (Priority 1)
|
|
||||||
|
|
||||||
### Configuration Files
|
|
||||||
- [ ] Create `config/.template_info.env` with template metadata
|
|
||||||
- [ ] Create `config/service.env` with minimal required settings
|
|
||||||
- [ ] Define LOGSERVER_HOST and LOGSERVER_PORT variables
|
|
||||||
- [ ] Add AUTH_MODE variable (mtls, apikey, basic)
|
|
||||||
- [ ] Add certificate/key path variables for mTLS
|
|
||||||
- [ ] Add API_KEY variable for API key auth
|
|
||||||
- [ ] Add USERNAME/PASSWORD for basic auth
|
|
||||||
- [ ] Add optional performance and filtering variables
|
|
||||||
- [ ] Set sensible defaults where possible
|
|
||||||
|
|
||||||
### Filebeat Configuration
|
|
||||||
- [ ] Create base `filebeat.yml` configuration template
|
|
||||||
- [ ] Configure Docker input using Docker API (not autodiscover with hints)
|
|
||||||
- [ ] Set containers.ids: ["*"] to collect from all containers
|
|
||||||
- [ ] Set up system log inputs for host logs
|
|
||||||
- [ ] Configure output to Logstash
|
|
||||||
- [ ] Add error handling and retry logic
|
|
||||||
- [ ] Set up local disk buffering
|
|
||||||
- [ ] Configure stream: "all" to get both stdout and stderr
|
|
||||||
|
|
||||||
### Required Scripts
|
|
||||||
- [ ] Implement `install.sh` - Pull Filebeat image, configure auth, start
|
|
||||||
- [ ] Implement `uninstall.sh` - Stop and remove container (preserve config and certs)
|
|
||||||
- [ ] Implement `start.sh` - Start Filebeat with auth config and proper mounts
|
|
||||||
- [ ] Implement `stop.sh` - Gracefully stop Filebeat
|
|
||||||
- [ ] Implement `status.sh` - Check Filebeat health and auth status
|
|
||||||
- [ ] Create `setup-auth.sh` - Helper script to configure authentication
|
|
||||||
|
|
||||||
## Phase 2: Docker API Log Collection (Priority 1)
|
|
||||||
|
|
||||||
### Docker API Input Configuration
|
|
||||||
- [ ] Configure Docker input type (NOT autodiscover, use direct Docker input)
|
|
||||||
- [ ] Mount Docker socket (/var/run/docker.sock) with proper permissions
|
|
||||||
- [ ] Configure Docker API endpoint (unix:///var/run/docker.sock)
|
|
||||||
- [ ] Set up real-time log streaming from Docker daemon
|
|
||||||
- [ ] Enable collection from ALL logging drivers (local, json-file, journald, etc.)
|
|
||||||
- [ ] Configure since_time to get recent logs on startup
|
|
||||||
|
|
||||||
### Container Metadata Extraction
|
|
||||||
- [ ] Extract container name, ID, image name, and image tag
|
|
||||||
- [ ] Map container labels to fields
|
|
||||||
- [ ] Handle docker-compose project names and service names
|
|
||||||
- [ ] Add container state information
|
|
||||||
- [ ] Include container environment variables (filtered)
|
|
||||||
- [ ] Handle container lifecycle events (start, stop, restart)
|
|
||||||
|
|
||||||
### Container Filtering
|
|
||||||
- [ ] Implement include/exclude by container name patterns
|
|
||||||
- [ ] Add label-based filtering (containers.labels)
|
|
||||||
- [ ] Create ignore patterns for system containers
|
|
||||||
- [ ] Add support for custom filter expressions
|
|
||||||
- [ ] Configure combine_partial to handle partial log lines
|
|
||||||
- [ ] Document filtering examples with Docker API syntax
|
|
||||||
|
|
||||||
## Phase 3: System Log Collection (Priority 1)
|
|
||||||
|
|
||||||
### Log File Inputs
|
|
||||||
- [ ] Configure /var/log/syslog or /var/log/messages
|
|
||||||
- [ ] Add /var/log/auth.log or /var/log/secure
|
|
||||||
- [ ] Include /var/log/kern.log
|
|
||||||
- [ ] Monitor /var/log/dpkg.log or /var/log/yum.log
|
|
||||||
- [ ] Add custom log path support via environment variable
|
|
||||||
|
|
||||||
### Journald Integration
|
|
||||||
- [ ] Detect if systemd/journald is available
|
|
||||||
- [ ] Configure journald input if present
|
|
||||||
- [ ] Set up unit filtering
|
|
||||||
- [ ] Extract systemd metadata
|
|
||||||
- [ ] Handle binary journal format
|
|
||||||
|
|
||||||
### Log Parsing
|
|
||||||
- [ ] Configure syslog parsing
|
|
||||||
- [ ] Extract severity levels
|
|
||||||
- [ ] Parse timestamps correctly
|
|
||||||
- [ ] Handle different syslog formats
|
|
||||||
- [ ] Add timezone handling
|
|
||||||
|
|
||||||
## Phase 4: Output Configuration (Priority 1)
|
|
||||||
|
|
||||||
### Logstash Output
|
|
||||||
- [ ] Configure primary Logstash endpoint
|
|
||||||
- [ ] Set up connection parameters (timeout, retry)
|
|
||||||
- [ ] Configure bulk operations settings
|
|
||||||
- [ ] Add compression support
|
|
||||||
- [ ] Implement backpressure handling
|
|
||||||
|
|
||||||
### Connection Management
|
|
||||||
- [ ] Configure automatic reconnection
|
|
||||||
- [ ] Set exponential backoff for retries
|
|
||||||
- [ ] Add connection pooling
|
|
||||||
- [ ] Configure keepalive settings
|
|
||||||
- [ ] Handle DNS resolution failures
|
|
||||||
|
|
||||||
### Authentication Configuration (Priority 1 - CRITICAL)
|
|
||||||
- [ ] Implement mTLS authentication support
|
|
||||||
- [ ] Configure client certificate and key loading
|
|
||||||
- [ ] Add CA certificate validation
|
|
||||||
- [ ] Implement API key authentication
|
|
||||||
- [ ] Add basic auth as fallback option
|
|
||||||
- [ ] Create authentication mode selection logic
|
|
||||||
- [ ] Handle authentication failures gracefully
|
|
||||||
- [ ] Add certificate expiry checking
|
|
||||||
- [ ] Implement secure credential storage
|
|
||||||
- [ ] Document authentication setup process
|
|
||||||
|
|
||||||
## Phase 5: Reliability Features (Priority 2)
|
|
||||||
|
|
||||||
### Local Buffering
|
|
||||||
- [ ] Configure disk queue for reliability
|
|
||||||
- [ ] Set queue size limits
|
|
||||||
- [ ] Configure memory queue settings
|
|
||||||
- [ ] Add overflow handling
|
|
||||||
- [ ] Set up automatic cleanup of old events
|
|
||||||
|
|
||||||
### Error Handling
|
|
||||||
- [ ] Add retry logic for failed sends
|
|
||||||
- [ ] Configure dead letter queue
|
|
||||||
- [ ] Add circuit breaker pattern
|
|
||||||
- [ ] Log transmission errors appropriately
|
|
||||||
- [ ] Add metrics for monitoring failures
|
|
||||||
|
|
||||||
### Performance Optimization
|
|
||||||
- [ ] Configure worker count
|
|
||||||
- [ ] Set batch size for sending
|
|
||||||
- [ ] Add compression level setting
|
|
||||||
- [ ] Configure CPU and memory limits
|
|
||||||
- [ ] Optimize for high-volume scenarios
|
|
||||||
|
|
||||||
## Phase 6: Optional Scripts (Priority 2)
|
|
||||||
|
|
||||||
### Operational Scripts
|
|
||||||
- [ ] Implement `logs.sh` - Show Filebeat logs
|
|
||||||
- [ ] Implement `destroy.sh` - Complete removal
|
|
||||||
- [ ] Implement `ssh.sh` - Shell into Filebeat container
|
|
||||||
- [ ] Create `test.sh` - Test connectivity to server
|
|
||||||
- [ ] Add `metrics.sh` - Show Filebeat statistics
|
|
||||||
|
|
||||||
### Diagnostic Scripts
|
|
||||||
- [ ] Create connectivity test script
|
|
||||||
- [ ] Add configuration validation script
|
|
||||||
- [ ] Create debug mode enabler
|
|
||||||
- [ ] Add log sampling script
|
|
||||||
- [ ] Create performance benchmark script
|
|
||||||
|
|
||||||
## Phase 7: Monitoring & Health (Priority 2)
|
|
||||||
|
|
||||||
### Health Checks
|
|
||||||
- [ ] Configure Filebeat HTTP endpoint
|
|
||||||
- [ ] Add Docker health check
|
|
||||||
- [ ] Monitor queue status
|
|
||||||
- [ ] Check connection to Logstash
|
|
||||||
- [ ] Track dropped events
|
|
||||||
|
|
||||||
### Metrics Collection
|
|
||||||
- [ ] Enable Filebeat monitoring
|
|
||||||
- [ ] Export metrics endpoint
|
|
||||||
- [ ] Track events sent/failed
|
|
||||||
- [ ] Monitor resource usage
|
|
||||||
- [ ] Add performance counters
|
|
||||||
|
|
||||||
### Status Reporting
|
|
||||||
- [ ] Implement detailed status in status.sh
|
|
||||||
- [ ] Show connection state
|
|
||||||
- [ ] Display queue status
|
|
||||||
- [ ] Report recent errors
|
|
||||||
- [ ] Show throughput metrics
|
|
||||||
|
|
||||||
## Phase 8: Advanced Features (Priority 3)
|
|
||||||
|
|
||||||
### Processors
|
|
||||||
- [ ] Add field renaming processor
|
|
||||||
- [ ] Configure drop_event conditions
|
|
||||||
- [ ] Add rate limiting processor
|
|
||||||
- [ ] Include fingerprinting for deduplication
|
|
||||||
- [ ] Add custom field enrichment
|
|
||||||
|
|
||||||
### Multiline Handling
|
|
||||||
- [ ] Configure patterns for common languages
|
|
||||||
- [ ] Java stack trace handling
|
|
||||||
- [ ] Python traceback handling
|
|
||||||
- [ ] Go panic handling
|
|
||||||
- [ ] Custom pattern support via environment
|
|
||||||
|
|
||||||
### Field Management
|
|
||||||
- [ ] Configure field inclusion/exclusion
|
|
||||||
- [ ] Add custom fields via environment
|
|
||||||
- [ ] Set up field type conversions
|
|
||||||
- [ ] Add timestamp parsing
|
|
||||||
- [ ] Configure field aliasing
|
|
||||||
|
|
||||||
## Phase 9: Testing (Priority 3)
|
|
||||||
|
|
||||||
### Unit Testing
|
|
||||||
- [ ] Test configuration generation
|
|
||||||
- [ ] Verify volume mounts
|
|
||||||
- [ ] Test environment variable substitution
|
|
||||||
- [ ] Validate filtering logic
|
|
||||||
- [ ] Test error conditions
|
|
||||||
|
|
||||||
### Integration Testing
|
|
||||||
- [ ] Test with logserver template
|
|
||||||
- [ ] Verify Docker log collection
|
|
||||||
- [ ] Test system log collection
|
|
||||||
- [ ] Validate SSL connectivity
|
|
||||||
- [ ] Test reconnection scenarios
|
|
||||||
- [ ] Verify buffering during outages
|
|
||||||
|
|
||||||
### Load Testing
|
|
||||||
- [ ] Test with high log volume
|
|
||||||
- [ ] Measure resource usage
|
|
||||||
- [ ] Test queue overflow handling
|
|
||||||
- [ ] Verify rate limiting
|
|
||||||
- [ ] Benchmark throughput
|
|
||||||
|
|
||||||
## Phase 10: Documentation (Priority 3)
|
|
||||||
|
|
||||||
### User Documentation
|
|
||||||
- [ ] Create README.txt for dropshell
|
|
||||||
- [ ] Document all configuration options
|
|
||||||
- [ ] Add troubleshooting guide
|
|
||||||
- [ ] Create quick start guide
|
|
||||||
- [ ] Add FAQ section
|
|
||||||
|
|
||||||
### Configuration Examples
|
|
||||||
- [ ] Minimal configuration example
|
|
||||||
- [ ] High-volume configuration
|
|
||||||
- [ ] Secure SSL configuration
|
|
||||||
- [ ] Filtered configuration
|
|
||||||
- [ ] Custom paths configuration
|
|
||||||
|
|
||||||
### Integration Guides
|
|
||||||
- [ ] Integration with logserver
|
|
||||||
- [ ] Docker Compose examples
|
|
||||||
- [ ] Kubernetes DaemonSet example
|
|
||||||
- [ ] Swarm mode configuration
|
|
||||||
- [ ] Custom application integration
|
|
||||||
|
|
||||||
## Phase 11: Production Readiness (Priority 4)
|
|
||||||
|
|
||||||
### Security Hardening
|
|
||||||
- [ ] Run as non-root user where possible
|
|
||||||
- [ ] Minimize container capabilities
|
|
||||||
- [ ] Add secrets management
|
|
||||||
- [ ] Configure log sanitization
|
|
||||||
- [ ] Add audit logging
|
|
||||||
|
|
||||||
### Updates & Maintenance
|
|
||||||
- [ ] Add update notification
|
|
||||||
- [ ] Create upgrade script
|
|
||||||
- [ ] Add configuration migration
|
|
||||||
- [ ] Document breaking changes
|
|
||||||
- [ ] Create rollback procedure
|
|
||||||
|
|
||||||
### Compatibility
|
|
||||||
- [ ] Test with different Filebeat versions
|
|
||||||
- [ ] Verify Docker API compatibility
|
|
||||||
- [ ] Test on different Linux distributions
|
|
||||||
- [ ] Validate with various log formats
|
|
||||||
- [ ] Ensure Logstash version compatibility
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
### Design Principles
|
|
||||||
1. **Minimal configuration**: Just needs LOGSERVER_HOST to work
|
|
||||||
2. **Docker API access**: Use Docker API for driver-independent log collection
|
|
||||||
3. **Automatic discovery**: Find all container logs without manual configuration
|
|
||||||
4. **Reliability first**: Never lose logs, buffer locally if needed
|
|
||||||
5. **Low overhead**: Minimal resource usage on host
|
|
||||||
6. **Non-intrusive**: No changes to existing containers needed
|
|
||||||
7. **Driver flexibility**: Allow containers to use any logging driver (especially `local`)
|
|
||||||
|
|
||||||
### Key Requirements
|
|
||||||
- Must work with zero configuration beyond server address
|
|
||||||
- Must use Docker API input, not file-based collection
|
|
||||||
- Must support all Docker logging drivers (local, json-file, etc.)
|
|
||||||
- Must handle Docker socket permissions properly
|
|
||||||
- Must be resilient to network failures
|
|
||||||
- Must not impact host performance significantly
|
|
||||||
- Must preserve configuration on uninstall
|
|
||||||
|
|
||||||
### Testing Checklist
|
|
||||||
- [ ] Validates with dropshell test-template
|
|
||||||
- [ ] Connects to logserver successfully
|
|
||||||
- [ ] Collects Docker logs automatically
|
|
||||||
- [ ] Collects system logs properly
|
|
||||||
- [ ] Handles server downtime gracefully
|
|
||||||
- [ ] Reconnects automatically
|
|
||||||
- [ ] Resource usage stays within limits
|
|
||||||
- [ ] Uninstall preserves configuration
|
|
||||||
@@ -1,7 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
# Define volume items for logclient container
|
|
||||||
# These are used across backup, restore, create, and destroy operations
|
|
||||||
|
|
||||||
get_logclient_volumes() {
|
|
||||||
echo "volume:datavolume:${DATA_VOLUME}" "volume:configvolume:${CONFIG_VOLUME}"
|
|
||||||
}
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
source "${AGENT_PATH}/common.sh"
|
|
||||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
|
||||||
|
|
||||||
_check_required_env_vars "CONTAINER_NAME"
|
|
||||||
|
|
||||||
echo "Backing up ${CONTAINER_NAME} volumes..."
|
|
||||||
|
|
||||||
# Stop the container to ensure data consistency
|
|
||||||
bash ./stop.sh || true
|
|
||||||
|
|
||||||
# Backup volumes
|
|
||||||
BACKUP_DIR="${CONFIG_PATH}/backups/$(date +%Y%m%d_%H%M%S)"
|
|
||||||
mkdir -p "$BACKUP_DIR"
|
|
||||||
|
|
||||||
# Export volumes
|
|
||||||
for volume in $(get_logclient_volumes); do
|
|
||||||
volume_name=$(echo $volume | cut -d: -f3)
|
|
||||||
echo "Backing up volume: $volume_name"
|
|
||||||
docker run --rm -v "$volume_name:/source:ro" -v "$BACKUP_DIR:/backup" alpine \
|
|
||||||
tar -czf "/backup/${volume_name}.tar.gz" -C /source .
|
|
||||||
done
|
|
||||||
|
|
||||||
# Backup configuration
|
|
||||||
cp -r "${CONFIG_PATH}" "$BACKUP_DIR/config_backup"
|
|
||||||
|
|
||||||
echo "Backup completed to: $BACKUP_DIR"
|
|
||||||
|
|
||||||
# Restart the container
|
|
||||||
bash ./start.sh
|
|
||||||
|
|
||||||
echo "Container restarted"
|
|
||||||
@@ -1,16 +0,0 @@
|
|||||||
# Template identifier - MUST match the directory name
|
|
||||||
TEMPLATE=logclient
|
|
||||||
|
|
||||||
# Requirements
|
|
||||||
REQUIRES_HOST_ROOT=false # No root access on host needed
|
|
||||||
REQUIRES_DOCKER=true # Docker is required
|
|
||||||
REQUIRES_DOCKER_ROOT=false # Docker root privileges not specifically needed
|
|
||||||
|
|
||||||
# Docker image settings
|
|
||||||
IMAGE_REGISTRY="docker.elastic.co"
|
|
||||||
IMAGE_REPO="beats/filebeat"
|
|
||||||
IMAGE_TAG="7.17.23"
|
|
||||||
|
|
||||||
# Volume definitions
|
|
||||||
DATA_VOLUME=logclient_data
|
|
||||||
CONFIG_VOLUME=logclient_config
|
|
||||||
@@ -1,34 +1,16 @@
|
|||||||
# Service identification
|
# Log Client Configuration (Promtail)
|
||||||
CONTAINER_NAME=logclient-filebeat
|
CONTAINER_NAME=logclient
|
||||||
|
|
||||||
# Server settings (REQUIRED by dropshell)
|
# Server settings (REQUIRED by dropshell)
|
||||||
SSH_USER="root"
|
SSH_USER="root"
|
||||||
|
|
||||||
# Docker image tag override (optional)
|
|
||||||
IMAGE_TAG="7.17.23"
|
|
||||||
|
|
||||||
# REQUIRED: Log server connection
|
# REQUIRED: Log server connection
|
||||||
LOGSERVER_HOST=
|
LOGSERVER_HOST=
|
||||||
LOGSERVER_PORT=5044
|
LOGSERVER_PORT=3100
|
||||||
|
|
||||||
# REQUIRED: API Key Authentication
|
# REQUIRED: Authentication (get from log server admin)
|
||||||
API_KEY="" # Get from logserver admin using generate-api-key.sh
|
LOKI_USER=logclient
|
||||||
|
LOKI_PASSWORD=
|
||||||
|
|
||||||
# Performance tuning
|
# Optional: Set a custom hostname label (defaults to actual hostname)
|
||||||
BULK_MAX_SIZE=2048 # Maximum batch size
|
# HOSTNAME_LABEL=
|
||||||
WORKER_THREADS=1 # Number of worker threads
|
|
||||||
QUEUE_SIZE=4096 # Internal queue size
|
|
||||||
MAX_BACKOFF=60s # Maximum retry backoff
|
|
||||||
|
|
||||||
# Filtering
|
|
||||||
EXCLUDE_CONTAINERS="" # Comma-separated container names to exclude
|
|
||||||
INCLUDE_CONTAINERS="" # Only include these containers (if set)
|
|
||||||
EXCLUDE_LABELS="" # Exclude containers with these labels
|
|
||||||
INCLUDE_LABELS="" # Only include containers with these labels
|
|
||||||
|
|
||||||
# Additional log paths
|
|
||||||
CUSTOM_LOG_PATHS="" # Comma-separated additional paths to monitor
|
|
||||||
|
|
||||||
# Resource limits
|
|
||||||
MAX_CPU=50 # Maximum CPU usage percentage
|
|
||||||
MAX_MEMORY=200MB # Maximum memory usage
|
|
||||||
@@ -1,22 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
source "${AGENT_PATH}/common.sh"
|
|
||||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
|
||||||
|
|
||||||
_check_required_env_vars "CONTAINER_NAME"
|
|
||||||
|
|
||||||
echo "WARNING: This will permanently delete all data for ${CONTAINER_NAME}"
|
|
||||||
echo "This action cannot be undone!"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Since scripts must be non-interactive, only proceed if explicitly called
|
|
||||||
bash ./stop.sh || true
|
|
||||||
_remove_container "$CONTAINER_NAME" || true
|
|
||||||
|
|
||||||
# Remove all volumes
|
|
||||||
for volume in $(get_logclient_volumes); do
|
|
||||||
volume_name=$(echo $volume | cut -d: -f3)
|
|
||||||
echo "Removing volume: $volume_name"
|
|
||||||
docker volume rm "$volume_name" 2>/dev/null || true
|
|
||||||
done
|
|
||||||
|
|
||||||
echo "Service and all data destroyed"
|
|
||||||
19
logclient/docker-compose.yml
Normal file
19
logclient/docker-compose.yml
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
services:
|
||||||
|
promtail:
|
||||||
|
image: grafana/promtail:latest
|
||||||
|
container_name: ${CONTAINER_NAME}
|
||||||
|
volumes:
|
||||||
|
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||||
|
- /var/log:/var/log:ro
|
||||||
|
- /var/lib/docker/containers:/var/lib/docker/containers:ro
|
||||||
|
- ${CONFIG_PATH}/promtail.yaml:/etc/promtail/config.yml:ro
|
||||||
|
- promtail_positions:/tmp/positions
|
||||||
|
command: -config.file=/etc/promtail/config.yml
|
||||||
|
environment:
|
||||||
|
- HOSTNAME=${HOSTNAME_LABEL:-$(hostname)}
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
promtail_positions:
|
||||||
@@ -2,52 +2,39 @@
|
|||||||
source "${AGENT_PATH}/common.sh"
|
source "${AGENT_PATH}/common.sh"
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
# Check required environment variables
|
_check_required_env_vars "CONTAINER_NAME" "LOGSERVER_HOST" "LOGSERVER_PORT" "LOKI_PASSWORD"
|
||||||
_check_required_env_vars "CONTAINER_NAME" "IMAGE_REGISTRY" "IMAGE_REPO" "IMAGE_TAG" "LOGSERVER_HOST" "LOGSERVER_PORT" "API_KEY"
|
|
||||||
|
|
||||||
# Validate API key
|
# Check Docker
|
||||||
if [ -z "$API_KEY" ]; then
|
|
||||||
echo ""
|
|
||||||
echo "ERROR: API_KEY is not configured"
|
|
||||||
echo ""
|
|
||||||
echo "To get an API key:"
|
|
||||||
echo "1. On the logserver, run: ./generate-api-key.sh"
|
|
||||||
echo "2. Enter this client's hostname when prompted"
|
|
||||||
echo "3. Copy the generated API_KEY to this client's service.env"
|
|
||||||
echo ""
|
|
||||||
_die "Missing API_KEY configuration"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check Docker is available
|
|
||||||
_check_docker_installed || _die "Docker test failed"
|
_check_docker_installed || _die "Docker test failed"
|
||||||
|
|
||||||
# Test connectivity to logserver
|
# Test connectivity to logserver
|
||||||
echo "Testing connectivity to logserver at ${LOGSERVER_HOST}:${LOGSERVER_PORT}..."
|
echo "Testing connectivity to log server at ${LOGSERVER_HOST}:${LOGSERVER_PORT}..."
|
||||||
nc -zv "$LOGSERVER_HOST" "$LOGSERVER_PORT" 2>/dev/null || echo "WARNING: Cannot connect to logserver. Will retry when container starts."
|
nc -zv "$LOGSERVER_HOST" "$LOGSERVER_PORT" 2>/dev/null || echo "WARNING: Cannot connect to log server. Will retry when container starts."
|
||||||
|
|
||||||
# Pull the Docker image
|
|
||||||
docker pull "$IMAGE_REGISTRY/$IMAGE_REPO:$IMAGE_TAG" || _die "Failed to pull Filebeat image"
|
|
||||||
|
|
||||||
# Stop any existing container
|
# Stop any existing container
|
||||||
bash ./stop.sh || true
|
bash ./stop.sh 2>/dev/null || true
|
||||||
|
|
||||||
# Remove old container
|
# Generate configuration
|
||||||
_remove_container "$CONTAINER_NAME" || true
|
echo "Generating configuration..."
|
||||||
|
|
||||||
# Generate Filebeat configuration (with actual hostname)
|
|
||||||
echo "Generating Filebeat configuration..."
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
export HOSTNAME=$(hostname)
|
export HOSTNAME=$(hostname)
|
||||||
bash "$SCRIPT_DIR/scripts/generate-config.sh" || _die "Failed to generate configuration"
|
bash "$SCRIPT_DIR/scripts/generate-config.sh" || _die "Failed to generate configuration"
|
||||||
|
|
||||||
# Create volumes using common function
|
# Start the client
|
||||||
source "$SCRIPT_DIR/_volumes.sh"
|
echo "Starting Log Client..."
|
||||||
echo "Creating volumes..."
|
docker compose up -d || _die "Failed to start"
|
||||||
create_items $(get_logclient_volumes)
|
|
||||||
|
|
||||||
# Start the new container
|
echo ""
|
||||||
bash ./start.sh || _die "Failed to start Filebeat"
|
echo "========================================="
|
||||||
|
echo "Log Client Installed!"
|
||||||
echo "Installation of ${CONTAINER_NAME} complete"
|
echo "========================================="
|
||||||
echo "Collecting logs from Docker API and shipping to ${LOGSERVER_HOST}:${LOGSERVER_PORT}"
|
echo ""
|
||||||
echo "Using API key authentication"
|
echo "Shipping logs to: ${LOGSERVER_HOST}:${LOGSERVER_PORT}"
|
||||||
|
echo "Using authentication: ${LOKI_USER:-logclient}"
|
||||||
|
echo "Hostname label: $(hostname)"
|
||||||
|
echo ""
|
||||||
|
echo "Collecting:"
|
||||||
|
echo " - All Docker container logs"
|
||||||
|
echo " - System logs (/var/log)"
|
||||||
|
echo ""
|
||||||
|
echo "View logs at: http://${LOGSERVER_HOST}:3000"
|
||||||
|
echo "========================================="
|
||||||
5
logclient/logs.sh
Executable file
5
logclient/logs.sh
Executable file
@@ -0,0 +1,5 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
source "${AGENT_PATH}/common.sh"
|
||||||
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
|
docker logs "${CONTAINER_NAME}" --tail 50
|
||||||
@@ -1,47 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
source "${AGENT_PATH}/common.sh"
|
|
||||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
|
||||||
|
|
||||||
_check_required_env_vars "CONTAINER_NAME"
|
|
||||||
|
|
||||||
if [ -z "$1" ]; then
|
|
||||||
echo "Usage: $0 <backup_directory>"
|
|
||||||
echo "Available backups:"
|
|
||||||
ls -la "${CONFIG_PATH}/backups/" 2>/dev/null || echo "No backups found"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
BACKUP_DIR="$1"
|
|
||||||
|
|
||||||
if [ ! -d "$BACKUP_DIR" ]; then
|
|
||||||
_die "Backup directory not found: $BACKUP_DIR"
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "Restoring from backup: $BACKUP_DIR"
|
|
||||||
|
|
||||||
# Stop the container
|
|
||||||
bash ./stop.sh || true
|
|
||||||
|
|
||||||
# Restore volumes
|
|
||||||
for volume in $(get_logclient_volumes); do
|
|
||||||
volume_name=$(echo $volume | cut -d: -f3)
|
|
||||||
backup_file="$BACKUP_DIR/${volume_name}.tar.gz"
|
|
||||||
|
|
||||||
if [ -f "$backup_file" ]; then
|
|
||||||
echo "Restoring volume: $volume_name"
|
|
||||||
docker run --rm -v "$volume_name:/target" -v "$BACKUP_DIR:/backup:ro" alpine \
|
|
||||||
sh -c "rm -rf /target/* && tar -xzf /backup/${volume_name}.tar.gz -C /target"
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# Restore configuration if exists
|
|
||||||
if [ -d "$BACKUP_DIR/config_backup" ]; then
|
|
||||||
cp -r "$BACKUP_DIR/config_backup/"* "${CONFIG_PATH}/"
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "Restore completed"
|
|
||||||
|
|
||||||
# Start the container
|
|
||||||
bash ./start.sh
|
|
||||||
|
|
||||||
echo "Container restarted"
|
|
||||||
@@ -1,163 +1,111 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
|
|
||||||
# Generate Filebeat configuration from template
|
# Generate Promtail configuration for Log Client
|
||||||
# This script creates a filebeat.yml configuration file with proper authentication
|
|
||||||
|
|
||||||
# Check required variables
|
# Check required variables
|
||||||
if [ -z "$LOGSERVER_HOST" ] || [ -z "$LOGSERVER_PORT" ]; then
|
if [ -z "$LOGSERVER_HOST" ] || [ -z "$LOGSERVER_PORT" ] || [ -z "$LOKI_PASSWORD" ]; then
|
||||||
echo "ERROR: Required environment variables not set"
|
echo "ERROR: Required environment variables not set"
|
||||||
echo " LOGSERVER_HOST: ${LOGSERVER_HOST:-NOT SET}"
|
echo " LOGSERVER_HOST: ${LOGSERVER_HOST:-NOT SET}"
|
||||||
echo " LOGSERVER_PORT: ${LOGSERVER_PORT:-NOT SET}"
|
echo " LOGSERVER_PORT: ${LOGSERVER_PORT:-NOT SET}"
|
||||||
|
echo " LOKI_PASSWORD: ${LOKI_PASSWORD:-NOT SET}"
|
||||||
echo ""
|
echo ""
|
||||||
echo "Please set these in config/service.env before running install"
|
echo "Please set these in config/service.env before running install"
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Determine config directory - use CONFIG_PATH from dropshell or fallback
|
# Get actual hostname
|
||||||
if [ -n "$CONFIG_PATH" ]; then
|
ACTUAL_HOSTNAME=${HOSTNAME_LABEL:-${HOSTNAME:-$(hostname 2>/dev/null || echo "unknown")}}
|
||||||
CONFIG_DIR="$CONFIG_PATH"
|
|
||||||
elif [ -d "./config" ]; then
|
|
||||||
CONFIG_DIR="./config"
|
|
||||||
else
|
|
||||||
CONFIG_DIR="."
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Ensure config directory exists
|
# Determine config directory
|
||||||
|
CONFIG_DIR="${CONFIG_PATH:-./config}"
|
||||||
mkdir -p "$CONFIG_DIR"
|
mkdir -p "$CONFIG_DIR"
|
||||||
|
|
||||||
# Set defaults for variables if not set
|
# Generate promtail.yaml configuration
|
||||||
BULK_MAX_SIZE=${BULK_MAX_SIZE:-2048}
|
cat > "$CONFIG_DIR/promtail.yaml" << EOF
|
||||||
WORKER_THREADS=${WORKER_THREADS:-1}
|
server:
|
||||||
QUEUE_SIZE=${QUEUE_SIZE:-4096}
|
http_listen_port: 9080
|
||||||
MAX_BACKOFF=${MAX_BACKOFF:-60s}
|
grpc_listen_port: 0
|
||||||
|
|
||||||
# Get actual hostname from the host system
|
positions:
|
||||||
ACTUAL_HOSTNAME=${HOSTNAME:-$(hostname 2>/dev/null || echo "unknown")}
|
filename: /tmp/positions/positions.yaml
|
||||||
|
|
||||||
# Generate filebeat.yml configuration with variable substitution
|
clients:
|
||||||
(
|
- url: http://${LOKI_USER:-logclient}:${LOKI_PASSWORD}@${LOGSERVER_HOST}:${LOGSERVER_PORT}/loki/api/v1/push
|
||||||
cat << 'TEMPLATE_EOF'
|
# Authentication via URL (HTTP Basic Auth)
|
||||||
# Filebeat Configuration for LogClient
|
|
||||||
# Generated by generate-config.sh
|
|
||||||
|
|
||||||
# ======================== Docker Input Configuration =========================
|
scrape_configs:
|
||||||
# Use Docker input to collect logs via Docker API
|
# Docker container logs via Docker API
|
||||||
filebeat.inputs:
|
- job_name: docker
|
||||||
- type: docker
|
docker_sd_configs:
|
||||||
enabled: true
|
- host: unix:///var/run/docker.sock
|
||||||
# Collect from all containers
|
refresh_interval: 5s
|
||||||
containers.ids:
|
relabel_configs:
|
||||||
- '*'
|
- source_labels: ['__meta_docker_container_name']
|
||||||
# Collect both stdout and stderr
|
regex: '/(.*)'
|
||||||
containers.stream: all
|
target_label: 'container_name'
|
||||||
# Combine partial log lines
|
- source_labels: ['__meta_docker_container_id']
|
||||||
combine_partial: true
|
target_label: 'container_id'
|
||||||
# Add Docker metadata
|
- source_labels: ['__meta_docker_container_image']
|
||||||
processors:
|
target_label: 'image'
|
||||||
- add_docker_metadata:
|
- target_label: 'hostname'
|
||||||
host: "unix:///var/run/docker.sock"
|
replacement: '${ACTUAL_HOSTNAME}'
|
||||||
|
- target_label: 'job'
|
||||||
|
replacement: 'docker'
|
||||||
|
|
||||||
# ======================== System Logs Configuration ==========================
|
# System logs
|
||||||
- type: log
|
- job_name: syslog
|
||||||
enabled: true
|
static_configs:
|
||||||
paths:
|
- targets:
|
||||||
- /var/log/syslog
|
- localhost
|
||||||
- /var/log/messages
|
labels:
|
||||||
exclude_lines: ['^#']
|
job: syslog
|
||||||
fields:
|
hostname: ${ACTUAL_HOSTNAME}
|
||||||
log_type: syslog
|
__path__: /var/log/syslog
|
||||||
|
|
||||||
- type: log
|
- job_name: messages
|
||||||
enabled: true
|
static_configs:
|
||||||
paths:
|
- targets:
|
||||||
- /var/log/auth.log
|
- localhost
|
||||||
- /var/log/secure
|
labels:
|
||||||
exclude_lines: ['^#']
|
job: messages
|
||||||
fields:
|
hostname: ${ACTUAL_HOSTNAME}
|
||||||
log_type: auth
|
__path__: /var/log/messages
|
||||||
|
|
||||||
# ======================== Processors Configuration ===========================
|
- job_name: auth
|
||||||
processors:
|
static_configs:
|
||||||
- add_host_metadata:
|
- targets:
|
||||||
when.not.contains:
|
- localhost
|
||||||
tags: forwarded
|
labels:
|
||||||
# Override hostname with actual host's hostname
|
job: auth
|
||||||
- add_fields:
|
hostname: ${ACTUAL_HOSTNAME}
|
||||||
target: agent
|
__path__: /var/log/auth.log
|
||||||
fields:
|
|
||||||
hostname: __ACTUAL_HOSTNAME__
|
|
||||||
- add_fields:
|
|
||||||
target: host
|
|
||||||
fields:
|
|
||||||
name: __ACTUAL_HOSTNAME__
|
|
||||||
|
|
||||||
# ======================== Output Configuration ===============================
|
# Docker container JSON logs (backup method)
|
||||||
output.logstash:
|
- job_name: containers
|
||||||
hosts: ["__LOGSERVER_HOST__:__LOGSERVER_PORT__"]
|
static_configs:
|
||||||
# SSL/TLS configuration
|
- targets:
|
||||||
ssl.enabled: false # Set to true when using TLS
|
- localhost
|
||||||
ssl.verification_mode: none # Set to full in production with proper certs
|
labels:
|
||||||
|
job: containers
|
||||||
|
hostname: ${ACTUAL_HOSTNAME}
|
||||||
|
__path__: /var/lib/docker/containers/*/*-json.log
|
||||||
|
pipeline_stages:
|
||||||
|
- json:
|
||||||
|
expressions:
|
||||||
|
output: log
|
||||||
|
stream: stream
|
||||||
|
time: time
|
||||||
|
- timestamp:
|
||||||
|
source: time
|
||||||
|
format: RFC3339Nano
|
||||||
|
- labels:
|
||||||
|
stream:
|
||||||
|
- output:
|
||||||
|
source: output
|
||||||
|
EOF
|
||||||
|
|
||||||
# Performance settings
|
echo "Promtail configuration generated at: $CONFIG_DIR/promtail.yaml"
|
||||||
bulk_max_size: __BULK_MAX_SIZE__
|
|
||||||
worker: __WORKER_THREADS__ # Must be >= 1
|
|
||||||
compression_level: 3
|
|
||||||
|
|
||||||
# Retry configuration
|
|
||||||
max_retries: 3
|
|
||||||
backoff.init: 1s
|
|
||||||
backoff.max: __MAX_BACKOFF__
|
|
||||||
|
|
||||||
# ======================== Global Fields =======================================
|
|
||||||
# Add API key as a field to all events
|
|
||||||
fields:
|
|
||||||
api_key: "__API_KEY__"
|
|
||||||
fields_under_root: false
|
|
||||||
|
|
||||||
# ======================== Queue Configuration ================================
|
|
||||||
queue.mem:
|
|
||||||
events: __QUEUE_SIZE__
|
|
||||||
flush.min_events: 512
|
|
||||||
flush.timeout: 5s
|
|
||||||
|
|
||||||
# ======================== Logging Configuration ==============================
|
|
||||||
logging.level: info
|
|
||||||
logging.to_files: true
|
|
||||||
logging.files:
|
|
||||||
path: /usr/share/filebeat/data/logs
|
|
||||||
name: filebeat
|
|
||||||
keepfiles: 3
|
|
||||||
permissions: 0600
|
|
||||||
|
|
||||||
# ======================== Monitoring ==========================================
|
|
||||||
monitoring.enabled: false
|
|
||||||
http.enabled: true
|
|
||||||
http.host: 0.0.0.0
|
|
||||||
http.port: 5066
|
|
||||||
|
|
||||||
# ======================== File Permissions ====================================
|
|
||||||
# Set strict permissions (disabled for Docker)
|
|
||||||
# filebeat.config.modules.path: ${path.config}/modules.d/*.yml
|
|
||||||
TEMPLATE_EOF
|
|
||||||
) | sed -e "s|__LOGSERVER_HOST__|${LOGSERVER_HOST}|g" \
|
|
||||||
-e "s|__LOGSERVER_PORT__|${LOGSERVER_PORT}|g" \
|
|
||||||
-e "s|__API_KEY__|${API_KEY}|g" \
|
|
||||||
-e "s|__BULK_MAX_SIZE__|${BULK_MAX_SIZE}|g" \
|
|
||||||
-e "s|__WORKER_THREADS__|${WORKER_THREADS}|g" \
|
|
||||||
-e "s|__QUEUE_SIZE__|${QUEUE_SIZE}|g" \
|
|
||||||
-e "s|__MAX_BACKOFF__|${MAX_BACKOFF}|g" \
|
|
||||||
-e "s|__ACTUAL_HOSTNAME__|${ACTUAL_HOSTNAME}|g" > "$CONFIG_DIR/filebeat.yml"
|
|
||||||
|
|
||||||
echo "Filebeat configuration generated at: $CONFIG_DIR/filebeat.yml"
|
|
||||||
echo "Configuration:"
|
echo "Configuration:"
|
||||||
echo " LOGSERVER_HOST: ${LOGSERVER_HOST}"
|
echo " LOGSERVER_HOST: ${LOGSERVER_HOST}"
|
||||||
echo " LOGSERVER_PORT: ${LOGSERVER_PORT}"
|
echo " LOGSERVER_PORT: ${LOGSERVER_PORT}"
|
||||||
echo " API_KEY: ${API_KEY:+[SET]}"
|
echo " HOSTNAME: ${ACTUAL_HOSTNAME}"
|
||||||
echo " WORKER_THREADS: ${WORKER_THREADS}"
|
|
||||||
|
|
||||||
# Additional warning if API_KEY is not set
|
|
||||||
if [ -z "$API_KEY" ]; then
|
|
||||||
echo ""
|
|
||||||
echo "WARNING: API_KEY is not set - logs may be rejected by the server"
|
|
||||||
echo "Get an API key from the LogServer admin using generate-api-key.sh"
|
|
||||||
fi
|
|
||||||
@@ -1,36 +1,6 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
source "${AGENT_PATH}/common.sh"
|
source "${AGENT_PATH}/common.sh"
|
||||||
_check_required_env_vars "CONTAINER_NAME" "IMAGE_REGISTRY" "IMAGE_REPO" "IMAGE_TAG"
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
# Check that config file exists
|
docker compose up -d || _die "Failed to start"
|
||||||
if [ ! -f "${CONFIG_PATH}/filebeat.yml" ]; then
|
echo "Log Client started"
|
||||||
_die "filebeat.yml not found in ${CONFIG_PATH}/filebeat.yml"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Create Docker command
|
|
||||||
DOCKER_RUN_CMD="docker run -d \
|
|
||||||
--name $CONTAINER_NAME \
|
|
||||||
--restart unless-stopped \
|
|
||||||
--user root \
|
|
||||||
-v /var/run/docker.sock:/var/run/docker.sock:ro \
|
|
||||||
-v /var/log:/var/log:ro \
|
|
||||||
-v ${CONFIG_PATH}:/usr/share/filebeat/config:ro \
|
|
||||||
-v ${DATA_VOLUME}:/usr/share/filebeat/data \
|
|
||||||
-e LOGSERVER_HOST=${LOGSERVER_HOST} \
|
|
||||||
-e LOGSERVER_PORT=${LOGSERVER_PORT} \
|
|
||||||
-e API_KEY=${API_KEY} \
|
|
||||||
-e HOSTNAME=$(hostname) \
|
|
||||||
$IMAGE_REGISTRY/$IMAGE_REPO:$IMAGE_TAG \
|
|
||||||
filebeat -e -strict.perms=false \
|
|
||||||
-c /usr/share/filebeat/config/filebeat.yml"
|
|
||||||
|
|
||||||
if ! _create_and_start_container "$DOCKER_RUN_CMD" "$CONTAINER_NAME"; then
|
|
||||||
_die "Failed to start container ${CONTAINER_NAME}"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check if the container is running
|
|
||||||
if ! _is_container_running "$CONTAINER_NAME"; then
|
|
||||||
_die "Container ${CONTAINER_NAME} is not running"
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "Container ${CONTAINER_NAME} started"
|
|
||||||
@@ -2,30 +2,10 @@
|
|||||||
source "${AGENT_PATH}/common.sh"
|
source "${AGENT_PATH}/common.sh"
|
||||||
_check_required_env_vars "CONTAINER_NAME"
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
# Check if container exists
|
if docker ps | grep -q "${CONTAINER_NAME}"; then
|
||||||
if ! docker ps -a --format "{{.Names}}" | grep -q "^${CONTAINER_NAME}$"; then
|
|
||||||
echo "Unknown"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check container state
|
|
||||||
STATE=$(docker inspect -f '{{.State.Status}}' "$CONTAINER_NAME" 2>/dev/null)
|
|
||||||
case "$STATE" in
|
|
||||||
running)
|
|
||||||
# Additional check: verify connection to logserver
|
|
||||||
if docker logs "$CONTAINER_NAME" 2>&1 | tail -20 | grep -q "ERROR"; then
|
|
||||||
echo "Error"
|
|
||||||
else
|
|
||||||
echo "Running"
|
echo "Running"
|
||||||
fi
|
echo " Shipping logs to: ${LOGSERVER_HOST}:${LOGSERVER_PORT}"
|
||||||
;;
|
docker ps --format "table {{.Names}}\t{{.Status}}" | grep "${CONTAINER_NAME}"
|
||||||
exited|stopped)
|
else
|
||||||
echo "Stopped"
|
echo "Stopped"
|
||||||
;;
|
fi
|
||||||
restarting|paused)
|
|
||||||
echo "Error"
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
echo "Unknown"
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
@@ -2,6 +2,5 @@
|
|||||||
source "${AGENT_PATH}/common.sh"
|
source "${AGENT_PATH}/common.sh"
|
||||||
_check_required_env_vars "CONTAINER_NAME"
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
docker stop "$CONTAINER_NAME" 2>/dev/null || true
|
docker compose stop || _die "Failed to stop"
|
||||||
|
echo "Log Client stopped"
|
||||||
echo "Filebeat stopped"
|
|
||||||
@@ -2,16 +2,5 @@
|
|||||||
source "${AGENT_PATH}/common.sh"
|
source "${AGENT_PATH}/common.sh"
|
||||||
_check_required_env_vars "CONTAINER_NAME"
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
# Stop the container
|
docker compose down || true
|
||||||
bash ./stop.sh || _die "Failed to stop container"
|
echo "Log Client uninstalled"
|
||||||
|
|
||||||
# Remove the container
|
|
||||||
_remove_container "$CONTAINER_NAME" || _die "Failed to remove container"
|
|
||||||
|
|
||||||
# CRITICAL: Never remove data volumes in uninstall.sh!
|
|
||||||
# Data volumes must be preserved for potential reinstallation
|
|
||||||
# Configuration and certificates must be preserved
|
|
||||||
|
|
||||||
echo "Uninstallation of ${CONTAINER_NAME} complete"
|
|
||||||
echo "Note: Configuration and certificates have been preserved."
|
|
||||||
echo "To remove all data, use destroy.sh"
|
|
||||||
@@ -1,279 +0,0 @@
|
|||||||
# Dropshell LogServer Template
|
|
||||||
|
|
||||||
A comprehensive centralized logging solution using the ELK Stack (Elasticsearch, Logstash, Kibana) for receiving, processing, and visualizing logs from multiple hosts.
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
This template deploys a full-featured ELK stack that:
|
|
||||||
- Receives logs from multiple sources via Beats protocol
|
|
||||||
- Stores and indexes logs in Elasticsearch
|
|
||||||
- Provides powerful search and visualization through Kibana
|
|
||||||
- Supports automatic log parsing and enrichment
|
|
||||||
- Handles Docker container logs and system logs from clients
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
### Components
|
|
||||||
|
|
||||||
1. **Elasticsearch** (7.17.x)
|
|
||||||
- Distributed search and analytics engine
|
|
||||||
- Stores and indexes all log data
|
|
||||||
- Provides fast full-text search capabilities
|
|
||||||
- Single-node configuration for simplicity (can be scaled)
|
|
||||||
|
|
||||||
2. **Logstash** (7.17.x)
|
|
||||||
- Log processing pipeline
|
|
||||||
- Receives logs from Filebeat clients
|
|
||||||
- Parses and enriches log data
|
|
||||||
- Routes logs to appropriate Elasticsearch indices
|
|
||||||
|
|
||||||
3. **Kibana** (7.17.x)
|
|
||||||
- Web UI for log exploration and visualization
|
|
||||||
- Create dashboards and alerts
|
|
||||||
- Real-time log streaming
|
|
||||||
- Advanced search queries
|
|
||||||
|
|
||||||
## Features
|
|
||||||
|
|
||||||
### Minimum Configuration Design
|
|
||||||
- Auto-discovery of log formats
|
|
||||||
- Pre-configured dashboards for common services
|
|
||||||
- Automatic index lifecycle management
|
|
||||||
- Built-in parsing for Docker and syslog formats
|
|
||||||
- Zero-configuration client connectivity
|
|
||||||
|
|
||||||
### Log Processing
|
|
||||||
- Automatic timestamp extraction
|
|
||||||
- Docker metadata enrichment (container name, image, labels)
|
|
||||||
- Syslog parsing with severity levels
|
|
||||||
- JSON log support
|
|
||||||
- Multi-line log handling (stacktraces, etc.)
|
|
||||||
- Grok pattern matching for common formats
|
|
||||||
|
|
||||||
### Security & Performance
|
|
||||||
- **Mutual TLS (mTLS)** authentication for client connections
|
|
||||||
- **API key authentication** as an alternative to certificates
|
|
||||||
- **Per-client authentication** with unique keys/certificates
|
|
||||||
- **SSL/TLS encryption** for all client connections
|
|
||||||
- **Basic authentication** for Kibana web access
|
|
||||||
- **IP whitelisting** for additional security
|
|
||||||
- Index lifecycle management for storage optimization
|
|
||||||
- Automatic old log cleanup
|
|
||||||
- Resource limits to prevent overconsumption
|
|
||||||
|
|
||||||
## Port Configuration
|
|
||||||
|
|
||||||
- **5601**: Kibana Web UI (HTTP/HTTPS with authentication)
|
|
||||||
- **9200**: Elasticsearch REST API (HTTP) - internal only
|
|
||||||
- **5044**: Logstash Beats input (TCP/TLS) - authenticated client connections
|
|
||||||
- **514**: Syslog input (UDP/TCP) - optional, unauthenticated
|
|
||||||
- **24224**: Fluentd forward input - optional Docker logging driver
|
|
||||||
|
|
||||||
## Storage Requirements
|
|
||||||
|
|
||||||
- **Minimum**: 10GB for basic operation
|
|
||||||
- **Recommended**: 50GB+ depending on log volume
|
|
||||||
- **Log Retention**: Default 30 days (configurable)
|
|
||||||
|
|
||||||
## Client Authentication
|
|
||||||
|
|
||||||
### Authentication Methods
|
|
||||||
|
|
||||||
1. **Mutual TLS (mTLS) - Recommended**
|
|
||||||
- Each client gets a unique certificate signed by the server's CA
|
|
||||||
- Strongest security with mutual authentication
|
|
||||||
- Automatic certificate validation
|
|
||||||
|
|
||||||
2. **API Keys**
|
|
||||||
- Each client gets a unique API key
|
|
||||||
- Simpler to manage than certificates
|
|
||||||
- Good for environments where certificate management is difficult
|
|
||||||
|
|
||||||
3. **Basic Auth (Not Recommended)**
|
|
||||||
- Shared username/password
|
|
||||||
- Least secure, only for testing
|
|
||||||
|
|
||||||
### Client Configuration
|
|
||||||
|
|
||||||
Clients using the `logclient` template will:
|
|
||||||
1. Authenticate using provided credentials (cert/key or API key)
|
|
||||||
2. Establish encrypted TLS connection
|
|
||||||
3. Ship all Docker container logs
|
|
||||||
4. Ship system logs (syslog, auth, kernel)
|
|
||||||
5. Maintain connection with automatic reconnection
|
|
||||||
6. Buffer logs locally during network outages
|
|
||||||
|
|
||||||
## Dashboard Features
|
|
||||||
|
|
||||||
### Pre-configured Dashboards
|
|
||||||
- **System Overview**: Overall health and log volume metrics
|
|
||||||
- **Docker Containers**: Container-specific logs and metrics
|
|
||||||
- **Error Analysis**: Aggregated error logs from all sources
|
|
||||||
- **Security Events**: Authentication and access logs
|
|
||||||
- **Application Logs**: Parsed application-specific logs
|
|
||||||
|
|
||||||
### Search Capabilities
|
|
||||||
- Full-text search across all logs
|
|
||||||
- Filter by time range, host, container, severity
|
|
||||||
- Save and share search queries
|
|
||||||
- Export search results
|
|
||||||
|
|
||||||
## Resource Requirements
|
|
||||||
|
|
||||||
### Minimum
|
|
||||||
- CPU: 2 cores
|
|
||||||
- RAM: 4GB
|
|
||||||
- Storage: 10GB
|
|
||||||
|
|
||||||
### Recommended
|
|
||||||
- CPU: 4+ cores
|
|
||||||
- RAM: 8GB+
|
|
||||||
- Storage: 50GB+ SSD
|
|
||||||
|
|
||||||
## Configuration Options
|
|
||||||
|
|
||||||
### Environment Variables (service.env)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Elasticsearch settings
|
|
||||||
ES_HEAP_SIZE=2g
|
|
||||||
ES_MAX_MAP_COUNT=262144
|
|
||||||
|
|
||||||
# Logstash settings
|
|
||||||
LS_HEAP_SIZE=1g
|
|
||||||
LS_PIPELINE_WORKERS=2
|
|
||||||
|
|
||||||
# Kibana settings
|
|
||||||
KIBANA_PASSWORD=changeme
|
|
||||||
KIBANA_BASE_PATH=/
|
|
||||||
|
|
||||||
# Log retention
|
|
||||||
LOG_RETENTION_DAYS=30
|
|
||||||
LOG_MAX_SIZE_GB=50
|
|
||||||
|
|
||||||
# Authentication Mode
|
|
||||||
AUTH_MODE=mtls # Options: mtls, apikey, basic
|
|
||||||
ENABLE_TLS=true
|
|
||||||
|
|
||||||
# mTLS Settings (if AUTH_MODE=mtls)
|
|
||||||
CA_CERT_PATH=/certs/ca.crt
|
|
||||||
SERVER_CERT_PATH=/certs/server.crt
|
|
||||||
SERVER_KEY_PATH=/certs/server.key
|
|
||||||
CLIENT_CERT_REQUIRED=true
|
|
||||||
|
|
||||||
# API Key Settings (if AUTH_MODE=apikey)
|
|
||||||
API_KEYS_PATH=/config/api-keys.yml
|
|
||||||
|
|
||||||
# Network Security
|
|
||||||
ALLOWED_IPS="" # Comma-separated list, empty = all
|
|
||||||
```
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
### Installation
|
|
||||||
```bash
|
|
||||||
dropshell install logserver
|
|
||||||
```
|
|
||||||
|
|
||||||
### Generate Client Credentials
|
|
||||||
|
|
||||||
#### For mTLS Authentication:
|
|
||||||
```bash
|
|
||||||
# Generate client certificate for a new host
|
|
||||||
dropshell exec logserver /scripts/generate-client-cert.sh hostname
|
|
||||||
# This creates hostname.crt and hostname.key files
|
|
||||||
```
|
|
||||||
|
|
||||||
#### For API Key Authentication:
|
|
||||||
```bash
|
|
||||||
# Generate API key for a new client
|
|
||||||
dropshell exec logserver /scripts/generate-api-key.sh hostname
|
|
||||||
# Returns an API key to configure in the client
|
|
||||||
```
|
|
||||||
|
|
||||||
### Access Kibana
|
|
||||||
Navigate to `https://<server-ip>:5601` in your browser.
|
|
||||||
|
|
||||||
Default credentials:
|
|
||||||
- Username: `elastic`
|
|
||||||
- Password: `changeme` (change in service.env)
|
|
||||||
|
|
||||||
### View Logs
|
|
||||||
```bash
|
|
||||||
dropshell logs logserver
|
|
||||||
```
|
|
||||||
|
|
||||||
### Backup
|
|
||||||
```bash
|
|
||||||
dropshell backup logserver
|
|
||||||
```
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Common Issues
|
|
||||||
|
|
||||||
1. **Elasticsearch failing to start**
|
|
||||||
- Check vm.max_map_count: `sysctl vm.max_map_count` (should be 262144+)
|
|
||||||
- Verify sufficient memory available
|
|
||||||
|
|
||||||
2. **No logs appearing in Kibana**
|
|
||||||
- Check Logstash is receiving data: port 5044 should be open
|
|
||||||
- Verify client connectivity
|
|
||||||
- Check index patterns in Kibana
|
|
||||||
|
|
||||||
3. **High memory usage**
|
|
||||||
- Adjust heap sizes in service.env
|
|
||||||
- Configure index lifecycle management
|
|
||||||
- Reduce retention period
|
|
||||||
|
|
||||||
## Integration
|
|
||||||
|
|
||||||
This template is designed to work seamlessly with the `logclient` template. Simply:
|
|
||||||
1. Deploy this logserver
|
|
||||||
2. Deploy logclient on each host you want to monitor
|
|
||||||
3. Configure logclient with the logserver address
|
|
||||||
4. Logs will automatically start flowing
|
|
||||||
|
|
||||||
## Security Considerations
|
|
||||||
|
|
||||||
1. **Authentication Setup**
|
|
||||||
- Use mTLS for production environments
|
|
||||||
- Generate unique credentials for each client
|
|
||||||
- Rotate certificates/keys regularly
|
|
||||||
- Store credentials securely
|
|
||||||
|
|
||||||
2. **Network Security**
|
|
||||||
- Always use TLS encryption for client connections
|
|
||||||
- Configure IP whitelisting when possible
|
|
||||||
- Use firewall rules to restrict access
|
|
||||||
- Consider VPN or private networks
|
|
||||||
|
|
||||||
3. **Access Control**
|
|
||||||
- Change default Kibana password immediately
|
|
||||||
- Create read-only users for viewing logs
|
|
||||||
- Implement role-based access control (RBAC)
|
|
||||||
- Audit access logs regularly
|
|
||||||
|
|
||||||
4. **Data Protection**
|
|
||||||
- Regular backups of Elasticsearch indices
|
|
||||||
- Encrypt data at rest (optional)
|
|
||||||
- Monitor disk usage to prevent data loss
|
|
||||||
- Implement log retention policies
|
|
||||||
|
|
||||||
## Maintenance
|
|
||||||
|
|
||||||
### Daily Tasks
|
|
||||||
- Monitor disk usage
|
|
||||||
- Check for failed log shipments
|
|
||||||
- Review error dashboards
|
|
||||||
|
|
||||||
### Weekly Tasks
|
|
||||||
- Verify all clients are reporting
|
|
||||||
- Check index health
|
|
||||||
- Review and optimize slow queries
|
|
||||||
|
|
||||||
### Monthly Tasks
|
|
||||||
- Update ELK stack components
|
|
||||||
- Archive old indices
|
|
||||||
- Review retention policies
|
|
||||||
- Performance tuning based on usage patterns
|
|
||||||
@@ -1,50 +1,73 @@
|
|||||||
# LogServer
|
# Log Server (Loki + Grafana)
|
||||||
|
|
||||||
Centralized logging with ELK Stack (Elasticsearch, Logstash, Kibana).
|
Central log server using Grafana + Loki. MUCH simpler than ELK/Kibana!
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
1. **System Setup**
|
1. **Install**
|
||||||
```bash
|
|
||||||
sudo sysctl -w vm.max_map_count=262144
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Configure**
|
|
||||||
Edit `config/service.env`:
|
|
||||||
- Set `SERVER_PUBLICBASEURL` to your actual server URL
|
|
||||||
- Set `KIBANA_USERNAME` to your preferred username
|
|
||||||
- Change `KIBANA_USER_PASSWORD` from default
|
|
||||||
|
|
||||||
3. **Install**
|
|
||||||
```bash
|
```bash
|
||||||
dropshell install logserver
|
dropshell install logserver
|
||||||
```
|
```
|
||||||
|
|
||||||
4. **Generate Client Keys**
|
2. **Access Web UI**
|
||||||
|
- URL: `http://<server-ip>:3000`
|
||||||
|
- Login: `admin` / `changeme` (set in service.env)
|
||||||
|
|
||||||
|
3. **View Logs**
|
||||||
|
- Click **Dashboards** (4 squares icon)
|
||||||
|
- Click **"Central Logs"**
|
||||||
|
- All logs from all servers appear!
|
||||||
|
|
||||||
|
## For Client Servers
|
||||||
|
|
||||||
|
On each server that should send logs here:
|
||||||
|
|
||||||
|
1. Edit `logclient/config/service.env`:
|
||||||
```bash
|
```bash
|
||||||
./generate-api-key.sh
|
LOGSERVER_HOST=<this-server-ip>
|
||||||
# Enter hostname when prompted
|
LOGSERVER_PORT=3100
|
||||||
# Save the API key for client configuration
|
LOKI_USER=logclient # Default username
|
||||||
|
LOKI_PASSWORD=<your-password> # From server's service.env
|
||||||
```
|
```
|
||||||
|
|
||||||
5. **Setup Kibana** (first time only)
|
2. Install the client:
|
||||||
```bash
|
```bash
|
||||||
./setup-kibana.sh
|
dropshell install logclient
|
||||||
```
|
```
|
||||||
|
|
||||||
6. **Access Kibana**
|
## Features
|
||||||
- URL: `http://<server-ip>:5601`
|
- **Super simple interface** - Just one dashboard
|
||||||
- Username: Set in `service.env` (KIBANA_USERNAME, default: `admin`)
|
- **Lightweight** - Uses 10x less resources than ELK
|
||||||
- Password: Set in `service.env` (KIBANA_USER_PASSWORD)
|
- **Real-time** - Logs appear instantly
|
||||||
- Click "Discover" → View your logs!
|
- **Multi-server** - See all servers in one place
|
||||||
|
- **Automatic labels** - Filter by hostname, container, etc.
|
||||||
|
|
||||||
|
## Security
|
||||||
|
- Loki endpoint requires authentication (HTTP Basic Auth)
|
||||||
|
- Default: `logclient` / `changeme`
|
||||||
|
- **IMPORTANT**: Change `LOKI_PASSWORD` in service.env!
|
||||||
|
|
||||||
## Ports
|
## Ports
|
||||||
- `5601` - Kibana Web UI
|
- `3000` - Grafana Web UI
|
||||||
- `5044` - Log ingestion (Filebeat)
|
- `3100` - Loki API (authenticated)
|
||||||
|
|
||||||
## Files
|
## How to Use
|
||||||
- `config/service.env` - Configuration
|
|
||||||
- `config/api-keys.yml` - Client API keys
|
|
||||||
- `generate-api-key.sh` - Add new clients
|
|
||||||
|
|
||||||
See [DOCUMENTATION.md](DOCUMENTATION.md) for full details.
|
### Filter by Server
|
||||||
|
- Use the "Server" dropdown at the top
|
||||||
|
- Or click any `hostname` label
|
||||||
|
|
||||||
|
### Filter by Container
|
||||||
|
- Click any `container_name` label
|
||||||
|
|
||||||
|
### Search Text
|
||||||
|
- Use the search box in each panel
|
||||||
|
|
||||||
|
### Time Range
|
||||||
|
- Top-right corner - adjust as needed
|
||||||
|
|
||||||
|
## Why This Instead of ELK?
|
||||||
|
- **10x simpler** - One clean dashboard vs Kibana complexity
|
||||||
|
- **10x smaller** - ~200MB RAM vs 2-3GB for ELK
|
||||||
|
- **Zero configuration** - Just works
|
||||||
|
- **Fast setup** - No index patterns, mappings, or complexity
|
||||||
@@ -1,100 +0,0 @@
|
|||||||
# LogServer Quick Setup Guide
|
|
||||||
|
|
||||||
## Prerequisites
|
|
||||||
- Docker and Docker Compose installed
|
|
||||||
- 4GB+ RAM, 10GB+ disk space
|
|
||||||
- Port 5601 (Kibana) and 5044 (Logstash) available
|
|
||||||
|
|
||||||
## Initial Setup
|
|
||||||
|
|
||||||
### 1. System Configuration
|
|
||||||
```bash
|
|
||||||
# Required for Elasticsearch
|
|
||||||
sudo sysctl -w vm.max_map_count=262144
|
|
||||||
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Configure Server
|
|
||||||
Edit `config/service.env`:
|
|
||||||
```bash
|
|
||||||
# Change default password
|
|
||||||
KIBANA_PASSWORD=your-secure-password
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Install
|
|
||||||
```bash
|
|
||||||
dropshell install logserver
|
|
||||||
```
|
|
||||||
|
|
||||||
## Generate Client API Keys
|
|
||||||
|
|
||||||
Run the interactive key generator:
|
|
||||||
```bash
|
|
||||||
./generate-api-key.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
Follow the prompts:
|
|
||||||
1. Enter hostname for each client
|
|
||||||
2. Script generates secure API key
|
|
||||||
3. Shows configuration to copy to client
|
|
||||||
4. Repeat for additional clients
|
|
||||||
|
|
||||||
## Access Kibana
|
|
||||||
|
|
||||||
1. Open browser: `http://your-server-ip:5601`
|
|
||||||
2. Login: `elastic` / `your-secure-password`
|
|
||||||
3. Create index pattern: `filebeat-*`
|
|
||||||
4. View logs in Discover tab
|
|
||||||
|
|
||||||
## Add Log Clients
|
|
||||||
|
|
||||||
On each client machine:
|
|
||||||
```bash
|
|
||||||
# Get API key from server admin (they run ./generate-api-key.sh)
|
|
||||||
|
|
||||||
# Edit logclient/config/service.env:
|
|
||||||
LOGSERVER_HOST=your-server-ip
|
|
||||||
LOGSERVER_PORT=5044
|
|
||||||
API_KEY=your-api-key-here
|
|
||||||
|
|
||||||
# Install and start
|
|
||||||
dropshell install logclient
|
|
||||||
```
|
|
||||||
|
|
||||||
## Verify Setup
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Check server status
|
|
||||||
dropshell status logserver
|
|
||||||
|
|
||||||
# View server logs
|
|
||||||
dropshell logs logserver
|
|
||||||
|
|
||||||
# Test client connection (from client)
|
|
||||||
docker logs logclient-filebeat | grep "connection"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
**Elasticsearch won't start**: Check `vm.max_map_count` is 262144+
|
|
||||||
|
|
||||||
**No logs in Kibana**:
|
|
||||||
- Verify client can reach server port 5044
|
|
||||||
- Check API key is correct in client's service.env
|
|
||||||
- Verify API key exists in server's api-keys.yml
|
|
||||||
- Refresh index pattern in Kibana
|
|
||||||
|
|
||||||
**High memory usage**: Adjust heap sizes in `service.env`:
|
|
||||||
```bash
|
|
||||||
ES_HEAP_SIZE=1g # Reduce from 2g
|
|
||||||
LS_HEAP_SIZE=512m # Reduce from 1g
|
|
||||||
```
|
|
||||||
|
|
||||||
## Security Checklist
|
|
||||||
|
|
||||||
- [ ] Changed default Kibana password
|
|
||||||
- [ ] Generated unique API key per client
|
|
||||||
- [ ] API keys stored securely
|
|
||||||
- [ ] Firewall allows only necessary ports (5601, 5044)
|
|
||||||
- [ ] Regular backup configured
|
|
||||||
- [ ] Reviewed api-keys.yml for old/unused keys
|
|
||||||
@@ -1,84 +0,0 @@
|
|||||||
# Simple Guide to Viewing Logs in Kibana
|
|
||||||
|
|
||||||
## First Time Setup
|
|
||||||
Run this once after installing LogServer:
|
|
||||||
```bash
|
|
||||||
./setup-kibana.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
## Viewing Logs - The Easy Way
|
|
||||||
|
|
||||||
### 1. Open Kibana
|
|
||||||
Go to: `http://<your-server>:5601`
|
|
||||||
|
|
||||||
### 2. Login
|
|
||||||
Use the username and password from your service.env
|
|
||||||
|
|
||||||
### 3. Click "Discover"
|
|
||||||
It's in the left menu (looks like a compass icon)
|
|
||||||
|
|
||||||
### 4. You're Done!
|
|
||||||
Your logs are now visible. That's it!
|
|
||||||
|
|
||||||
## Simple Controls
|
|
||||||
|
|
||||||
### See Recent Logs Only
|
|
||||||
- Top-right corner: Click the time picker
|
|
||||||
- Choose "Last 15 minutes" or "Last 1 hour"
|
|
||||||
|
|
||||||
### Filter by Container
|
|
||||||
- Find any log entry
|
|
||||||
- Next to `container_name`: click the `+` button
|
|
||||||
- Now you only see logs from that container
|
|
||||||
|
|
||||||
### Filter by Server
|
|
||||||
- Next to `host.name`: click the `+` button
|
|
||||||
- Now you only see logs from that host
|
|
||||||
|
|
||||||
### Search for Text
|
|
||||||
- Top search bar: Type any word
|
|
||||||
- Press Enter
|
|
||||||
- Shows only logs containing that word
|
|
||||||
|
|
||||||
### Live Updates
|
|
||||||
- Top-right: Click "Refresh"
|
|
||||||
- Choose "Every 5 seconds"
|
|
||||||
- Logs update automatically
|
|
||||||
|
|
||||||
### Remove Filters
|
|
||||||
- Look for filter pills under the search bar
|
|
||||||
- Click the `x` on any filter to remove it
|
|
||||||
|
|
||||||
## Common Searches
|
|
||||||
|
|
||||||
**Show errors only:**
|
|
||||||
```
|
|
||||||
error OR ERROR OR Error
|
|
||||||
```
|
|
||||||
|
|
||||||
**Show warnings and errors:**
|
|
||||||
```
|
|
||||||
error OR ERROR OR warn OR WARN
|
|
||||||
```
|
|
||||||
|
|
||||||
**Show specific container:**
|
|
||||||
```
|
|
||||||
container_name: "myapp"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Show multiple containers:**
|
|
||||||
```
|
|
||||||
container_name: ("app1" OR "app2")
|
|
||||||
```
|
|
||||||
|
|
||||||
## Tips
|
|
||||||
|
|
||||||
1. **Too many columns?** Click "container_name" and "message" in the left sidebar to show just those
|
|
||||||
|
|
||||||
2. **Want raw logs?** Click the ">" arrow next to any log entry to expand it
|
|
||||||
|
|
||||||
3. **Export logs?** Click "Share" → "CSV Reports" → "Generate CSV"
|
|
||||||
|
|
||||||
4. **Time zone wrong?** Click your profile icon → "Advanced Settings" → search "timezone"
|
|
||||||
|
|
||||||
That's all you need to know! Kibana has many advanced features, but for basic log viewing and searching, these commands are sufficient.
|
|
||||||
@@ -1,244 +0,0 @@
|
|||||||
# LogServer Template - Implementation TODO
|
|
||||||
|
|
||||||
## Phase 1: Core Infrastructure (Priority 1)
|
|
||||||
|
|
||||||
### Configuration Files
|
|
||||||
- [ ] Create `config/.template_info.env` with template metadata
|
|
||||||
- [ ] Create `config/service.env` with user-configurable settings
|
|
||||||
- [ ] Define all required environment variables (ports, passwords, heap sizes)
|
|
||||||
- [ ] Set appropriate default values for zero-config experience
|
|
||||||
|
|
||||||
### Docker Compose Setup
|
|
||||||
- [ ] Create `docker-compose.yml` with ELK stack services
|
|
||||||
- [ ] Configure Elasticsearch single-node setup
|
|
||||||
- [ ] Configure Logstash with Beats input pipeline
|
|
||||||
- [ ] Configure Kibana with Elasticsearch connection
|
|
||||||
- [ ] Set up proper networking between services
|
|
||||||
- [ ] Define named volumes for data persistence
|
|
||||||
- [ ] Configure health checks for each service
|
|
||||||
|
|
||||||
### Required Scripts
|
|
||||||
- [ ] Implement `install.sh` - Pull images, create volumes, start services
|
|
||||||
- [ ] Implement `uninstall.sh` - Stop and remove containers (preserve volumes!)
|
|
||||||
- [ ] Implement `start.sh` - Start all ELK services with docker-compose
|
|
||||||
- [ ] Implement `stop.sh` - Gracefully stop all services
|
|
||||||
- [ ] Implement `status.sh` - Check health of all three services
|
|
||||||
|
|
||||||
## Phase 2: Logstash Configuration (Priority 1)
|
|
||||||
|
|
||||||
### Input Configuration
|
|
||||||
- [ ] Configure Beats input on port 5044 with TLS/SSL
|
|
||||||
- [ ] Set up mutual TLS (mTLS) authentication
|
|
||||||
- [ ] Configure client certificate validation
|
|
||||||
- [ ] Add API key authentication option
|
|
||||||
- [ ] Implement IP whitelisting
|
|
||||||
- [ ] Add Syslog input on port 514 (UDP/TCP) - unauthenticated
|
|
||||||
- [ ] Add Docker Fluentd input on port 24224 (optional)
|
|
||||||
|
|
||||||
### Filter Pipeline
|
|
||||||
- [ ] Create Docker log parser (extract container metadata)
|
|
||||||
- [ ] Create Syslog parser (RFC3164 and RFC5424)
|
|
||||||
- [ ] Add JSON parser for structured logs
|
|
||||||
- [ ] Implement multiline pattern for stack traces
|
|
||||||
- [ ] Add timestamp extraction and normalization
|
|
||||||
- [ ] Create field enrichment (add host metadata)
|
|
||||||
- [ ] Implement conditional routing based on log type
|
|
||||||
|
|
||||||
### Output Configuration
|
|
||||||
- [ ] Configure Elasticsearch output with index patterns
|
|
||||||
- [ ] Set up index templates for different log types
|
|
||||||
- [ ] Configure index lifecycle management (ILM)
|
|
||||||
|
|
||||||
## Phase 3: Elasticsearch Setup (Priority 1)
|
|
||||||
|
|
||||||
### System Configuration
|
|
||||||
- [ ] Set appropriate heap size defaults (ES_HEAP_SIZE)
|
|
||||||
- [ ] Configure vm.max_map_count requirement check
|
|
||||||
- [ ] Set up single-node discovery settings
|
|
||||||
- [ ] Configure data persistence volume
|
|
||||||
- [ ] Set up index templates for:
|
|
||||||
- [ ] Docker logs (docker-*)
|
|
||||||
- [ ] System logs (syslog-*)
|
|
||||||
- [ ] Application logs (app-*)
|
|
||||||
- [ ] Error logs (errors-*)
|
|
||||||
|
|
||||||
### Index Management
|
|
||||||
- [ ] Configure ILM policies for log rotation
|
|
||||||
- [ ] Set retention period (default 30 days)
|
|
||||||
- [ ] Configure max index size limits
|
|
||||||
- [ ] Set up automatic cleanup of old indices
|
|
||||||
- [ ] Create snapshot repository configuration
|
|
||||||
|
|
||||||
## Phase 4: Kibana Configuration (Priority 2)
|
|
||||||
|
|
||||||
### Initial Setup
|
|
||||||
- [ ] Configure Kibana with Elasticsearch URL
|
|
||||||
- [ ] Set up basic authentication
|
|
||||||
- [ ] Configure server base path
|
|
||||||
- [ ] Set appropriate memory limits
|
|
||||||
|
|
||||||
### Pre-built Dashboards
|
|
||||||
- [ ] Create System Overview dashboard
|
|
||||||
- [ ] Create Docker Containers dashboard
|
|
||||||
- [ ] Create Error Analysis dashboard
|
|
||||||
- [ ] Create Security Events dashboard
|
|
||||||
- [ ] Create Host Metrics dashboard
|
|
||||||
|
|
||||||
### Saved Searches
|
|
||||||
- [ ] Error logs across all sources
|
|
||||||
- [ ] Authentication events
|
|
||||||
- [ ] Container lifecycle events
|
|
||||||
- [ ] Slow queries/performance issues
|
|
||||||
- [ ] Critical system events
|
|
||||||
|
|
||||||
### Index Patterns
|
|
||||||
- [ ] Configure docker-* pattern
|
|
||||||
- [ ] Configure syslog-* pattern
|
|
||||||
- [ ] Configure app-* pattern
|
|
||||||
- [ ] Configure filebeat-* pattern
|
|
||||||
|
|
||||||
## Phase 5: Optional Scripts (Priority 2)
|
|
||||||
|
|
||||||
### Operational Scripts
|
|
||||||
- [ ] Implement `logs.sh` - Show logs from all ELK services
|
|
||||||
- [ ] Implement `backup.sh` - Snapshot Elasticsearch indices
|
|
||||||
- [ ] Implement `restore.sh` - Restore from snapshots
|
|
||||||
- [ ] Implement `destroy.sh` - Complete removal including volumes
|
|
||||||
- [ ] Implement `ports.sh` - Display all exposed ports
|
|
||||||
- [ ] Implement `ssh.sh` - Shell into specific container
|
|
||||||
|
|
||||||
### Helper Scripts
|
|
||||||
- [ ] Create `_volumes.sh` for volume management helpers
|
|
||||||
- [ ] Add health check script for all services
|
|
||||||
- [ ] Create performance tuning script
|
|
||||||
- [ ] Add certificate generation script for SSL
|
|
||||||
|
|
||||||
## Phase 6: Security Features (Priority 1 - CRITICAL)
|
|
||||||
|
|
||||||
### Certificate Authority Setup
|
|
||||||
- [ ] Create CA certificate and key for signing client certs
|
|
||||||
- [ ] Generate server certificate for Logstash
|
|
||||||
- [ ] Create certificate generation script for clients
|
|
||||||
- [ ] Set up certificate storage structure
|
|
||||||
- [ ] Implement certificate rotation mechanism
|
|
||||||
|
|
||||||
### mTLS Authentication
|
|
||||||
- [ ] Configure Logstash for mutual TLS
|
|
||||||
- [ ] Set up client certificate validation
|
|
||||||
- [ ] Create client certificate generation script
|
|
||||||
- [ ] Implement certificate revocation list (CRL)
|
|
||||||
- [ ] Add certificate expiry monitoring
|
|
||||||
|
|
||||||
### API Key Authentication
|
|
||||||
- [ ] Create API key generation script
|
|
||||||
- [ ] Configure Logstash to accept API keys
|
|
||||||
- [ ] Implement API key storage (encrypted)
|
|
||||||
- [ ] Add API key rotation mechanism
|
|
||||||
- [ ] Create API key revocation process
|
|
||||||
|
|
||||||
### Network Security
|
|
||||||
- [ ] Implement IP whitelisting in Logstash
|
|
||||||
- [ ] Configure firewall rules
|
|
||||||
- [ ] Set up rate limiting
|
|
||||||
- [ ] Add connection throttling
|
|
||||||
- [ ] Implement DDoS protection
|
|
||||||
|
|
||||||
### Kibana Security
|
|
||||||
- [ ] Configure Kibana HTTPS
|
|
||||||
- [ ] Set up basic authentication
|
|
||||||
- [ ] Create user management scripts
|
|
||||||
- [ ] Implement session management
|
|
||||||
- [ ] Add audit logging
|
|
||||||
|
|
||||||
## Phase 7: Performance & Optimization (Priority 3)
|
|
||||||
|
|
||||||
### Resource Management
|
|
||||||
- [ ] Configure CPU limits for each service
|
|
||||||
- [ ] Set memory limits appropriately
|
|
||||||
- [ ] Add swap handling configuration
|
|
||||||
- [ ] Configure JVM options files
|
|
||||||
- [ ] Add performance monitoring
|
|
||||||
|
|
||||||
### Optimization
|
|
||||||
- [ ] Configure pipeline workers
|
|
||||||
- [ ] Set batch sizes for optimal throughput
|
|
||||||
- [ ] Configure queue sizes
|
|
||||||
- [ ] Add caching configuration
|
|
||||||
- [ ] Optimize index refresh intervals
|
|
||||||
|
|
||||||
## Phase 8: Testing & Documentation (Priority 3)
|
|
||||||
|
|
||||||
### Testing
|
|
||||||
- [ ] Test installation process
|
|
||||||
- [ ] Test uninstall (verify volume preservation)
|
|
||||||
- [ ] Test log ingestion from sample client
|
|
||||||
- [ ] Test all dashboard functionality
|
|
||||||
- [ ] Test backup and restore procedures
|
|
||||||
- [ ] Load test with high log volume
|
|
||||||
- [ ] Test failover and recovery
|
|
||||||
|
|
||||||
### Documentation
|
|
||||||
- [ ] Create README.txt for dropshell format
|
|
||||||
- [ ] Document all configuration options
|
|
||||||
- [ ] Add troubleshooting guide
|
|
||||||
- [ ] Create quick start guide
|
|
||||||
- [ ] Document upgrade procedures
|
|
||||||
- [ ] Add performance tuning guide
|
|
||||||
|
|
||||||
## Phase 9: Integration Testing (Priority 3)
|
|
||||||
|
|
||||||
### With LogClient
|
|
||||||
- [ ] Test automatic discovery
|
|
||||||
- [ ] Verify log flow from client to server
|
|
||||||
- [ ] Test reconnection scenarios
|
|
||||||
- [ ] Verify all log types are parsed correctly
|
|
||||||
- [ ] Test SSL communication
|
|
||||||
- [ ] Measure end-to-end latency
|
|
||||||
|
|
||||||
### Compatibility Testing
|
|
||||||
- [ ] Test with different Docker versions
|
|
||||||
- [ ] Test on various Linux distributions
|
|
||||||
- [ ] Verify with different log formats
|
|
||||||
- [ ] Test with high-volume producers
|
|
||||||
- [ ] Validate resource usage
|
|
||||||
|
|
||||||
## Phase 10: Production Readiness (Priority 4)
|
|
||||||
|
|
||||||
### Monitoring & Alerting
|
|
||||||
- [ ] Add Elasticsearch monitoring
|
|
||||||
- [ ] Configure disk space alerts
|
|
||||||
- [ ] Set up index health monitoring
|
|
||||||
- [ ] Add performance metrics collection
|
|
||||||
- [ ] Create alert rules in Kibana
|
|
||||||
|
|
||||||
### Maintenance Features
|
|
||||||
- [ ] Add automatic update check
|
|
||||||
- [ ] Create maintenance mode
|
|
||||||
- [ ] Add data export functionality
|
|
||||||
- [ ] Create migration scripts
|
|
||||||
- [ ] Add configuration validation
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
### Design Principles
|
|
||||||
1. **Minimum configuration**: Should work with just `dropshell install logserver`
|
|
||||||
2. **Data safety**: Never delete volumes in uninstall.sh
|
|
||||||
3. **Non-interactive**: All scripts must run without user input
|
|
||||||
4. **Idempotent**: Scripts can be run multiple times safely
|
|
||||||
5. **Clear feedback**: Provide clear status and error messages
|
|
||||||
|
|
||||||
### Dependencies
|
|
||||||
- Docker and Docker Compose
|
|
||||||
- Sufficient system resources (4GB+ RAM recommended)
|
|
||||||
- Network connectivity for clients
|
|
||||||
- Persistent storage for logs
|
|
||||||
|
|
||||||
### Testing Checklist
|
|
||||||
- [ ] All required scripts present and executable
|
|
||||||
- [ ] Template validates with dropshell test-template
|
|
||||||
- [ ] Services start and connect properly
|
|
||||||
- [ ] Logs flow from client to Kibana
|
|
||||||
- [ ] Data persists across container restarts
|
|
||||||
- [ ] Uninstall preserves data volumes
|
|
||||||
- [ ] Resource limits are enforced
|
|
||||||
- [ ] Error handling works correctly
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
# Define volume items for logserver container
|
|
||||||
# These are used across backup, restore, create, and destroy operations
|
|
||||||
|
|
||||||
get_logserver_volumes() {
|
|
||||||
echo "volume:elasticsearch_data:${CONTAINER_NAME}_elasticsearch_data" \
|
|
||||||
"volume:logstash_data:${CONTAINER_NAME}_logstash_data" \
|
|
||||||
"volume:kibana_data:${CONTAINER_NAME}_kibana_data"
|
|
||||||
}
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
source "${AGENT_PATH}/common.sh"
|
|
||||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
|
||||||
|
|
||||||
_check_required_env_vars "CONTAINER_NAME"
|
|
||||||
|
|
||||||
echo "Backing up ${CONTAINER_NAME} ELK stack volumes..."
|
|
||||||
echo "Note: This may take a while for large log databases"
|
|
||||||
|
|
||||||
# Stop the containers to ensure data consistency
|
|
||||||
bash ./stop.sh || true
|
|
||||||
|
|
||||||
# Backup volumes
|
|
||||||
BACKUP_DIR="${CONFIG_PATH}/backups/$(date +%Y%m%d_%H%M%S)"
|
|
||||||
mkdir -p "$BACKUP_DIR"
|
|
||||||
|
|
||||||
# Export volumes
|
|
||||||
for volume in $(get_logserver_volumes); do
|
|
||||||
volume_name=$(echo $volume | cut -d: -f3)
|
|
||||||
echo "Backing up volume: $volume_name"
|
|
||||||
docker run --rm -v "$volume_name:/source:ro" -v "$BACKUP_DIR:/backup" alpine \
|
|
||||||
tar -czf "/backup/${volume_name}.tar.gz" -C /source .
|
|
||||||
done
|
|
||||||
|
|
||||||
# Backup configuration
|
|
||||||
cp -r "${CONFIG_PATH}" "$BACKUP_DIR/config_backup"
|
|
||||||
|
|
||||||
# Backup docker-compose.yml
|
|
||||||
cp docker-compose.yml "$BACKUP_DIR/" 2>/dev/null || true
|
|
||||||
|
|
||||||
echo "Backup completed to: $BACKUP_DIR"
|
|
||||||
echo "Size: $(du -sh $BACKUP_DIR | cut -f1)"
|
|
||||||
|
|
||||||
# Restart the containers
|
|
||||||
bash ./start.sh
|
|
||||||
|
|
||||||
echo "ELK stack restarted"
|
|
||||||
@@ -1,17 +0,0 @@
|
|||||||
# Template identifier - MUST match the directory name
|
|
||||||
TEMPLATE=logserver
|
|
||||||
|
|
||||||
# Requirements
|
|
||||||
REQUIRES_HOST_ROOT=false # No root access on host needed
|
|
||||||
REQUIRES_DOCKER=true # Docker is required
|
|
||||||
REQUIRES_DOCKER_ROOT=false # Docker root privileges not specifically needed
|
|
||||||
|
|
||||||
# Docker compose used for ELK stack
|
|
||||||
USES_DOCKER_COMPOSE=true
|
|
||||||
|
|
||||||
# Volume definitions for persistence
|
|
||||||
DATA_VOLUME="${CONTAINER_NAME}_elasticsearch_data"
|
|
||||||
LOGSTASH_VOLUME="${CONTAINER_NAME}_logstash_data"
|
|
||||||
KIBANA_VOLUME="${CONTAINER_NAME}_kibana_data"
|
|
||||||
CERTS_VOLUME="${CONTAINER_NAME}_certs"
|
|
||||||
CONFIG_VOLUME="${CONTAINER_NAME}_config"
|
|
||||||
@@ -1,6 +0,0 @@
|
|||||||
# API Keys for LogServer Authentication
|
|
||||||
# Format: hostname:api_key
|
|
||||||
# Generated by generate-api-key.sh
|
|
||||||
|
|
||||||
api_keys:
|
|
||||||
video: a7798c63c2ac439b5ba20f3bf8bf27b5361231cdcbdc4fc9d7af715308fdf707
|
|
||||||
154
logserver/config/dashboards/central-logs.json
Normal file
154
logserver/config/dashboards/central-logs.json
Normal file
@@ -0,0 +1,154 @@
|
|||||||
|
{
|
||||||
|
"annotations": {
|
||||||
|
"list": [
|
||||||
|
{
|
||||||
|
"builtIn": 1,
|
||||||
|
"datasource": "-- Grafana --",
|
||||||
|
"enable": true,
|
||||||
|
"hide": true,
|
||||||
|
"iconColor": "rgba(0, 211, 255, 1)",
|
||||||
|
"name": "Annotations & Alerts",
|
||||||
|
"type": "dashboard"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"editable": true,
|
||||||
|
"gnetId": null,
|
||||||
|
"graphTooltip": 0,
|
||||||
|
"id": null,
|
||||||
|
"links": [],
|
||||||
|
"panels": [
|
||||||
|
{
|
||||||
|
"datasource": "Loki",
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"custom": {}
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 3,
|
||||||
|
"w": 24,
|
||||||
|
"x": 0,
|
||||||
|
"y": 0
|
||||||
|
},
|
||||||
|
"id": 4,
|
||||||
|
"options": {
|
||||||
|
"content": "## Central Log Viewer\n\n**Quick Filters:** Click any label (hostname, container_name, job) to filter | **Search:** Use the search box above each panel | **Time Range:** Top right corner",
|
||||||
|
"mode": "markdown"
|
||||||
|
},
|
||||||
|
"pluginVersion": "7.5.7",
|
||||||
|
"title": "",
|
||||||
|
"type": "text"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": "Loki",
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 10,
|
||||||
|
"w": 24,
|
||||||
|
"x": 0,
|
||||||
|
"y": 3
|
||||||
|
},
|
||||||
|
"id": 2,
|
||||||
|
"options": {
|
||||||
|
"dedupStrategy": "none",
|
||||||
|
"enableLogDetails": true,
|
||||||
|
"prettifyLogMessage": false,
|
||||||
|
"showCommonLabels": false,
|
||||||
|
"showLabels": true,
|
||||||
|
"showTime": true,
|
||||||
|
"sortOrder": "Descending",
|
||||||
|
"wrapLogMessage": true
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "{job=\"docker\"}",
|
||||||
|
"legendFormat": "{{hostname}} - {{container_name}}",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "Docker Container Logs (All Servers)",
|
||||||
|
"type": "logs"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": "Loki",
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 24,
|
||||||
|
"x": 0,
|
||||||
|
"y": 13
|
||||||
|
},
|
||||||
|
"id": 3,
|
||||||
|
"options": {
|
||||||
|
"dedupStrategy": "none",
|
||||||
|
"enableLogDetails": true,
|
||||||
|
"prettifyLogMessage": false,
|
||||||
|
"showCommonLabels": false,
|
||||||
|
"showLabels": true,
|
||||||
|
"showTime": true,
|
||||||
|
"sortOrder": "Descending",
|
||||||
|
"wrapLogMessage": true
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "{job=~\"syslog|auth\"}",
|
||||||
|
"legendFormat": "{{hostname}} - {{job}}",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "System Logs (All Servers)",
|
||||||
|
"type": "logs"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"refresh": "10s",
|
||||||
|
"schemaVersion": 27,
|
||||||
|
"style": "dark",
|
||||||
|
"tags": ["logs", "central"],
|
||||||
|
"templating": {
|
||||||
|
"list": [
|
||||||
|
{
|
||||||
|
"allValue": ".*",
|
||||||
|
"current": {
|
||||||
|
"selected": true,
|
||||||
|
"text": ["All"],
|
||||||
|
"value": ["$__all"]
|
||||||
|
},
|
||||||
|
"datasource": "Loki",
|
||||||
|
"definition": "label_values(hostname)",
|
||||||
|
"description": "Filter by server",
|
||||||
|
"error": null,
|
||||||
|
"hide": 0,
|
||||||
|
"includeAll": true,
|
||||||
|
"label": "Server",
|
||||||
|
"multi": true,
|
||||||
|
"name": "hostname",
|
||||||
|
"options": [],
|
||||||
|
"query": "label_values(hostname)",
|
||||||
|
"refresh": 1,
|
||||||
|
"regex": "",
|
||||||
|
"skipUrlSync": false,
|
||||||
|
"sort": 1,
|
||||||
|
"type": "query"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"time": {
|
||||||
|
"from": "now-15m",
|
||||||
|
"to": "now"
|
||||||
|
},
|
||||||
|
"timepicker": {
|
||||||
|
"refresh_intervals": ["5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h"]
|
||||||
|
},
|
||||||
|
"timezone": "",
|
||||||
|
"title": "Central Logs",
|
||||||
|
"uid": "central-logs",
|
||||||
|
"version": 0
|
||||||
|
}
|
||||||
69
logserver/config/dashboards/simple-logs.json
Normal file
69
logserver/config/dashboards/simple-logs.json
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
{
|
||||||
|
"annotations": {
|
||||||
|
"list": [
|
||||||
|
{
|
||||||
|
"builtIn": 1,
|
||||||
|
"datasource": "-- Grafana --",
|
||||||
|
"enable": true,
|
||||||
|
"hide": true,
|
||||||
|
"iconColor": "rgba(0, 211, 255, 1)",
|
||||||
|
"name": "Annotations & Alerts",
|
||||||
|
"type": "dashboard"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"editable": true,
|
||||||
|
"gnetId": null,
|
||||||
|
"graphTooltip": 0,
|
||||||
|
"id": null,
|
||||||
|
"links": [],
|
||||||
|
"panels": [
|
||||||
|
{
|
||||||
|
"datasource": "Loki",
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 20,
|
||||||
|
"w": 24,
|
||||||
|
"x": 0,
|
||||||
|
"y": 0
|
||||||
|
},
|
||||||
|
"id": 2,
|
||||||
|
"options": {
|
||||||
|
"dedupStrategy": "none",
|
||||||
|
"enableLogDetails": true,
|
||||||
|
"prettifyLogMessage": false,
|
||||||
|
"showCommonLabels": false,
|
||||||
|
"showLabels": true,
|
||||||
|
"showTime": true,
|
||||||
|
"sortOrder": "Descending",
|
||||||
|
"wrapLogMessage": true
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "{job=~\"docker|syslog|auth\"}",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "All Logs",
|
||||||
|
"type": "logs"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"schemaVersion": 27,
|
||||||
|
"style": "dark",
|
||||||
|
"tags": ["logs"],
|
||||||
|
"templating": {
|
||||||
|
"list": []
|
||||||
|
},
|
||||||
|
"time": {
|
||||||
|
"from": "now-15m",
|
||||||
|
"to": "now"
|
||||||
|
},
|
||||||
|
"timepicker": {},
|
||||||
|
"timezone": "",
|
||||||
|
"title": "Simple Logs",
|
||||||
|
"uid": "simple-logs",
|
||||||
|
"version": 0
|
||||||
|
}
|
||||||
12
logserver/config/grafana-dashboard.yml
Normal file
12
logserver/config/grafana-dashboard.yml
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
apiVersion: 1
|
||||||
|
|
||||||
|
providers:
|
||||||
|
- name: 'default'
|
||||||
|
orgId: 1
|
||||||
|
folder: ''
|
||||||
|
type: file
|
||||||
|
disableDeletion: false
|
||||||
|
updateIntervalSeconds: 10
|
||||||
|
allowUiUpdates: true
|
||||||
|
options:
|
||||||
|
path: /var/lib/grafana/dashboards
|
||||||
9
logserver/config/grafana-datasources.yml
Normal file
9
logserver/config/grafana-datasources.yml
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
apiVersion: 1
|
||||||
|
|
||||||
|
datasources:
|
||||||
|
- name: Loki
|
||||||
|
type: loki
|
||||||
|
access: proxy
|
||||||
|
url: http://loki:3100
|
||||||
|
isDefault: true
|
||||||
|
editable: false
|
||||||
@@ -1,142 +0,0 @@
|
|||||||
# Logstash Configuration for LogServer
|
|
||||||
# Handles Beats input with API key authentication
|
|
||||||
|
|
||||||
input {
|
|
||||||
# Beats input for Filebeat clients
|
|
||||||
beats {
|
|
||||||
port => 5044
|
|
||||||
ssl => false # Set to true for production with proper certificates
|
|
||||||
|
|
||||||
# API key authentication handled via filter below
|
|
||||||
}
|
|
||||||
|
|
||||||
# Optional: Syslog input for direct syslog shipping
|
|
||||||
tcp {
|
|
||||||
port => 514
|
|
||||||
type => "syslog"
|
|
||||||
}
|
|
||||||
|
|
||||||
udp {
|
|
||||||
port => 514
|
|
||||||
type => "syslog"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
filter {
|
|
||||||
# API Key validation - check if client provided a valid key
|
|
||||||
# The API key should be in the [fields][api_key] field from Filebeat
|
|
||||||
if [fields][api_key] {
|
|
||||||
# Load and validate API key
|
|
||||||
ruby {
|
|
||||||
init => "
|
|
||||||
require 'yaml'
|
|
||||||
@api_keys = {}
|
|
||||||
|
|
||||||
# Load API keys from file
|
|
||||||
begin
|
|
||||||
if File.exist?('/usr/share/logstash/config/api-keys.yml')
|
|
||||||
config = YAML.load_file('/usr/share/logstash/config/api-keys.yml')
|
|
||||||
if config && config['api_keys']
|
|
||||||
config['api_keys'].each do |hostname, key|
|
|
||||||
@api_keys[key.to_s.strip] = hostname.to_s.strip
|
|
||||||
end
|
|
||||||
end
|
|
||||||
end
|
|
||||||
rescue => e
|
|
||||||
@logger.error('Failed to load API keys', :error => e.message)
|
|
||||||
end
|
|
||||||
"
|
|
||||||
code => "
|
|
||||||
api_key = event.get('[fields][api_key]')
|
|
||||||
|
|
||||||
if api_key && @api_keys.has_key?(api_key)
|
|
||||||
# Valid API key - add hostname to event
|
|
||||||
event.set('[@metadata][client_hostname]', @api_keys[api_key])
|
|
||||||
event.set('[@metadata][authenticated]', true)
|
|
||||||
else
|
|
||||||
# Invalid API key
|
|
||||||
event.set('[@metadata][authenticated]', false)
|
|
||||||
event.tag('_authfailure')
|
|
||||||
end
|
|
||||||
"
|
|
||||||
}
|
|
||||||
|
|
||||||
# Drop unauthorized events
|
|
||||||
if "_authfailure" in [tags] {
|
|
||||||
drop { }
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
# No API key provided - mark as unauthenticated
|
|
||||||
# You can choose to drop these or allow them based on your security requirements
|
|
||||||
mutate {
|
|
||||||
add_tag => [ "no_api_key" ]
|
|
||||||
}
|
|
||||||
# Uncomment to require API keys for all connections:
|
|
||||||
# drop { }
|
|
||||||
}
|
|
||||||
|
|
||||||
# Parse Docker logs
|
|
||||||
if [docker] {
|
|
||||||
# Docker metadata is already parsed by Filebeat
|
|
||||||
mutate {
|
|
||||||
add_field => {
|
|
||||||
"container_name" => "%{[docker][container][name]}"
|
|
||||||
"container_id" => "%{[docker][container][id]}"
|
|
||||||
"container_image" => "%{[docker][container][image]}"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
# Parse syslog
|
|
||||||
if [type] == "syslog" {
|
|
||||||
grok {
|
|
||||||
match => {
|
|
||||||
"message" => "%{SYSLOGLINE}"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
date {
|
|
||||||
match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
# Parse JSON logs if they exist
|
|
||||||
if [message] =~ /^\{.*\}$/ {
|
|
||||||
json {
|
|
||||||
source => "message"
|
|
||||||
target => "json_message"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
# Add timestamp if not present
|
|
||||||
if ![timestamp] {
|
|
||||||
mutate {
|
|
||||||
add_field => { "timestamp" => "%{@timestamp}" }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
# Clean up metadata
|
|
||||||
mutate {
|
|
||||||
remove_field => [ "@version", "beat", "offset", "prospector" ]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
output {
|
|
||||||
# Send to Elasticsearch with authentication
|
|
||||||
elasticsearch {
|
|
||||||
hosts => ["elasticsearch:9200"]
|
|
||||||
user => "elastic"
|
|
||||||
password => "${ELASTIC_PASSWORD:changeme}"
|
|
||||||
|
|
||||||
# Use different indices based on input type
|
|
||||||
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
|
|
||||||
|
|
||||||
# Manage index templates
|
|
||||||
manage_template => true
|
|
||||||
template_overwrite => true
|
|
||||||
}
|
|
||||||
|
|
||||||
# Optional: Debug output (comment out in production)
|
|
||||||
# stdout {
|
|
||||||
# codec => rubydebug
|
|
||||||
# }
|
|
||||||
}
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
# Logstash Configuration Settings
|
|
||||||
# This file contains Logstash settings (not pipeline configuration)
|
|
||||||
|
|
||||||
# Node name
|
|
||||||
node.name: "logstash"
|
|
||||||
|
|
||||||
# Pipeline settings
|
|
||||||
pipeline.workers: 2
|
|
||||||
pipeline.batch.size: 125
|
|
||||||
pipeline.batch.delay: 50
|
|
||||||
pipeline.ecs_compatibility: disabled
|
|
||||||
|
|
||||||
# HTTP API settings
|
|
||||||
http.host: "0.0.0.0"
|
|
||||||
http.port: 9600
|
|
||||||
|
|
||||||
# Monitoring
|
|
||||||
monitoring.enabled: false
|
|
||||||
|
|
||||||
# Queue settings
|
|
||||||
queue.type: memory
|
|
||||||
queue.max_bytes: 1gb
|
|
||||||
|
|
||||||
# Path settings are handled by Docker volumes
|
|
||||||
# path.data: /usr/share/logstash/data
|
|
||||||
# path.logs: /usr/share/logstash/logs
|
|
||||||
|
|
||||||
# Log level
|
|
||||||
log.level: info
|
|
||||||
51
logserver/config/loki.yaml
Normal file
51
logserver/config/loki.yaml
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
auth_enabled: false
|
||||||
|
|
||||||
|
server:
|
||||||
|
http_listen_port: 3100
|
||||||
|
grpc_listen_port: 9096
|
||||||
|
|
||||||
|
ingester:
|
||||||
|
wal:
|
||||||
|
enabled: true
|
||||||
|
dir: /loki/wal
|
||||||
|
lifecycler:
|
||||||
|
address: 127.0.0.1
|
||||||
|
ring:
|
||||||
|
kvstore:
|
||||||
|
store: inmemory
|
||||||
|
replication_factor: 1
|
||||||
|
|
||||||
|
schema_config:
|
||||||
|
configs:
|
||||||
|
- from: 2020-10-24
|
||||||
|
store: boltdb-shipper
|
||||||
|
object_store: filesystem
|
||||||
|
schema: v11
|
||||||
|
index:
|
||||||
|
prefix: index_
|
||||||
|
period: 24h
|
||||||
|
|
||||||
|
storage_config:
|
||||||
|
boltdb_shipper:
|
||||||
|
active_index_directory: /loki/boltdb-shipper-active
|
||||||
|
cache_location: /loki/boltdb-shipper-cache
|
||||||
|
cache_ttl: 24h
|
||||||
|
filesystem:
|
||||||
|
directory: /loki/chunks
|
||||||
|
|
||||||
|
compactor:
|
||||||
|
working_directory: /loki/boltdb-shipper-compactor
|
||||||
|
|
||||||
|
limits_config:
|
||||||
|
enforce_metric_name: false
|
||||||
|
reject_old_samples: true
|
||||||
|
reject_old_samples_max_age: 168h
|
||||||
|
max_entries_limit_per_query: 5000
|
||||||
|
retention_period: 168h # 7 days
|
||||||
|
|
||||||
|
chunk_store_config:
|
||||||
|
max_look_back_period: 168h
|
||||||
|
|
||||||
|
table_manager:
|
||||||
|
retention_deletes_enabled: true
|
||||||
|
retention_period: 168h
|
||||||
30
logserver/config/nginx.conf
Normal file
30
logserver/config/nginx.conf
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
events {
|
||||||
|
worker_connections 1024;
|
||||||
|
}
|
||||||
|
|
||||||
|
http {
|
||||||
|
upstream loki {
|
||||||
|
server loki:3100;
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 80;
|
||||||
|
|
||||||
|
# Require authentication for all requests
|
||||||
|
auth_basic "Loki Authentication";
|
||||||
|
auth_basic_user_file /etc/nginx/.htpasswd;
|
||||||
|
|
||||||
|
# Proxy all requests to Loki
|
||||||
|
location / {
|
||||||
|
proxy_pass http://loki;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
|
||||||
|
# Important for Loki push endpoint
|
||||||
|
client_max_body_size 100M;
|
||||||
|
client_body_buffer_size 10M;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,46 +1,20 @@
|
|||||||
# Service identification
|
# Log Server Configuration (Loki + Grafana)
|
||||||
CONTAINER_NAME=logserver
|
CONTAINER_NAME=logserver
|
||||||
|
|
||||||
# Server settings (REQUIRED by dropshell)
|
# Server settings (REQUIRED by dropshell)
|
||||||
SSH_USER="root"
|
SSH_USER="root"
|
||||||
|
|
||||||
# Elasticsearch settings
|
|
||||||
ES_VERSION=7.17.23
|
|
||||||
ES_HEAP_SIZE=2g
|
|
||||||
ES_MAX_MAP_COUNT=262144
|
|
||||||
|
|
||||||
# Logstash settings
|
|
||||||
LS_VERSION=7.17.23
|
|
||||||
LS_HEAP_SIZE=1g
|
|
||||||
LS_PIPELINE_WORKERS=2
|
|
||||||
|
|
||||||
# Kibana settings
|
|
||||||
KIBANA_VERSION=7.17.23
|
|
||||||
|
|
||||||
# Authentication (IMPORTANT: Change these!)
|
|
||||||
ELASTIC_PASSWORD=changeme # Password for 'elastic' superuser (internal use)
|
|
||||||
KIBANA_USERNAME=admin # Your login username for Kibana
|
|
||||||
KIBANA_USER_PASSWORD=changeme # Your login password for Kibana
|
|
||||||
|
|
||||||
# Ports
|
# Ports
|
||||||
KIBANA_PORT=5601
|
WEB_PORT=3000 # Grafana web UI
|
||||||
LOGSTASH_BEATS_PORT=5044
|
LOKI_PORT=3100 # Loki API (for clients to send logs)
|
||||||
LOGSTASH_SYSLOG_PORT=514
|
|
||||||
|
|
||||||
# Server configuration
|
# Authentication for Grafana
|
||||||
SERVER_PUBLICBASEURL=http://localhost:5601 # Change to your server's actual URL
|
ADMIN_USER=admin
|
||||||
|
ADMIN_PASSWORD=changeme
|
||||||
|
|
||||||
# Log retention
|
# Authentication for Loki (clients must use these)
|
||||||
LOG_RETENTION_DAYS=30
|
LOKI_USER=logclient
|
||||||
LOG_MAX_SIZE_GB=50
|
LOKI_PASSWORD=changeme
|
||||||
|
|
||||||
# Authentication
|
# Log retention (days)
|
||||||
ENABLE_TLS=true
|
LOG_RETENTION=7
|
||||||
API_KEYS_FILE=${CONFIG_PATH}/api-keys.yml
|
|
||||||
|
|
||||||
# Network Security
|
|
||||||
ALLOWED_IPS="" # Comma-separated list, empty = all
|
|
||||||
|
|
||||||
# Resource limits
|
|
||||||
MAX_CPU_PERCENT=80
|
|
||||||
MAX_MEMORY=4GB
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
# Ruby script for Logstash to validate API keys
|
|
||||||
# This is a simplified validation - in production, use proper authentication
|
|
||||||
|
|
||||||
require 'yaml'
|
|
||||||
|
|
||||||
def register(params)
|
|
||||||
@api_keys_file = params["api_keys_file"]
|
|
||||||
end
|
|
||||||
|
|
||||||
def filter(event)
|
|
||||||
# Get the API key from the event
|
|
||||||
api_key = event.get("[api_key]") || event.get("[@metadata][api_key]")
|
|
||||||
|
|
||||||
# If no API key, pass through (for backwards compatibility)
|
|
||||||
# In production, you should reject events without valid keys
|
|
||||||
if api_key.nil? || api_key.empty?
|
|
||||||
# For now, allow events without API keys
|
|
||||||
# event.cancel # Uncomment to require API keys
|
|
||||||
return [event]
|
|
||||||
end
|
|
||||||
|
|
||||||
# Load API keys from file
|
|
||||||
begin
|
|
||||||
if File.exist?(@api_keys_file)
|
|
||||||
config = YAML.load_file(@api_keys_file)
|
|
||||||
valid_keys = config['api_keys'].values if config && config['api_keys']
|
|
||||||
|
|
||||||
# Check if the provided key is valid
|
|
||||||
if valid_keys && valid_keys.include?(api_key)
|
|
||||||
# Valid key - let the event through
|
|
||||||
event.set("[@metadata][authenticated]", true)
|
|
||||||
else
|
|
||||||
# Invalid key - drop the event
|
|
||||||
event.cancel
|
|
||||||
end
|
|
||||||
end
|
|
||||||
rescue => e
|
|
||||||
# Log error but don't crash
|
|
||||||
event.set("[@metadata][auth_error]", e.message)
|
|
||||||
end
|
|
||||||
|
|
||||||
return [event]
|
|
||||||
end
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
source "${AGENT_PATH}/common.sh"
|
|
||||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
|
||||||
|
|
||||||
_check_required_env_vars "CONTAINER_NAME"
|
|
||||||
|
|
||||||
echo "WARNING: This will PERMANENTLY DELETE all ELK stack data!"
|
|
||||||
echo "This includes all logs, dashboards, and configurations"
|
|
||||||
echo "This action cannot be undone!"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Since scripts must be non-interactive, only proceed if explicitly called
|
|
||||||
# Stop all containers
|
|
||||||
docker compose down || true
|
|
||||||
|
|
||||||
# Remove all volumes
|
|
||||||
for volume in $(get_logserver_volumes); do
|
|
||||||
volume_name=$(echo $volume | cut -d: -f3)
|
|
||||||
echo "Removing volume: $volume_name"
|
|
||||||
docker volume rm "$volume_name" 2>/dev/null || true
|
|
||||||
done
|
|
||||||
|
|
||||||
# Also remove the Docker network if it exists
|
|
||||||
docker network rm "${CONTAINER_NAME}_elk" 2>/dev/null || true
|
|
||||||
|
|
||||||
echo "ELK stack and all data destroyed"
|
|
||||||
@@ -1,80 +1,57 @@
|
|||||||
|
version: '3.8'
|
||||||
|
|
||||||
services:
|
services:
|
||||||
elasticsearch:
|
# Grafana - Simple Web UI
|
||||||
image: docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION:-7.17.23}
|
grafana:
|
||||||
container_name: ${CONTAINER_NAME}_elasticsearch
|
image: grafana/grafana:latest
|
||||||
|
container_name: ${CONTAINER_NAME}_grafana
|
||||||
environment:
|
environment:
|
||||||
- discovery.type=single-node
|
- GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
|
||||||
- "ES_JAVA_OPTS=-Xms${ES_HEAP_SIZE:-2g} -Xmx${ES_HEAP_SIZE:-2g}"
|
- GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-changeme}
|
||||||
- xpack.security.enabled=true
|
- GF_USERS_ALLOW_SIGN_UP=false
|
||||||
- xpack.security.authc.api_key.enabled=true
|
|
||||||
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD:-${KIBANA_PASSWORD:-changeme}}
|
|
||||||
- xpack.monitoring.enabled=false
|
|
||||||
- cluster.routing.allocation.disk.threshold_enabled=false
|
|
||||||
volumes:
|
volumes:
|
||||||
- elasticsearch_data:/usr/share/elasticsearch/data
|
- grafana_data:/var/lib/grafana
|
||||||
|
- ${CONFIG_PATH}/grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yaml:ro
|
||||||
|
- ${CONFIG_PATH}/grafana-dashboard.yml:/etc/grafana/provisioning/dashboards/dashboard.yaml:ro
|
||||||
|
- ${CONFIG_PATH}/dashboards:/var/lib/grafana/dashboards:ro
|
||||||
ports:
|
ports:
|
||||||
- "127.0.0.1:9200:9200"
|
- "${WEB_PORT:-3000}:3000"
|
||||||
networks:
|
|
||||||
- elk
|
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
ulimits:
|
|
||||||
memlock:
|
|
||||||
soft: -1
|
|
||||||
hard: -1
|
|
||||||
nofile:
|
|
||||||
soft: 65536
|
|
||||||
hard: 65536
|
|
||||||
|
|
||||||
logstash:
|
|
||||||
image: docker.elastic.co/logstash/logstash:${LS_VERSION:-7.17.23}
|
|
||||||
container_name: ${CONTAINER_NAME}_logstash
|
|
||||||
environment:
|
|
||||||
- "LS_JAVA_OPTS=-Xms${LS_HEAP_SIZE:-1g} -Xmx${LS_HEAP_SIZE:-1g}"
|
|
||||||
- "xpack.monitoring.enabled=false"
|
|
||||||
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD:-${KIBANA_PASSWORD:-changeme}}
|
|
||||||
- CONTAINER_NAME=${CONTAINER_NAME}
|
|
||||||
command: logstash -f /usr/share/logstash/config/logstash.conf
|
|
||||||
volumes:
|
|
||||||
- ${CONFIG_PATH}:/usr/share/logstash/config
|
|
||||||
- logstash_data:/usr/share/logstash/data
|
|
||||||
ports:
|
|
||||||
- "${LOGSTASH_BEATS_PORT:-5044}:5044"
|
|
||||||
- "${LOGSTASH_SYSLOG_PORT:-514}:514/tcp"
|
|
||||||
- "${LOGSTASH_SYSLOG_PORT:-514}:514/udp"
|
|
||||||
networks:
|
|
||||||
- elk
|
|
||||||
depends_on:
|
depends_on:
|
||||||
- elasticsearch
|
- loki
|
||||||
restart: unless-stopped
|
|
||||||
|
|
||||||
kibana:
|
|
||||||
image: docker.elastic.co/kibana/kibana:${KIBANA_VERSION:-7.17.23}
|
|
||||||
container_name: ${CONTAINER_NAME}_kibana
|
|
||||||
environment:
|
|
||||||
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
|
|
||||||
- ELASTICSEARCH_USERNAME=elastic
|
|
||||||
- ELASTICSEARCH_PASSWORD=${ELASTIC_PASSWORD:-${KIBANA_PASSWORD:-changeme}}
|
|
||||||
- XPACK_SECURITY_ENABLED=true
|
|
||||||
- NODE_OPTIONS=--openssl-legacy-provider
|
|
||||||
- SERVER_PUBLICBASEURL=${SERVER_PUBLICBASEURL:-http://localhost:5601}
|
|
||||||
volumes:
|
|
||||||
- kibana_data:/usr/share/kibana/data
|
|
||||||
ports:
|
|
||||||
- "${KIBANA_PORT:-5601}:5601"
|
|
||||||
networks:
|
networks:
|
||||||
- elk
|
- loki-net
|
||||||
depends_on:
|
|
||||||
- elasticsearch
|
# Loki - Log storage (not exposed directly)
|
||||||
|
loki:
|
||||||
|
image: grafana/loki:latest
|
||||||
|
container_name: ${CONTAINER_NAME}_loki
|
||||||
|
volumes:
|
||||||
|
- loki_data:/loki
|
||||||
|
- ${CONFIG_PATH}/loki.yaml:/etc/loki/local-config.yaml:ro
|
||||||
|
command: -config.file=/etc/loki/local-config.yaml
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
networks:
|
||||||
|
- loki-net
|
||||||
|
|
||||||
|
# Nginx - Authentication proxy for Loki
|
||||||
|
nginx:
|
||||||
|
image: nginx:alpine
|
||||||
|
container_name: ${CONTAINER_NAME}_nginx
|
||||||
|
volumes:
|
||||||
|
- ${CONFIG_PATH}/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||||
|
- ${CONFIG_PATH}/.htpasswd:/etc/nginx/.htpasswd:ro
|
||||||
|
ports:
|
||||||
|
- "${LOKI_PORT:-3100}:80"
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
- loki
|
||||||
|
networks:
|
||||||
|
- loki-net
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
elk:
|
loki-net:
|
||||||
driver: bridge
|
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
elasticsearch_data:
|
grafana_data:
|
||||||
name: ${CONTAINER_NAME}_elasticsearch_data
|
loki_data:
|
||||||
logstash_data:
|
|
||||||
name: ${CONTAINER_NAME}_logstash_data
|
|
||||||
kibana_data:
|
|
||||||
name: ${CONTAINER_NAME}_kibana_data
|
|
||||||
@@ -1,172 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Interactive API Key Generation Script for LogServer
|
|
||||||
# This script generates secure API keys and adds them to api-keys.yml
|
|
||||||
|
|
||||||
# Determine where to put the api-keys.yml file
|
|
||||||
determine_api_keys_location() {
|
|
||||||
# 1. If api-keys.yml already exists in current folder, use it
|
|
||||||
if [ -f "./api-keys.yml" ]; then
|
|
||||||
echo "./api-keys.yml"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
# 2. If service.env exists in current folder, put keys here
|
|
||||||
if [ -f "./service.env" ]; then
|
|
||||||
echo "./api-keys.yml"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
# 3. If config folder exists, put keys there
|
|
||||||
if [ -d "./config" ]; then
|
|
||||||
echo "./config/api-keys.yml"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
# No valid location found
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
# Try to determine location
|
|
||||||
if API_KEYS_FILE=$(determine_api_keys_location); then
|
|
||||||
: # Location found, continue
|
|
||||||
else
|
|
||||||
echo -e "${RED}Error: Cannot determine where to place api-keys.yml${NC}"
|
|
||||||
echo ""
|
|
||||||
echo "This script must be run from one of these locations:"
|
|
||||||
echo " 1. A deployed service directory (contains service.env)"
|
|
||||||
echo " 2. The logserver template directory (contains config/ folder)"
|
|
||||||
echo " 3. A directory with existing api-keys.yml file"
|
|
||||||
echo ""
|
|
||||||
echo "Current directory: $(pwd)"
|
|
||||||
echo "Contents: $(ls -la 2>/dev/null | head -5)"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Colors for output
|
|
||||||
RED='\033[0;31m'
|
|
||||||
GREEN='\033[0;32m'
|
|
||||||
YELLOW='\033[1;33m'
|
|
||||||
NC='\033[0m' # No Color
|
|
||||||
|
|
||||||
# Generate a secure random API key
|
|
||||||
generate_key() {
|
|
||||||
openssl rand -hex 32 2>/dev/null || cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1
|
|
||||||
}
|
|
||||||
|
|
||||||
# Initialize api-keys.yml if it doesn't exist
|
|
||||||
init_api_keys_file() {
|
|
||||||
if [ ! -f "$API_KEYS_FILE" ]; then
|
|
||||||
# Create directory if needed
|
|
||||||
local dir=$(dirname "$API_KEYS_FILE")
|
|
||||||
if [ ! -d "$dir" ]; then
|
|
||||||
mkdir -p "$dir"
|
|
||||||
echo -e "${GREEN}Created directory: $dir${NC}"
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "# API Keys for LogServer Authentication" > "$API_KEYS_FILE"
|
|
||||||
echo "# Format: hostname:api_key" >> "$API_KEYS_FILE"
|
|
||||||
echo "# Generated by generate-api-key.sh" >> "$API_KEYS_FILE"
|
|
||||||
echo "" >> "$API_KEYS_FILE"
|
|
||||||
echo "api_keys:" >> "$API_KEYS_FILE"
|
|
||||||
echo -e "${GREEN}Created new api-keys.yml file at: $API_KEYS_FILE${NC}"
|
|
||||||
else
|
|
||||||
echo -e "${GREEN}Using existing api-keys.yml at: $API_KEYS_FILE${NC}"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
# Check if hostname already has a key
|
|
||||||
check_existing_key() {
|
|
||||||
local hostname=$1
|
|
||||||
if grep -q "^ ${hostname}:" "$API_KEYS_FILE" 2>/dev/null; then
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
# Add key to api-keys.yml
|
|
||||||
add_key_to_file() {
|
|
||||||
local hostname=$1
|
|
||||||
local api_key=$2
|
|
||||||
echo " ${hostname}: ${api_key}" >> "$API_KEYS_FILE"
|
|
||||||
}
|
|
||||||
|
|
||||||
# Main script
|
|
||||||
echo -e "${GREEN}=== LogServer API Key Generator ===${NC}"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Initialize file if needed
|
|
||||||
init_api_keys_file
|
|
||||||
|
|
||||||
# Interactive mode
|
|
||||||
while true; do
|
|
||||||
echo -e "${YELLOW}Enter hostname for the client (or 'done' to finish):${NC}"
|
|
||||||
read -p "> " hostname
|
|
||||||
|
|
||||||
if [ "$hostname" = "done" ] || [ -z "$hostname" ]; then
|
|
||||||
break
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Validate hostname - allow simple names, must start with alphanumeric
|
|
||||||
if [[ ! "$hostname" =~ ^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$ ]]; then
|
|
||||||
echo -e "${RED}Invalid hostname format. Use only letters, numbers, dots, dashes, and underscores.${NC}"
|
|
||||||
echo -e "${RED}Hostname must start and end with a letter or number.${NC}"
|
|
||||||
continue
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check if key already exists
|
|
||||||
if check_existing_key "$hostname"; then
|
|
||||||
echo -e "${YELLOW}Key already exists for ${hostname}${NC}"
|
|
||||||
read -p "Generate new key? (y/n): " overwrite
|
|
||||||
if [ "$overwrite" != "y" ]; then
|
|
||||||
continue
|
|
||||||
fi
|
|
||||||
# Remove old key
|
|
||||||
sed -i "/^ ${hostname}:/d" "$API_KEYS_FILE"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Generate new key
|
|
||||||
api_key=$(generate_key)
|
|
||||||
|
|
||||||
# Add to file
|
|
||||||
add_key_to_file "$hostname" "$api_key"
|
|
||||||
|
|
||||||
echo -e "${GREEN}✓ Generated API key for ${hostname}${NC}"
|
|
||||||
echo ""
|
|
||||||
echo "Configuration for ${hostname}:"
|
|
||||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
|
||||||
echo "Add to client's service.env:"
|
|
||||||
echo ""
|
|
||||||
echo "LOGSERVER_HOST=$(hostname -I | awk '{print $1}')"
|
|
||||||
echo "LOGSERVER_PORT=5044"
|
|
||||||
echo "API_KEY=${api_key}"
|
|
||||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Option to add more
|
|
||||||
read -p "Add another client? (y/n): " add_more
|
|
||||||
if [ "$add_more" != "y" ]; then
|
|
||||||
break
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# Show summary
|
|
||||||
echo ""
|
|
||||||
echo -e "${GREEN}=== Summary ===${NC}"
|
|
||||||
echo "API keys file: $API_KEYS_FILE"
|
|
||||||
echo "Total clients configured: $(grep -c "^ " "$API_KEYS_FILE" 2>/dev/null || echo 0)"
|
|
||||||
echo ""
|
|
||||||
echo "To view all keys: cat $API_KEYS_FILE"
|
|
||||||
echo "To revoke a key: Edit $API_KEYS_FILE and remove the line"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Show location-specific restart instructions
|
|
||||||
if [[ "$API_KEYS_FILE" == "./api-keys.yml" ]] && [ -f "./service.env" ]; then
|
|
||||||
# We're in a deployed service directory
|
|
||||||
echo -e "${YELLOW}Remember to restart the service to apply changes:${NC}"
|
|
||||||
echo " dropshell restart logserver"
|
|
||||||
else
|
|
||||||
# We're in the template directory
|
|
||||||
echo -e "${YELLOW}Note: Deploy this template to use these keys:${NC}"
|
|
||||||
echo " dropshell install logserver"
|
|
||||||
fi
|
|
||||||
@@ -2,128 +2,48 @@
|
|||||||
source "${AGENT_PATH}/common.sh"
|
source "${AGENT_PATH}/common.sh"
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
# Check required environment variables
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
_check_required_env_vars "CONTAINER_NAME" "ES_VERSION" "LS_VERSION" "KIBANA_VERSION"
|
|
||||||
|
|
||||||
# Check Docker and Docker Compose are available
|
# Check Docker
|
||||||
_check_docker_installed || _die "Docker test failed"
|
_check_docker_installed || _die "Docker test failed"
|
||||||
docker compose version >/dev/null 2>&1 || _die "Docker Compose is not installed (requires Docker Compose V2)"
|
docker compose version >/dev/null 2>&1 || _die "Docker Compose V2 is required"
|
||||||
|
|
||||||
# Check vm.max_map_count for Elasticsearch
|
|
||||||
current_max_map_count=$(sysctl -n vm.max_map_count 2>/dev/null || echo 0)
|
|
||||||
if [ "$current_max_map_count" -lt 262144 ]; then
|
|
||||||
echo "WARNING: vm.max_map_count is too low ($current_max_map_count)"
|
|
||||||
echo "Elasticsearch requires at least 262144"
|
|
||||||
echo "Please run: sudo sysctl -w vm.max_map_count=262144"
|
|
||||||
echo "And add to /etc/sysctl.conf to persist"
|
|
||||||
_die "System configuration needs adjustment"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check available memory
|
|
||||||
available_mem=$(free -m | awk '/^Mem:/{print $7}')
|
|
||||||
if [ "$available_mem" -lt 3000 ]; then
|
|
||||||
echo "WARNING: Low available memory (${available_mem}MB)"
|
|
||||||
echo "ELK stack requires at least 3-4GB free memory for proper operation"
|
|
||||||
echo "Services may take longer to start or fail to start"
|
|
||||||
echo ""
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Stop any existing containers
|
# Stop any existing containers
|
||||||
bash ./stop.sh || true
|
bash ./stop.sh 2>/dev/null || true
|
||||||
|
|
||||||
# Remove old containers
|
# Create config directory
|
||||||
docker compose down --remove-orphans 2>/dev/null || true
|
mkdir -p "${CONFIG_PATH}/dashboards"
|
||||||
|
|
||||||
# Pull the Docker images
|
# Copy configuration files
|
||||||
echo "Pulling ELK stack images..."
|
cp "$SCRIPT_DIR/config/"*.yaml "$SCRIPT_DIR/config/"*.yml "$SCRIPT_DIR/config/"*.conf "${CONFIG_PATH}/" 2>/dev/null || true
|
||||||
docker pull docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION} || _die "Failed to pull Elasticsearch"
|
cp "$SCRIPT_DIR/config/dashboards/"*.json "${CONFIG_PATH}/dashboards/" 2>/dev/null || true
|
||||||
docker pull docker.elastic.co/logstash/logstash:${LS_VERSION} || _die "Failed to pull Logstash"
|
|
||||||
docker pull docker.elastic.co/kibana/kibana:${KIBANA_VERSION} || _die "Failed to pull Kibana"
|
|
||||||
|
|
||||||
# Create volumes using common function
|
# Generate htpasswd file for Loki authentication
|
||||||
source "$SCRIPT_DIR/_volumes.sh"
|
echo "Generating authentication file..."
|
||||||
echo "Creating volumes..."
|
# Use openssl to generate htpasswd (available on most systems)
|
||||||
create_items $(get_logserver_volumes)
|
if command -v openssl >/dev/null 2>&1; then
|
||||||
|
# Generate password hash
|
||||||
# Ensure config directory exists
|
PASS_HASH=$(openssl passwd -apr1 "${LOKI_PASSWORD:-changeme}")
|
||||||
mkdir -p "${CONFIG_PATH}"
|
echo "${LOKI_USER:-logclient}:$PASS_HASH" > "${CONFIG_PATH}/.htpasswd"
|
||||||
|
elif command -v htpasswd >/dev/null 2>&1; then
|
||||||
# Initialize API keys file if it doesn't exist
|
# Use htpasswd if available
|
||||||
if [ ! -f "${CONFIG_PATH}/api-keys.yml" ]; then
|
htpasswd -cb "${CONFIG_PATH}/.htpasswd" "${LOKI_USER:-logclient}" "${LOKI_PASSWORD:-changeme}"
|
||||||
echo "No API keys configured yet."
|
else
|
||||||
echo "Run ./generate-api-key.sh to add client keys"
|
echo "WARNING: Cannot generate password file - no openssl or htpasswd found"
|
||||||
echo "api_keys:" > "${CONFIG_PATH}/api-keys.yml"
|
echo "Using basic auth with plain text (NOT SECURE):"
|
||||||
|
echo "${LOKI_USER:-logclient}:${LOKI_PASSWORD:-changeme}" > "${CONFIG_PATH}/.htpasswd"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Copy Logstash configurations if they don't exist
|
# Start the stack
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
echo "Starting Log Server..."
|
||||||
|
docker compose up -d || _die "Failed to start"
|
||||||
|
|
||||||
if [ ! -f "${CONFIG_PATH}/logstash.conf" ]; then
|
# Wait for services
|
||||||
if [ -f "$SCRIPT_DIR/config/logstash.conf" ]; then
|
echo -n "Waiting for services to start..."
|
||||||
cp "$SCRIPT_DIR/config/logstash.conf" "${CONFIG_PATH}/logstash.conf"
|
MAX_WAIT=60
|
||||||
echo "Copied Logstash pipeline configuration to ${CONFIG_PATH}"
|
|
||||||
else
|
|
||||||
echo "WARNING: logstash.conf not found in template"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ ! -f "${CONFIG_PATH}/logstash.yml" ]; then
|
|
||||||
if [ -f "$SCRIPT_DIR/config/logstash.yml" ]; then
|
|
||||||
cp "$SCRIPT_DIR/config/logstash.yml" "${CONFIG_PATH}/logstash.yml"
|
|
||||||
echo "Copied Logstash settings to ${CONFIG_PATH}"
|
|
||||||
else
|
|
||||||
echo "WARNING: logstash.yml not found in template"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Start the ELK stack
|
|
||||||
echo "Starting ELK stack..."
|
|
||||||
docker compose up -d --build || _die "Failed to start ELK stack"
|
|
||||||
|
|
||||||
# Wait for services to be ready with polling
|
|
||||||
echo "Waiting for services to start (this can take 2-3 minutes on first run)..."
|
|
||||||
MAX_WAIT=240 # Maximum 4 minutes
|
|
||||||
WAITED=0
|
WAITED=0
|
||||||
while [ $WAITED -lt $MAX_WAIT ]; do
|
while [ $WAITED -lt $MAX_WAIT ]; do
|
||||||
# Check if all services are running
|
if curl -s "http://localhost:${WEB_PORT:-3000}/api/health" 2>/dev/null | grep -q "ok"; then
|
||||||
if bash ./status.sh 2>/dev/null | grep -q "Running"; then
|
|
||||||
echo " All services are up!"
|
|
||||||
break
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Show progress with time elapsed
|
|
||||||
if [ $((WAITED % 10)) -eq 0 ] && [ $WAITED -gt 0 ]; then
|
|
||||||
echo -n " [${WAITED}s]"
|
|
||||||
else
|
|
||||||
echo -n "."
|
|
||||||
fi
|
|
||||||
|
|
||||||
sleep 2
|
|
||||||
WAITED=$((WAITED + 2))
|
|
||||||
done
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
if [ $WAITED -ge $MAX_WAIT ]; then
|
|
||||||
echo "Warning: Services are still starting. This is normal on first run."
|
|
||||||
echo "Current status:"
|
|
||||||
bash ./status.sh || true
|
|
||||||
echo ""
|
|
||||||
echo "You can check the logs with:"
|
|
||||||
echo " docker logs ${CONTAINER_NAME}_elasticsearch"
|
|
||||||
echo " docker logs ${CONTAINER_NAME}_logstash"
|
|
||||||
echo " docker logs ${CONTAINER_NAME}_kibana"
|
|
||||||
echo ""
|
|
||||||
echo "The services will continue starting in the background."
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Create custom user
|
|
||||||
echo "Setting up custom user '${KIBANA_USERNAME:-admin}'..."
|
|
||||||
echo -n "Waiting for Elasticsearch API..."
|
|
||||||
|
|
||||||
# First wait for Elasticsearch to be ready
|
|
||||||
WAITED=0
|
|
||||||
while [ $WAITED -lt 60 ]; do
|
|
||||||
if docker exec ${CONTAINER_NAME}_elasticsearch curl -s -u elastic:${ELASTIC_PASSWORD} http://localhost:9200/_cluster/health 2>/dev/null | grep -q '"status":"yellow"\|"status":"green"'; then
|
|
||||||
echo " Ready!"
|
echo " Ready!"
|
||||||
break
|
break
|
||||||
fi
|
fi
|
||||||
@@ -131,56 +51,28 @@ while [ $WAITED -lt 60 ]; do
|
|||||||
sleep 2
|
sleep 2
|
||||||
WAITED=$((WAITED + 2))
|
WAITED=$((WAITED + 2))
|
||||||
done
|
done
|
||||||
|
echo ""
|
||||||
|
|
||||||
if [ $WAITED -lt 60 ]; then
|
|
||||||
# Now create the user
|
|
||||||
docker exec ${CONTAINER_NAME}_elasticsearch bash -c "
|
|
||||||
|
|
||||||
result=\$(curl -s -X POST -u elastic:${ELASTIC_PASSWORD} \
|
|
||||||
-H 'Content-Type: application/json' \
|
|
||||||
http://localhost:9200/_security/user/${KIBANA_USERNAME:-admin} \
|
|
||||||
-d '{
|
|
||||||
\"password\" : \"${KIBANA_USER_PASSWORD:-changeme}\",
|
|
||||||
\"roles\" : [ \"superuser\" ],
|
|
||||||
\"full_name\" : \"Admin User\"
|
|
||||||
}' 2>/dev/null)
|
|
||||||
|
|
||||||
if echo \"\$result\" | grep -q '\"created\":true'; then
|
|
||||||
echo \"User created successfully\"
|
|
||||||
else
|
|
||||||
echo \"User already exists (this is fine)\"
|
|
||||||
fi
|
|
||||||
"
|
|
||||||
else
|
|
||||||
echo "Warning: Elasticsearch API not ready after 60 seconds"
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "Installation of ${CONTAINER_NAME} complete"
|
|
||||||
echo ""
|
echo ""
|
||||||
echo "========================================="
|
echo "========================================="
|
||||||
echo "Kibana UI: ${SERVER_PUBLICBASEURL:-http://$(hostname -I | awk '{print $1}'):${KIBANA_PORT}}"
|
echo "Log Server Installed!"
|
||||||
echo ""
|
|
||||||
echo "Login with your custom user:"
|
|
||||||
echo " Username: ${KIBANA_USERNAME:-admin}"
|
|
||||||
echo " Password: ${KIBANA_USER_PASSWORD:-changeme}"
|
|
||||||
echo ""
|
|
||||||
echo "Or the superuser:"
|
|
||||||
echo " Username: elastic"
|
|
||||||
echo " Password: ${ELASTIC_PASSWORD:-changeme}"
|
|
||||||
echo "========================================="
|
echo "========================================="
|
||||||
echo ""
|
echo ""
|
||||||
# Only show reminder if using default values
|
echo "Web UI: http://$(hostname -I | awk '{print $1}'):${WEB_PORT:-3000}"
|
||||||
if [ "${ELASTIC_PASSWORD}" = "changeme" ] || [ -z "${SERVER_PUBLICBASEURL}" ] || [ "${SERVER_PUBLICBASEURL}" = "http://localhost:5601" ]; then
|
echo "Login: ${ADMIN_USER:-admin} / ${ADMIN_PASSWORD:-changeme}"
|
||||||
echo "REMINDER: Update service.env with:"
|
|
||||||
if [ "${ELASTIC_PASSWORD}" = "changeme" ]; then
|
|
||||||
echo " - A secure password in ELASTIC_PASSWORD"
|
|
||||||
fi
|
|
||||||
if [ -z "${SERVER_PUBLICBASEURL}" ] || [ "${SERVER_PUBLICBASEURL}" = "http://localhost:5601" ]; then
|
|
||||||
echo " - Your actual server IP/domain in SERVER_PUBLICBASEURL"
|
|
||||||
fi
|
|
||||||
echo ""
|
|
||||||
fi
|
|
||||||
echo "Logstash listening on port ${LOGSTASH_BEATS_PORT} for Filebeat clients"
|
|
||||||
echo ""
|
echo ""
|
||||||
echo "To add client authentication:"
|
echo "TO VIEW LOGS:"
|
||||||
echo " ./generate-api-key.sh"
|
echo "1. Click 'Dashboards' (4 squares icon)"
|
||||||
|
echo "2. Click 'Central Logs'"
|
||||||
|
echo "3. See all logs from all servers!"
|
||||||
|
echo ""
|
||||||
|
echo "FOR CLIENTS TO SEND LOGS HERE:"
|
||||||
|
echo "Server: $(hostname -I | awk '{print $1}')"
|
||||||
|
echo "Port: ${LOKI_PORT:-3100}"
|
||||||
|
echo "Username: ${LOKI_USER:-logclient}"
|
||||||
|
echo "Password: ${LOKI_PASSWORD:-changeme}"
|
||||||
|
echo ""
|
||||||
|
echo "IMPORTANT: Change LOKI_PASSWORD in service.env!"
|
||||||
|
echo ""
|
||||||
|
echo "Install 'logclient' on other servers"
|
||||||
|
echo "========================================="
|
||||||
@@ -1,56 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
source "${AGENT_PATH}/common.sh"
|
|
||||||
source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh"
|
|
||||||
|
|
||||||
_check_required_env_vars "CONTAINER_NAME"
|
|
||||||
|
|
||||||
if [ -z "$1" ]; then
|
|
||||||
echo "Usage: $0 <backup_directory>"
|
|
||||||
echo "Available backups:"
|
|
||||||
ls -la "${CONFIG_PATH}/backups/" 2>/dev/null || echo "No backups found"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
BACKUP_DIR="$1"
|
|
||||||
|
|
||||||
if [ ! -d "$BACKUP_DIR" ]; then
|
|
||||||
_die "Backup directory not found: $BACKUP_DIR"
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "Restoring ELK stack from backup: $BACKUP_DIR"
|
|
||||||
echo "WARNING: This will overwrite all current data!"
|
|
||||||
|
|
||||||
# Stop the containers
|
|
||||||
bash ./stop.sh || true
|
|
||||||
|
|
||||||
# Restore volumes
|
|
||||||
for volume in $(get_logserver_volumes); do
|
|
||||||
volume_name=$(echo $volume | cut -d: -f3)
|
|
||||||
backup_file="$BACKUP_DIR/${volume_name}.tar.gz"
|
|
||||||
|
|
||||||
if [ -f "$backup_file" ]; then
|
|
||||||
echo "Restoring volume: $volume_name"
|
|
||||||
# Clear existing data and restore
|
|
||||||
docker run --rm -v "$volume_name:/target" -v "$BACKUP_DIR:/backup:ro" alpine \
|
|
||||||
sh -c "rm -rf /target/* && tar -xzf /backup/${volume_name}.tar.gz -C /target"
|
|
||||||
else
|
|
||||||
echo "Warning: Backup file not found for $volume_name"
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# Restore configuration if exists
|
|
||||||
if [ -d "$BACKUP_DIR/config_backup" ]; then
|
|
||||||
cp -r "$BACKUP_DIR/config_backup/"* "${CONFIG_PATH}/"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Restore docker-compose.yml if exists
|
|
||||||
if [ -f "$BACKUP_DIR/docker-compose.yml" ]; then
|
|
||||||
cp "$BACKUP_DIR/docker-compose.yml" .
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "Restore completed"
|
|
||||||
|
|
||||||
# Start the containers
|
|
||||||
bash ./start.sh
|
|
||||||
|
|
||||||
echo "ELK stack restarted with restored data"
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Script to create a custom user for Kibana login
|
|
||||||
# This runs after Elasticsearch is up and creates the user defined in service.env
|
|
||||||
|
|
||||||
# Wait for Elasticsearch to be ready
|
|
||||||
echo "Waiting for Elasticsearch to be ready..."
|
|
||||||
until curl -s -u elastic:${ELASTIC_PASSWORD} http://localhost:9200/_cluster/health | grep -q '"status":"yellow"\|"status":"green"'; do
|
|
||||||
sleep 5
|
|
||||||
done
|
|
||||||
|
|
||||||
echo "Creating user '${KIBANA_USERNAME}'..."
|
|
||||||
|
|
||||||
# Create the custom user with superuser role
|
|
||||||
curl -X POST -u elastic:${ELASTIC_PASSWORD} \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
http://localhost:9200/_security/user/${KIBANA_USERNAME} \
|
|
||||||
-d '{
|
|
||||||
"password" : "'${KIBANA_USER_PASSWORD}'",
|
|
||||||
"roles" : [ "superuser" ],
|
|
||||||
"full_name" : "Admin User",
|
|
||||||
"email" : "admin@example.com"
|
|
||||||
}'
|
|
||||||
|
|
||||||
if [ $? -eq 0 ]; then
|
|
||||||
echo ""
|
|
||||||
echo "User '${KIBANA_USERNAME}' created successfully!"
|
|
||||||
echo "You can now log in to Kibana with:"
|
|
||||||
echo " Username: ${KIBANA_USERNAME}"
|
|
||||||
echo " Password: ${KIBANA_USER_PASSWORD}"
|
|
||||||
else
|
|
||||||
echo "Note: User might already exist or there was an error"
|
|
||||||
fi
|
|
||||||
@@ -1,108 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Setup Kibana for simple log viewing
|
|
||||||
# This creates index patterns and saved searches for easy log access
|
|
||||||
|
|
||||||
source "${AGENT_PATH}/common.sh"
|
|
||||||
_check_required_env_vars "CONTAINER_NAME" "ELASTIC_PASSWORD" "KIBANA_PORT"
|
|
||||||
|
|
||||||
KIBANA_URL="http://localhost:${KIBANA_PORT}"
|
|
||||||
AUTH="elastic:${ELASTIC_PASSWORD}"
|
|
||||||
|
|
||||||
echo "Setting up Kibana for simple log viewing..."
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Wait for Kibana to be ready
|
|
||||||
echo -n "Waiting for Kibana to be ready..."
|
|
||||||
MAX_WAIT=60
|
|
||||||
WAITED=0
|
|
||||||
while [ $WAITED -lt $MAX_WAIT ]; do
|
|
||||||
if docker exec ${CONTAINER_NAME}_kibana curl -s -u "$AUTH" "${KIBANA_URL}/api/status" 2>/dev/null | grep -q '"level":"available"'; then
|
|
||||||
echo " Ready!"
|
|
||||||
break
|
|
||||||
fi
|
|
||||||
echo -n "."
|
|
||||||
sleep 2
|
|
||||||
WAITED=$((WAITED + 2))
|
|
||||||
done
|
|
||||||
|
|
||||||
if [ $WAITED -ge $MAX_WAIT ]; then
|
|
||||||
echo ""
|
|
||||||
echo "ERROR: Kibana is not ready after ${MAX_WAIT} seconds"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Create index pattern for Filebeat
|
|
||||||
echo "Creating Filebeat index pattern..."
|
|
||||||
docker exec ${CONTAINER_NAME}_kibana curl -s -X POST \
|
|
||||||
-u "$AUTH" \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-H "kbn-xsrf: true" \
|
|
||||||
"${KIBANA_URL}/api/saved_objects/index-pattern/filebeat-*" \
|
|
||||||
-d '{
|
|
||||||
"attributes": {
|
|
||||||
"title": "filebeat-*",
|
|
||||||
"timeFieldName": "@timestamp",
|
|
||||||
"fields": "[]"
|
|
||||||
}
|
|
||||||
}' > /dev/null 2>&1
|
|
||||||
|
|
||||||
# Set as default index pattern
|
|
||||||
docker exec ${CONTAINER_NAME}_kibana curl -s -X POST \
|
|
||||||
-u "$AUTH" \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-H "kbn-xsrf: true" \
|
|
||||||
"${KIBANA_URL}/api/kibana/settings" \
|
|
||||||
-d '{
|
|
||||||
"changes": {
|
|
||||||
"defaultIndex": "filebeat-*"
|
|
||||||
}
|
|
||||||
}' > /dev/null 2>&1
|
|
||||||
|
|
||||||
# Create a simple saved search for Docker logs
|
|
||||||
echo "Creating saved searches..."
|
|
||||||
docker exec ${CONTAINER_NAME}_kibana curl -s -X POST \
|
|
||||||
-u "$AUTH" \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-H "kbn-xsrf: true" \
|
|
||||||
"${KIBANA_URL}/api/saved_objects/search" \
|
|
||||||
-d '{
|
|
||||||
"attributes": {
|
|
||||||
"title": "Docker Container Logs",
|
|
||||||
"description": "View all Docker container logs",
|
|
||||||
"columns": ["container_name", "message"],
|
|
||||||
"sort": ["@timestamp", "desc"],
|
|
||||||
"kibanaSavedObjectMeta": {
|
|
||||||
"searchSourceJSON": "{\"index\":\"filebeat-*\",\"query\":{\"match_all\":{}},\"filter\":[{\"meta\":{\"index\":\"filebeat-*\",\"negate\":false,\"disabled\":false,\"alias\":null,\"type\":\"exists\",\"key\":\"container_name\",\"value\":\"exists\"},\"exists\":{\"field\":\"container_name\"}}]}"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}' > /dev/null 2>&1
|
|
||||||
|
|
||||||
echo ""
|
|
||||||
echo "========================================="
|
|
||||||
echo "Kibana Setup Complete!"
|
|
||||||
echo "========================================="
|
|
||||||
echo ""
|
|
||||||
echo "QUICK START GUIDE:"
|
|
||||||
echo ""
|
|
||||||
echo "1. Open Kibana: ${SERVER_PUBLICBASEURL:-http://$(hostname -I | awk '{print $1}'):${KIBANA_PORT}}"
|
|
||||||
echo ""
|
|
||||||
echo "2. Login with:"
|
|
||||||
echo " Username: ${KIBANA_USERNAME:-elastic}"
|
|
||||||
echo " Password: [your password]"
|
|
||||||
echo ""
|
|
||||||
echo "3. TO VIEW LOGS SIMPLY:"
|
|
||||||
echo " a) Click 'Discover' in the left menu"
|
|
||||||
echo " b) Time range is in top-right (set to 'Last 15 minutes' or 'Today')"
|
|
||||||
echo " c) Your logs will appear below"
|
|
||||||
echo ""
|
|
||||||
echo "4. TO FILTER LOGS:"
|
|
||||||
echo " - By container: Click '+' next to any 'container_name' value"
|
|
||||||
echo " - By host: Click '+' next to any 'host.name' value"
|
|
||||||
echo " - Search box: Type keywords to search all logs"
|
|
||||||
echo ""
|
|
||||||
echo "5. TO VIEW LIVE LOGS:"
|
|
||||||
echo " - Click the 'Refresh' button in top-right"
|
|
||||||
echo " - Set it to refresh every 5 seconds"
|
|
||||||
echo ""
|
|
||||||
echo "========================================="
|
|
||||||
@@ -2,16 +2,5 @@
|
|||||||
source "${AGENT_PATH}/common.sh"
|
source "${AGENT_PATH}/common.sh"
|
||||||
_check_required_env_vars "CONTAINER_NAME"
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
echo "Starting ELK stack..."
|
docker compose up -d || _die "Failed to start"
|
||||||
docker compose up -d || _die "Failed to start ELK stack"
|
echo "Log Server started"
|
||||||
|
|
||||||
# Wait for services to be ready
|
|
||||||
echo "Waiting for services to start..."
|
|
||||||
sleep 5
|
|
||||||
|
|
||||||
# Check if services are running
|
|
||||||
if docker compose ps | grep -q "Up"; then
|
|
||||||
echo "ELK stack started successfully"
|
|
||||||
else
|
|
||||||
_die "Failed to start ELK stack services"
|
|
||||||
fi
|
|
||||||
@@ -2,38 +2,11 @@
|
|||||||
source "${AGENT_PATH}/common.sh"
|
source "${AGENT_PATH}/common.sh"
|
||||||
_check_required_env_vars "CONTAINER_NAME"
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
# For verbose mode (when called directly, not from install script)
|
if docker ps | grep -q "${CONTAINER_NAME}_grafana"; then
|
||||||
if [ "$1" = "-v" ] || [ "$1" = "--verbose" ]; then
|
|
||||||
VERBOSE=true
|
|
||||||
else
|
|
||||||
VERBOSE=false
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check if containers exist
|
|
||||||
if ! docker ps -a --format "{{.Names}}" | grep -q "^${CONTAINER_NAME}_"; then
|
|
||||||
echo "Unknown"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check individual service status using docker directly (more reliable)
|
|
||||||
es_running=$(docker ps --format "{{.Names}}" | grep -c "^${CONTAINER_NAME}_elasticsearch$" || echo 0)
|
|
||||||
ls_running=$(docker ps --format "{{.Names}}" | grep -c "^${CONTAINER_NAME}_logstash$" || echo 0)
|
|
||||||
kb_running=$(docker ps --format "{{.Names}}" | grep -c "^${CONTAINER_NAME}_kibana$" || echo 0)
|
|
||||||
|
|
||||||
if [ "$VERBOSE" = "true" ]; then
|
|
||||||
echo "Elasticsearch: $([ $es_running -eq 1 ] && echo "Running" || echo "Not running")"
|
|
||||||
echo "Logstash: $([ $ls_running -eq 1 ] && echo "Running" || echo "Not running")"
|
|
||||||
echo "Kibana: $([ $kb_running -eq 1 ] && echo "Running" || echo "Not running")"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Return overall status
|
|
||||||
if [ "$es_running" -eq 1 ] && [ "$ls_running" -eq 1 ] && [ "$kb_running" -eq 1 ]; then
|
|
||||||
echo "Running"
|
echo "Running"
|
||||||
elif [ "$es_running" -eq 0 ] && [ "$ls_running" -eq 0 ] && [ "$kb_running" -eq 0 ]; then
|
echo " Web UI: http://$(hostname -I | awk '{print $1}'):${WEB_PORT:-3000}"
|
||||||
echo "Stopped"
|
echo " Loki API: http://$(hostname -I | awk '{print $1}'):${LOKI_PORT:-3100}"
|
||||||
|
docker ps --format "table {{.Names}}\t{{.Status}}" | grep "${CONTAINER_NAME}_"
|
||||||
else
|
else
|
||||||
# At least one service is having issues
|
echo "Stopped"
|
||||||
if [ "$VERBOSE" = "false" ]; then
|
|
||||||
echo "Starting" # More accurate than "Error" during startup
|
|
||||||
fi
|
|
||||||
fi
|
fi
|
||||||
@@ -2,7 +2,5 @@
|
|||||||
source "${AGENT_PATH}/common.sh"
|
source "${AGENT_PATH}/common.sh"
|
||||||
_check_required_env_vars "CONTAINER_NAME"
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
echo "Stopping ELK stack..."
|
docker compose stop || _die "Failed to stop"
|
||||||
docker compose stop || true
|
echo "Log Server stopped"
|
||||||
|
|
||||||
echo "ELK stack stopped"
|
|
||||||
@@ -1,54 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Test script to verify API key authentication is working
|
|
||||||
# This sends test log messages with and without API keys to verify authentication
|
|
||||||
|
|
||||||
source "${AGENT_PATH}/common.sh"
|
|
||||||
_check_required_env_vars "CONTAINER_NAME" "LOGSTASH_BEATS_PORT"
|
|
||||||
|
|
||||||
echo "=== Testing Logstash API Key Authentication ==="
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Check if service is running
|
|
||||||
if ! bash ./status.sh | grep -q "Running"; then
|
|
||||||
echo "Error: LogServer is not running. Please start it first."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Get server IP
|
|
||||||
SERVER_IP=$(hostname -I | awk '{print $1}')
|
|
||||||
|
|
||||||
echo "Testing authentication on ${SERVER_IP}:${LOGSTASH_BEATS_PORT}"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Test 1: Send without API key (should be tagged or dropped based on config)
|
|
||||||
echo "Test 1: Sending log without API key..."
|
|
||||||
echo '{"message":"Test log without API key","timestamp":"'$(date -Iseconds)'"}' | \
|
|
||||||
nc -w 2 ${SERVER_IP} ${LOGSTASH_BEATS_PORT} 2>/dev/null || true
|
|
||||||
|
|
||||||
# Test 2: Send with invalid API key (should be dropped)
|
|
||||||
echo "Test 2: Sending log with invalid API key..."
|
|
||||||
echo '{"message":"Test log with invalid key","fields":{"api_key":"invalid-key-12345"},"timestamp":"'$(date -Iseconds)'"}' | \
|
|
||||||
nc -w 2 ${SERVER_IP} ${LOGSTASH_BEATS_PORT} 2>/dev/null || true
|
|
||||||
|
|
||||||
# Test 3: Send with valid API key (if one exists)
|
|
||||||
if [ -f "${CONFIG_PATH}/api-keys.yml" ]; then
|
|
||||||
# Extract first API key from the file
|
|
||||||
API_KEY=$(grep -E "^ [^:]+: " "${CONFIG_PATH}/api-keys.yml" | head -1 | awk '{print $2}')
|
|
||||||
if [ -n "$API_KEY" ]; then
|
|
||||||
echo "Test 3: Sending log with valid API key..."
|
|
||||||
echo '{"message":"Test log with valid key","fields":{"api_key":"'${API_KEY}'"},"timestamp":"'$(date -Iseconds)'"}' | \
|
|
||||||
nc -w 2 ${SERVER_IP} ${LOGSTASH_BEATS_PORT} 2>/dev/null || true
|
|
||||||
else
|
|
||||||
echo "Test 3: Skipped - no API keys found"
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
echo "Test 3: Skipped - api-keys.yml not found"
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo ""
|
|
||||||
echo "Tests sent. Check Logstash logs to verify authentication:"
|
|
||||||
echo " docker logs ${CONTAINER_NAME}_logstash --tail 20"
|
|
||||||
echo ""
|
|
||||||
echo "Valid logs should appear in Elasticsearch."
|
|
||||||
echo "Invalid logs should be dropped."
|
|
||||||
@@ -2,15 +2,5 @@
|
|||||||
source "${AGENT_PATH}/common.sh"
|
source "${AGENT_PATH}/common.sh"
|
||||||
_check_required_env_vars "CONTAINER_NAME"
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
# Stop the containers
|
docker compose down || true
|
||||||
bash ./stop.sh || _die "Failed to stop containers"
|
echo "Log Server uninstalled (data preserved)"
|
||||||
|
|
||||||
# Remove the containers
|
|
||||||
docker compose down --remove-orphans || _die "Failed to remove containers"
|
|
||||||
|
|
||||||
# CRITICAL: Never remove data volumes in uninstall.sh!
|
|
||||||
# Data volumes must be preserved for potential reinstallation
|
|
||||||
# Only destroy.sh should remove volumes, and it must be explicit
|
|
||||||
|
|
||||||
echo "Uninstallation of ${CONTAINER_NAME} complete"
|
|
||||||
echo "Note: Data volumes have been preserved. To remove all data, use destroy.sh"
|
|
||||||
61
simple-logs/README.md
Normal file
61
simple-logs/README.md
Normal file
@@ -0,0 +1,61 @@
|
|||||||
|
# Simple Logs
|
||||||
|
|
||||||
|
A dead-simple log viewer using Grafana + Loki. Much easier than ELK/Kibana!
|
||||||
|
|
||||||
|
## Features
|
||||||
|
- **ALL logs in one place** - Docker containers + system logs
|
||||||
|
- **Simple interface** - Just one dashboard, no complex setup
|
||||||
|
- **Fast & lightweight** - Uses 10x less resources than ELK
|
||||||
|
- **Real-time** - Logs appear instantly
|
||||||
|
- **No configuration** - Works out of the box
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
1. **Install**
|
||||||
|
```bash
|
||||||
|
dropshell install simple-logs
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Access**
|
||||||
|
- Go to: `http://<server-ip>:3000`
|
||||||
|
- Login: `admin` / `admin` (change on first login)
|
||||||
|
|
||||||
|
3. **View Logs**
|
||||||
|
- Click the **Dashboards** icon (4 squares) on left
|
||||||
|
- Click **"Simple Logs"**
|
||||||
|
- That's it! All your logs are there
|
||||||
|
|
||||||
|
## How to Use
|
||||||
|
|
||||||
|
### Filter by Container
|
||||||
|
Click any `container_name` label to see only that container's logs
|
||||||
|
|
||||||
|
### Search Text
|
||||||
|
Use the search box at the top to find specific text
|
||||||
|
|
||||||
|
### Change Time Range
|
||||||
|
Top-right corner - set to "Last 5 minutes" for recent logs
|
||||||
|
|
||||||
|
### View Details
|
||||||
|
Click any log line to expand and see full details
|
||||||
|
|
||||||
|
## What Gets Logged
|
||||||
|
- All Docker container logs
|
||||||
|
- System logs (/var/log/syslog)
|
||||||
|
- Authentication logs (/var/log/auth.log)
|
||||||
|
|
||||||
|
## Ports
|
||||||
|
- `3000` - Grafana Web UI
|
||||||
|
- `3100` - Loki API (internal)
|
||||||
|
|
||||||
|
## Why This Instead of ELK?
|
||||||
|
- **10x simpler** - One dashboard vs Kibana's complexity
|
||||||
|
- **10x smaller** - ~200MB RAM vs 2-3GB for ELK
|
||||||
|
- **Instant setup** - No index patterns, no field mappings
|
||||||
|
- **Just works** - No configuration needed
|
||||||
|
|
||||||
|
## Advanced: Custom Queries
|
||||||
|
In the logs panel, you can use LogQL queries:
|
||||||
|
- `{container_name="myapp"}` - Specific container
|
||||||
|
- `{job="syslog"} |= "error"` - System errors
|
||||||
|
- `{job="docker"} |~ "(?i)error|warn"` - All Docker warnings/errors
|
||||||
69
simple-logs/config/dashboards/simple-logs.json
Normal file
69
simple-logs/config/dashboards/simple-logs.json
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
{
|
||||||
|
"annotations": {
|
||||||
|
"list": [
|
||||||
|
{
|
||||||
|
"builtIn": 1,
|
||||||
|
"datasource": "-- Grafana --",
|
||||||
|
"enable": true,
|
||||||
|
"hide": true,
|
||||||
|
"iconColor": "rgba(0, 211, 255, 1)",
|
||||||
|
"name": "Annotations & Alerts",
|
||||||
|
"type": "dashboard"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"editable": true,
|
||||||
|
"gnetId": null,
|
||||||
|
"graphTooltip": 0,
|
||||||
|
"id": null,
|
||||||
|
"links": [],
|
||||||
|
"panels": [
|
||||||
|
{
|
||||||
|
"datasource": "Loki",
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 20,
|
||||||
|
"w": 24,
|
||||||
|
"x": 0,
|
||||||
|
"y": 0
|
||||||
|
},
|
||||||
|
"id": 2,
|
||||||
|
"options": {
|
||||||
|
"dedupStrategy": "none",
|
||||||
|
"enableLogDetails": true,
|
||||||
|
"prettifyLogMessage": false,
|
||||||
|
"showCommonLabels": false,
|
||||||
|
"showLabels": true,
|
||||||
|
"showTime": true,
|
||||||
|
"sortOrder": "Descending",
|
||||||
|
"wrapLogMessage": true
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "{job=~\"docker|syslog|auth\"}",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "All Logs",
|
||||||
|
"type": "logs"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"schemaVersion": 27,
|
||||||
|
"style": "dark",
|
||||||
|
"tags": ["logs"],
|
||||||
|
"templating": {
|
||||||
|
"list": []
|
||||||
|
},
|
||||||
|
"time": {
|
||||||
|
"from": "now-15m",
|
||||||
|
"to": "now"
|
||||||
|
},
|
||||||
|
"timepicker": {},
|
||||||
|
"timezone": "",
|
||||||
|
"title": "Simple Logs",
|
||||||
|
"uid": "simple-logs",
|
||||||
|
"version": 0
|
||||||
|
}
|
||||||
12
simple-logs/config/grafana-dashboard.yml
Normal file
12
simple-logs/config/grafana-dashboard.yml
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
apiVersion: 1
|
||||||
|
|
||||||
|
providers:
|
||||||
|
- name: 'default'
|
||||||
|
orgId: 1
|
||||||
|
folder: ''
|
||||||
|
type: file
|
||||||
|
disableDeletion: false
|
||||||
|
updateIntervalSeconds: 10
|
||||||
|
allowUiUpdates: true
|
||||||
|
options:
|
||||||
|
path: /var/lib/grafana/dashboards
|
||||||
9
simple-logs/config/grafana-datasources.yml
Normal file
9
simple-logs/config/grafana-datasources.yml
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
apiVersion: 1
|
||||||
|
|
||||||
|
datasources:
|
||||||
|
- name: Loki
|
||||||
|
type: loki
|
||||||
|
access: proxy
|
||||||
|
url: http://loki:3100
|
||||||
|
isDefault: true
|
||||||
|
editable: false
|
||||||
36
simple-logs/config/loki.yaml
Normal file
36
simple-logs/config/loki.yaml
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
auth_enabled: false
|
||||||
|
|
||||||
|
server:
|
||||||
|
http_listen_port: 3100
|
||||||
|
grpc_listen_port: 9096
|
||||||
|
|
||||||
|
common:
|
||||||
|
path_prefix: /loki
|
||||||
|
storage:
|
||||||
|
filesystem:
|
||||||
|
chunks_directory: /loki/chunks
|
||||||
|
rules_directory: /loki/rules
|
||||||
|
replication_factor: 1
|
||||||
|
ring:
|
||||||
|
instance_addr: 127.0.0.1
|
||||||
|
kvstore:
|
||||||
|
store: inmemory
|
||||||
|
|
||||||
|
schema_config:
|
||||||
|
configs:
|
||||||
|
- from: 2020-10-24
|
||||||
|
store: boltdb-shipper
|
||||||
|
object_store: filesystem
|
||||||
|
schema: v11
|
||||||
|
index:
|
||||||
|
prefix: index_
|
||||||
|
period: 24h
|
||||||
|
|
||||||
|
ruler:
|
||||||
|
alertmanager_url: http://localhost:9093
|
||||||
|
|
||||||
|
limits_config:
|
||||||
|
retention_period: 168h # 7 days
|
||||||
|
enforce_metric_name: false
|
||||||
|
reject_old_samples: true
|
||||||
|
reject_old_samples_max_age: 168h
|
||||||
67
simple-logs/config/promtail.yaml
Normal file
67
simple-logs/config/promtail.yaml
Normal file
@@ -0,0 +1,67 @@
|
|||||||
|
server:
|
||||||
|
http_listen_port: 9080
|
||||||
|
grpc_listen_port: 0
|
||||||
|
|
||||||
|
positions:
|
||||||
|
filename: /tmp/positions/positions.yaml
|
||||||
|
|
||||||
|
clients:
|
||||||
|
- url: http://loki:3100/loki/api/v1/push
|
||||||
|
|
||||||
|
scrape_configs:
|
||||||
|
# Docker container logs
|
||||||
|
- job_name: docker
|
||||||
|
static_configs:
|
||||||
|
- targets:
|
||||||
|
- localhost
|
||||||
|
labels:
|
||||||
|
job: docker
|
||||||
|
__path__: /var/lib/docker/containers/*/*-json.log
|
||||||
|
pipeline_stages:
|
||||||
|
- json:
|
||||||
|
expressions:
|
||||||
|
stream: stream
|
||||||
|
time: time
|
||||||
|
log: log
|
||||||
|
- timestamp:
|
||||||
|
source: time
|
||||||
|
format: RFC3339Nano
|
||||||
|
- labels:
|
||||||
|
stream:
|
||||||
|
- regex:
|
||||||
|
expression: '(?P<container_id>[0-9a-f]{12})'
|
||||||
|
source: filename
|
||||||
|
- labels:
|
||||||
|
container_id:
|
||||||
|
|
||||||
|
# System logs
|
||||||
|
- job_name: syslog
|
||||||
|
static_configs:
|
||||||
|
- targets:
|
||||||
|
- localhost
|
||||||
|
labels:
|
||||||
|
job: syslog
|
||||||
|
__path__: /var/log/syslog
|
||||||
|
host: ${HOSTNAME}
|
||||||
|
|
||||||
|
- job_name: auth
|
||||||
|
static_configs:
|
||||||
|
- targets:
|
||||||
|
- localhost
|
||||||
|
labels:
|
||||||
|
job: auth
|
||||||
|
__path__: /var/log/auth.log
|
||||||
|
host: ${HOSTNAME}
|
||||||
|
|
||||||
|
# Docker events via Docker socket
|
||||||
|
- job_name: docker_events
|
||||||
|
docker_sd_configs:
|
||||||
|
- host: unix:///var/run/docker.sock
|
||||||
|
refresh_interval: 5s
|
||||||
|
relabel_configs:
|
||||||
|
- source_labels: [__meta_docker_container_name]
|
||||||
|
target_label: container_name
|
||||||
|
- source_labels: [__meta_docker_container_id]
|
||||||
|
target_label: container_id
|
||||||
|
- source_labels: [__meta_docker_container_image]
|
||||||
|
target_label: image
|
||||||
15
simple-logs/config/service.env
Normal file
15
simple-logs/config/service.env
Normal file
@@ -0,0 +1,15 @@
|
|||||||
|
# Simple Log Viewer Configuration
|
||||||
|
CONTAINER_NAME=simple-logs
|
||||||
|
|
||||||
|
# Server settings (REQUIRED by dropshell)
|
||||||
|
SSH_USER="root"
|
||||||
|
|
||||||
|
# Web UI Port
|
||||||
|
WEB_PORT=3000
|
||||||
|
|
||||||
|
# Log retention (days)
|
||||||
|
LOG_RETENTION=7
|
||||||
|
|
||||||
|
# Authentication (optional - leave empty for no auth)
|
||||||
|
AUTH_USERNAME=""
|
||||||
|
AUTH_PASSWORD=""
|
||||||
54
simple-logs/docker-compose.yml
Normal file
54
simple-logs/docker-compose.yml
Normal file
@@ -0,0 +1,54 @@
|
|||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
services:
|
||||||
|
# Grafana - Simple Web UI for viewing logs
|
||||||
|
grafana:
|
||||||
|
image: grafana/grafana:latest
|
||||||
|
container_name: ${CONTAINER_NAME}_grafana
|
||||||
|
environment:
|
||||||
|
- GF_SECURITY_ADMIN_USER=${AUTH_USERNAME:-admin}
|
||||||
|
- GF_SECURITY_ADMIN_PASSWORD=${AUTH_PASSWORD:-admin}
|
||||||
|
- GF_AUTH_ANONYMOUS_ENABLED=${AUTH_USERNAME:+false}
|
||||||
|
- GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer
|
||||||
|
- GF_INSTALL_PLUGINS=
|
||||||
|
volumes:
|
||||||
|
- grafana_data:/var/lib/grafana
|
||||||
|
- ${CONFIG_PATH}/grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yaml:ro
|
||||||
|
- ${CONFIG_PATH}/grafana-dashboard.yml:/etc/grafana/provisioning/dashboards/dashboard.yaml:ro
|
||||||
|
- ${CONFIG_PATH}/dashboards:/var/lib/grafana/dashboards:ro
|
||||||
|
ports:
|
||||||
|
- "${WEB_PORT:-3000}:3000"
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
- loki
|
||||||
|
|
||||||
|
# Loki - Lightweight log storage (100x simpler than Elasticsearch)
|
||||||
|
loki:
|
||||||
|
image: grafana/loki:2.9.0
|
||||||
|
container_name: ${CONTAINER_NAME}_loki
|
||||||
|
volumes:
|
||||||
|
- loki_data:/loki
|
||||||
|
- ${CONFIG_PATH}/loki.yaml:/etc/loki/local-config.yaml:ro
|
||||||
|
ports:
|
||||||
|
- "3100:3100"
|
||||||
|
command: -config.file=/etc/loki/local-config.yaml
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
# Promtail - Collects all logs (Docker + System)
|
||||||
|
promtail:
|
||||||
|
image: grafana/promtail:2.9.0
|
||||||
|
container_name: ${CONTAINER_NAME}_promtail
|
||||||
|
volumes:
|
||||||
|
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||||
|
- /var/log:/var/log:ro
|
||||||
|
- ${CONFIG_PATH}/promtail.yaml:/etc/promtail/config.yml:ro
|
||||||
|
- promtail_positions:/tmp/positions
|
||||||
|
command: -config.file=/etc/promtail/config.yml
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
- loki
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
grafana_data:
|
||||||
|
loki_data:
|
||||||
|
promtail_positions:
|
||||||
62
simple-logs/install.sh
Executable file
62
simple-logs/install.sh
Executable file
@@ -0,0 +1,62 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
source "${AGENT_PATH}/common.sh"
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
|
# Check Docker
|
||||||
|
_check_docker_installed || _die "Docker test failed"
|
||||||
|
docker compose version >/dev/null 2>&1 || _die "Docker Compose V2 is required"
|
||||||
|
|
||||||
|
# Stop any existing containers
|
||||||
|
bash ./stop.sh 2>/dev/null || true
|
||||||
|
|
||||||
|
# Create config directory
|
||||||
|
mkdir -p "${CONFIG_PATH}/dashboards"
|
||||||
|
|
||||||
|
# Copy configuration files
|
||||||
|
cp "$SCRIPT_DIR/config/"*.yaml "$SCRIPT_DIR/config/"*.yml "${CONFIG_PATH}/" 2>/dev/null || true
|
||||||
|
cp "$SCRIPT_DIR/config/dashboards/"*.json "${CONFIG_PATH}/dashboards/" 2>/dev/null || true
|
||||||
|
|
||||||
|
# Start the stack
|
||||||
|
echo "Starting Simple Logs stack..."
|
||||||
|
docker compose up -d || _die "Failed to start"
|
||||||
|
|
||||||
|
# Wait for Grafana to be ready
|
||||||
|
echo -n "Waiting for Grafana to start..."
|
||||||
|
MAX_WAIT=60
|
||||||
|
WAITED=0
|
||||||
|
while [ $WAITED -lt $MAX_WAIT ]; do
|
||||||
|
if curl -s "http://localhost:${WEB_PORT:-3000}/api/health" 2>/dev/null | grep -q "ok"; then
|
||||||
|
echo " Ready!"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
echo -n "."
|
||||||
|
sleep 2
|
||||||
|
WAITED=$((WAITED + 2))
|
||||||
|
done
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo "========================================="
|
||||||
|
echo "Simple Logs Installed!"
|
||||||
|
echo "========================================="
|
||||||
|
echo ""
|
||||||
|
echo "Access at: http://$(hostname -I | awk '{print $1}'):${WEB_PORT:-3000}"
|
||||||
|
echo ""
|
||||||
|
if [ -n "${AUTH_USERNAME}" ]; then
|
||||||
|
echo "Login: ${AUTH_USERNAME} / ${AUTH_PASSWORD}"
|
||||||
|
else
|
||||||
|
echo "Login: admin / admin"
|
||||||
|
echo "(You'll be asked to change password on first login)"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
echo "TO VIEW LOGS:"
|
||||||
|
echo "1. Click 'Dashboards' icon (4 squares) on left"
|
||||||
|
echo "2. Click 'Simple Logs'"
|
||||||
|
echo "3. Your logs appear immediately!"
|
||||||
|
echo ""
|
||||||
|
echo "TO FILTER:"
|
||||||
|
echo "- Click any label to filter by it"
|
||||||
|
echo "- Use the search box for text search"
|
||||||
|
echo "- Time range selector is in top-right"
|
||||||
|
echo "========================================="
|
||||||
12
simple-logs/logs.sh
Executable file
12
simple-logs/logs.sh
Executable file
@@ -0,0 +1,12 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
source "${AGENT_PATH}/common.sh"
|
||||||
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
|
echo "=== Grafana Logs ==="
|
||||||
|
docker logs "${CONTAINER_NAME}_grafana" --tail 20
|
||||||
|
echo ""
|
||||||
|
echo "=== Loki Logs ==="
|
||||||
|
docker logs "${CONTAINER_NAME}_loki" --tail 20
|
||||||
|
echo ""
|
||||||
|
echo "=== Promtail Logs ==="
|
||||||
|
docker logs "${CONTAINER_NAME}_promtail" --tail 20
|
||||||
6
simple-logs/start.sh
Executable file
6
simple-logs/start.sh
Executable file
@@ -0,0 +1,6 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
source "${AGENT_PATH}/common.sh"
|
||||||
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
|
docker compose up -d || _die "Failed to start Simple Logs"
|
||||||
|
echo "Simple Logs started"
|
||||||
11
simple-logs/status.sh
Executable file
11
simple-logs/status.sh
Executable file
@@ -0,0 +1,11 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
source "${AGENT_PATH}/common.sh"
|
||||||
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
|
if docker ps | grep -q "${CONTAINER_NAME}_grafana"; then
|
||||||
|
echo "Running"
|
||||||
|
echo " Grafana: http://$(hostname -I | awk '{print $1}'):${WEB_PORT:-3000}"
|
||||||
|
docker ps --format "table {{.Names}}\t{{.Status}}" | grep "${CONTAINER_NAME}_"
|
||||||
|
else
|
||||||
|
echo "Stopped"
|
||||||
|
fi
|
||||||
6
simple-logs/stop.sh
Executable file
6
simple-logs/stop.sh
Executable file
@@ -0,0 +1,6 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
source "${AGENT_PATH}/common.sh"
|
||||||
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
|
docker compose stop || _die "Failed to stop Simple Logs"
|
||||||
|
echo "Simple Logs stopped"
|
||||||
6
simple-logs/uninstall.sh
Executable file
6
simple-logs/uninstall.sh
Executable file
@@ -0,0 +1,6 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
source "${AGENT_PATH}/common.sh"
|
||||||
|
_check_required_env_vars "CONTAINER_NAME"
|
||||||
|
|
||||||
|
docker compose down || true
|
||||||
|
echo "Simple Logs uninstalled (data preserved)"
|
||||||
Reference in New Issue
Block a user