diff --git a/logclient/DOCUMENTATION.md b/logclient/DOCUMENTATION.md deleted file mode 100644 index 44aa748..0000000 --- a/logclient/DOCUMENTATION.md +++ /dev/null @@ -1,389 +0,0 @@ -# Dropshell LogClient Template - -An auto-configuring Filebeat agent that collects Docker container logs via the Docker API and system logs, shipping them to a centralized logging server with minimal configuration. - -## Overview - -This template deploys a lightweight Filebeat agent that: -- Uses Docker API to collect logs from all containers (regardless of logging driver) -- Allows containers to use any Docker logging driver (json-file, local, journald, etc.) -- Collects system logs (syslog, auth logs, kernel logs) -- Ships logs to a centralized ELK stack (logserver) -- Requires minimal configuration - just the server address -- Handles connection failures with local buffering -- Auto-reconnects when the server becomes available - -## Features - -### Docker API Log Collection -- **Direct API access**: Reads logs via Docker API, not from files -- **Driver independent**: Works with any Docker logging driver (local, json-file, journald) -- **Automatic discovery**: Finds all running containers dynamically -- **Container metadata**: Enriches logs with container names, images, labels -- **Real-time streaming**: Gets logs as they're generated -- **Multi-line handling**: Properly handles stack traces and multi-line logs -- **JSON parsing**: Automatically parses JSON-formatted logs -- **Label-based filtering**: Can include/exclude containers based on labels - -### System Log Collection -- **/var/log/syslog** or **/var/log/messages**: System events -- **/var/log/auth.log** or **/var/log/secure**: Authentication events -- **/var/log/kern.log**: Kernel messages -- **journald**: SystemD journal (if available) -- **Custom paths**: Configurable additional log paths - -### Reliability Features -- **Local buffering**: Stores logs locally when server is unreachable -- **Automatic retry**: Reconnects automatically with exponential backoff -- **Compression**: Compresses logs before sending to save bandwidth -- **Secure transmission**: Optional TLS/SSL encryption -- **Backpressure handling**: Slows down when server is overwhelmed - -## Architecture - -### How It Works -1. Filebeat runs as a container with Docker socket access -2. Uses Docker API to stream logs from all containers -3. Monitors Docker API for container lifecycle events -4. Automatically starts collecting logs from new containers -5. Reads host system logs from mounted volumes -6. Ships all logs to configured Logstash/Elasticsearch endpoint -7. Maintains connection state and buffering information - -### Log Flow -``` -Docker Containers → Docker API → - ↘ - Filebeat → Logstash → Elasticsearch → Kibana - ↗ -System Logs (mounted volumes) → -``` - -### Why Docker API Instead of Log Files? -- **Logging driver flexibility**: Containers can use `local`, `json-file`, `journald`, or any driver -- **No log file management**: Don't need to worry about log rotation or file paths -- **Better performance**: Direct streaming without file I/O overhead -- **Consistent access**: Same method regardless of storage backend -- **Real-time streaming**: Get logs immediately as they're generated -- **Simplified permissions**: Only need Docker socket access - -## Minimum Configuration - -The template requires minimal configuration - server address and authentication: - -```bash -# Required - Server connection -LOGSERVER_HOST=192.168.1.100 -LOGSERVER_PORT=5044 - -# Required - Authentication (choose one method) -AUTH_MODE=mtls # Options: mtls, apikey, basic - -# For mTLS authentication -CLIENT_CERT_PATH=/certs/client.crt -CLIENT_KEY_PATH=/certs/client.key -CA_CERT_PATH=/certs/ca.crt - -# For API key authentication -API_KEY=your-api-key-here - -# For basic auth (not recommended) -USERNAME=filebeat -PASSWORD=changeme -``` - -## Configuration Options - -### Environment Variables (service.env) - -```bash -# REQUIRED: Log server connection -LOGSERVER_HOST=logserver.example.com -LOGSERVER_PORT=5044 - -# REQUIRED: Authentication method -AUTH_MODE=mtls # mtls, apikey, or basic - -# mTLS Authentication (if AUTH_MODE=mtls) -CLIENT_CERT_PATH=/certs/${HOSTNAME}.crt -CLIENT_KEY_PATH=/certs/${HOSTNAME}.key -CA_CERT_PATH=/certs/ca.crt -SSL_VERIFICATION_MODE=full - -# API Key Authentication (if AUTH_MODE=apikey) -API_KEY="" # Will be provided by logserver admin - -# Basic Authentication (if AUTH_MODE=basic) -USERNAME=filebeat -PASSWORD=changeme - -# Optional: Performance tuning -BULK_MAX_SIZE=2048 # Maximum batch size -WORKER_THREADS=1 # Number of worker threads -QUEUE_SIZE=4096 # Internal queue size -MAX_BACKOFF=60s # Maximum retry backoff - -# Optional: Filtering -EXCLUDE_CONTAINERS="" # Comma-separated container names to exclude -INCLUDE_CONTAINERS="" # Only include these containers (if set) -EXCLUDE_LABELS="" # Exclude containers with these labels -INCLUDE_LABELS="" # Only include containers with these labels - -# Optional: Additional log paths -CUSTOM_LOG_PATHS="" # Comma-separated additional paths to monitor - -# Optional: Resource limits -MAX_CPU=50 # Maximum CPU usage percentage -MAX_MEMORY=200MB # Maximum memory usage -``` - -## Collected Log Types - -### Docker Container Logs (via Docker API) -- **stdout/stderr**: All container output regardless of logging driver -- **Container metadata**: Name, ID, image, labels -- **Docker events**: Start, stop, die, kill events -- **Health check results**: If configured -- **Works with all logging drivers**: local, json-file, journald, syslog, etc. - -### System Logs -- **System messages**: Service starts/stops, errors -- **Authentication**: SSH logins, sudo usage -- **Kernel messages**: Hardware events, driver messages -- **Package management**: apt/yum operations -- **Cron jobs**: Scheduled task execution - -## Log Enrichment - -Logs are automatically enriched with: -- **Hostname**: Source host identification -- **Timestamp**: Precise event time with timezone -- **Log level**: Parsed from log content when possible -- **Container info**: For Docker logs -- **Process info**: PID, command for system logs -- **File path**: Source log file - -## Resource Requirements - -### Minimum -- CPU: 0.5 cores -- RAM: 128MB -- Storage: 1GB (for buffer) - -### Typical Usage -- CPU: 1-5% of one core -- RAM: 150-200MB -- Network: Varies with log volume -- Storage: Depends on buffer size - -## Installation - -### Prerequisites -1. A running logserver (ELK stack) -2. Network connectivity to logserver -3. Docker installed on host -4. Authentication credentials from logserver admin - -### Setup Authentication - -#### For mTLS (Recommended): -```bash -# Get client certificate from logserver admin -# They will run: dropshell exec logserver /scripts/generate-client-cert.sh $(hostname) -# Copy the generated certificate files to this client -mkdir -p /etc/dropshell/certs -# Copy ca.crt, client.crt, and client.key to /etc/dropshell/certs/ -``` - -#### For API Key: -```bash -# Get API key from logserver admin -# They will run: dropshell exec logserver /scripts/generate-api-key.sh $(hostname) -# Add the API key to service.env -``` - -### Deploy -```bash -# Configure authentication in service.env -dropshell install logclient -``` - -## Monitoring - -### Check Status -```bash -dropshell status logclient -``` - -### View Filebeat Logs -```bash -dropshell logs logclient -``` - -### Verify Connectivity -```bash -# Check if logs are being shipped -docker exec logclient-filebeat filebeat test output -``` - -### Monitor Metrics -```bash -# View Filebeat statistics -docker exec logclient-filebeat curl -s http://localhost:5066/stats -``` - -## Troubleshooting - -### No Logs Appearing on Server - -1. **Check connectivity** - ```bash - telnet $LOGSERVER_HOST $LOGSERVER_PORT - ``` - -2. **Verify Filebeat is running** - ```bash - dropshell status logclient - ``` - -3. **Check Filebeat logs** - ```bash - dropshell logs logclient | tail -50 - ``` - -4. **Test configuration** - ```bash - docker exec logclient-filebeat filebeat test config - ``` - -### High CPU Usage - -1. Reduce worker threads in service.env -2. Increase bulk_max_size to send larger batches -3. Add exclude filters for noisy containers - -### Missing Container Logs - -1. Verify Docker socket is mounted -2. Check container isn't in exclude list -3. Ensure Filebeat has permissions to Docker socket -4. Verify container is actually producing output -5. Check if container uses a supported logging driver - -### Buffer Full Errors - -1. Increase queue_size in service.env -2. Check network connectivity to server -3. Verify server isn't overwhelmed - -## Security Considerations - -1. **Authentication**: - - Always use mTLS or API keys in production - - Never use basic auth except for testing - - Store credentials securely - - Rotate certificates/keys regularly - -2. **Docker Socket Access**: - - Requires Docker socket access to read logs via API - - Understand security implications of socket access - - Consider read-only socket access if possible - -3. **Network Security**: - - All connections are TLS encrypted - - Verify server certificates - - Configure firewall rules appropriately - - Use private networks when possible - -4. **Data Protection**: - - Logs may contain sensitive data - - Filter sensitive information before shipping - - Exclude containers with sensitive data if needed - -5. **Resource Limits**: - - Set CPU and memory limits - - Monitor resource usage - - Prevent resource exhaustion attacks - -## Performance Tuning - -### For High-Volume Environments -```bash -# Increase workers and batch size -WORKER_THREADS=4 -BULK_MAX_SIZE=4096 -QUEUE_SIZE=8192 -``` - -### For Low-Resource Hosts -```bash -# Reduce resource usage -WORKER_THREADS=1 -BULK_MAX_SIZE=512 -MAX_MEMORY=100MB -MAX_CPU=25 -``` - -### Network Optimization -```bash -# Enable compression (CPU vs bandwidth tradeoff) -COMPRESSION_LEVEL=3 # 0-9, higher = more compression -``` - -## Integration with LogServer - -This template is designed to work seamlessly with the `logserver` template: - -1. Deploy logserver first -2. Note the logserver's IP/hostname -3. Configure logclient with server address -4. Logs automatically start flowing - -## Maintenance - -### Regular Tasks -- Monitor buffer usage -- Check for connection errors -- Review excluded/included containers -- Update Filebeat version - -### Logs Rotation -Filebeat handles log rotation automatically: -- Detects renamed/rotated files -- Continues reading from correct position -- Cleans up old file handles - -## Advanced Configuration - -### Custom Filebeat Configuration -Create a custom `filebeat.yml` in the config directory for advanced scenarios: -- Custom processors -- Additional inputs -- Complex filtering rules -- Multiple outputs - -### Docker Labels for Control -Control log collection per container: -```yaml -# In docker-compose.yml -services: - myapp: - logging: - driver: local # Can use any driver - Filebeat reads via API - options: - max-size: "10m" - max-file: "3" - labels: - - "filebeat.enable=false" # Exclude this container - - "filebeat.multiline.pattern=^\\[" # Custom multiline pattern -``` - -### Logging Driver Compatibility -The Docker API input works with all Docker logging drivers: -- **local**: Recommended for production (efficient, no file access needed) -- **json-file**: Traditional default driver -- **journald**: SystemD journal integration -- **syslog**: Forward to syslog -- **none**: Disables logging (Filebeat won't collect) - -You can use the `local` driver for better performance since Filebeat doesn't need to read files. \ No newline at end of file diff --git a/logclient/README.md b/logclient/README.md index 6cc4ba9..9a5a7f1 100644 --- a/logclient/README.md +++ b/logclient/README.md @@ -1,34 +1,56 @@ -# LogClient +# Log Client (Promtail) -Ships Docker container and system logs to LogServer using Filebeat. +Ships Docker and system logs to Log Server. Zero configuration! ## Quick Start -1. **Get API Key** - - Ask LogServer admin to run `./generate-api-key.sh` - - They'll provide your API key - -2. **Configure** -Edit `config/service.env`: +1. **Configure** (edit `config/service.env`): ```bash -LOGSERVER_HOST= -LOGSERVER_PORT=5044 -API_KEY= +LOGSERVER_HOST= # Your log server IP +LOGSERVER_PORT=3100 # Default Loki port +LOKI_USER=logclient # Default username +LOKI_PASSWORD= # Get from server admin ``` -3. **Install** +2. **Install**: ```bash dropshell install logclient ``` -## What It Does -- Collects all Docker container logs via API -- Collects system logs (/var/log) -- Ships to central LogServer -- Works with any Docker logging driver +That's it! Logs are now shipping to your central server. -## Requirements -- Docker socket access -- Network connection to LogServer port 5044 +## What Gets Sent +- All Docker container logs (via Docker API) +- System logs (/var/log/syslog) +- Auth logs (/var/log/auth.log) +- Hostname is automatically included -See [DOCUMENTATION.md](DOCUMENTATION.md) for full details. \ No newline at end of file +## View Your Logs +Go to: `http://:3000` +- Click Dashboards → Central Logs +- Your server's logs appear with its hostname + +## Features +- **Zero maintenance** - Just runs +- **Lightweight** - ~20MB RAM +- **Automatic** - Finds all containers +- **Labeled** - Hostname included automatically + +## Simple Authentication +Just a username/password - much simpler than ELK's API key system! + +## Troubleshooting +Check if client is running: +```bash +./status.sh +``` + +View client logs: +```bash +./logs.sh +``` + +Test connection to server: +```bash +nc -zv 3100 +``` \ No newline at end of file diff --git a/logclient/TODO.md b/logclient/TODO.md deleted file mode 100644 index 19f8209..0000000 --- a/logclient/TODO.md +++ /dev/null @@ -1,294 +0,0 @@ -# LogClient Template - Implementation TODO - -## Phase 1: Core Infrastructure (Priority 1) - -### Configuration Files -- [ ] Create `config/.template_info.env` with template metadata -- [ ] Create `config/service.env` with minimal required settings -- [ ] Define LOGSERVER_HOST and LOGSERVER_PORT variables -- [ ] Add AUTH_MODE variable (mtls, apikey, basic) -- [ ] Add certificate/key path variables for mTLS -- [ ] Add API_KEY variable for API key auth -- [ ] Add USERNAME/PASSWORD for basic auth -- [ ] Add optional performance and filtering variables -- [ ] Set sensible defaults where possible - -### Filebeat Configuration -- [ ] Create base `filebeat.yml` configuration template -- [ ] Configure Docker input using Docker API (not autodiscover with hints) -- [ ] Set containers.ids: ["*"] to collect from all containers -- [ ] Set up system log inputs for host logs -- [ ] Configure output to Logstash -- [ ] Add error handling and retry logic -- [ ] Set up local disk buffering -- [ ] Configure stream: "all" to get both stdout and stderr - -### Required Scripts -- [ ] Implement `install.sh` - Pull Filebeat image, configure auth, start -- [ ] Implement `uninstall.sh` - Stop and remove container (preserve config and certs) -- [ ] Implement `start.sh` - Start Filebeat with auth config and proper mounts -- [ ] Implement `stop.sh` - Gracefully stop Filebeat -- [ ] Implement `status.sh` - Check Filebeat health and auth status -- [ ] Create `setup-auth.sh` - Helper script to configure authentication - -## Phase 2: Docker API Log Collection (Priority 1) - -### Docker API Input Configuration -- [ ] Configure Docker input type (NOT autodiscover, use direct Docker input) -- [ ] Mount Docker socket (/var/run/docker.sock) with proper permissions -- [ ] Configure Docker API endpoint (unix:///var/run/docker.sock) -- [ ] Set up real-time log streaming from Docker daemon -- [ ] Enable collection from ALL logging drivers (local, json-file, journald, etc.) -- [ ] Configure since_time to get recent logs on startup - -### Container Metadata Extraction -- [ ] Extract container name, ID, image name, and image tag -- [ ] Map container labels to fields -- [ ] Handle docker-compose project names and service names -- [ ] Add container state information -- [ ] Include container environment variables (filtered) -- [ ] Handle container lifecycle events (start, stop, restart) - -### Container Filtering -- [ ] Implement include/exclude by container name patterns -- [ ] Add label-based filtering (containers.labels) -- [ ] Create ignore patterns for system containers -- [ ] Add support for custom filter expressions -- [ ] Configure combine_partial to handle partial log lines -- [ ] Document filtering examples with Docker API syntax - -## Phase 3: System Log Collection (Priority 1) - -### Log File Inputs -- [ ] Configure /var/log/syslog or /var/log/messages -- [ ] Add /var/log/auth.log or /var/log/secure -- [ ] Include /var/log/kern.log -- [ ] Monitor /var/log/dpkg.log or /var/log/yum.log -- [ ] Add custom log path support via environment variable - -### Journald Integration -- [ ] Detect if systemd/journald is available -- [ ] Configure journald input if present -- [ ] Set up unit filtering -- [ ] Extract systemd metadata -- [ ] Handle binary journal format - -### Log Parsing -- [ ] Configure syslog parsing -- [ ] Extract severity levels -- [ ] Parse timestamps correctly -- [ ] Handle different syslog formats -- [ ] Add timezone handling - -## Phase 4: Output Configuration (Priority 1) - -### Logstash Output -- [ ] Configure primary Logstash endpoint -- [ ] Set up connection parameters (timeout, retry) -- [ ] Configure bulk operations settings -- [ ] Add compression support -- [ ] Implement backpressure handling - -### Connection Management -- [ ] Configure automatic reconnection -- [ ] Set exponential backoff for retries -- [ ] Add connection pooling -- [ ] Configure keepalive settings -- [ ] Handle DNS resolution failures - -### Authentication Configuration (Priority 1 - CRITICAL) -- [ ] Implement mTLS authentication support -- [ ] Configure client certificate and key loading -- [ ] Add CA certificate validation -- [ ] Implement API key authentication -- [ ] Add basic auth as fallback option -- [ ] Create authentication mode selection logic -- [ ] Handle authentication failures gracefully -- [ ] Add certificate expiry checking -- [ ] Implement secure credential storage -- [ ] Document authentication setup process - -## Phase 5: Reliability Features (Priority 2) - -### Local Buffering -- [ ] Configure disk queue for reliability -- [ ] Set queue size limits -- [ ] Configure memory queue settings -- [ ] Add overflow handling -- [ ] Set up automatic cleanup of old events - -### Error Handling -- [ ] Add retry logic for failed sends -- [ ] Configure dead letter queue -- [ ] Add circuit breaker pattern -- [ ] Log transmission errors appropriately -- [ ] Add metrics for monitoring failures - -### Performance Optimization -- [ ] Configure worker count -- [ ] Set batch size for sending -- [ ] Add compression level setting -- [ ] Configure CPU and memory limits -- [ ] Optimize for high-volume scenarios - -## Phase 6: Optional Scripts (Priority 2) - -### Operational Scripts -- [ ] Implement `logs.sh` - Show Filebeat logs -- [ ] Implement `destroy.sh` - Complete removal -- [ ] Implement `ssh.sh` - Shell into Filebeat container -- [ ] Create `test.sh` - Test connectivity to server -- [ ] Add `metrics.sh` - Show Filebeat statistics - -### Diagnostic Scripts -- [ ] Create connectivity test script -- [ ] Add configuration validation script -- [ ] Create debug mode enabler -- [ ] Add log sampling script -- [ ] Create performance benchmark script - -## Phase 7: Monitoring & Health (Priority 2) - -### Health Checks -- [ ] Configure Filebeat HTTP endpoint -- [ ] Add Docker health check -- [ ] Monitor queue status -- [ ] Check connection to Logstash -- [ ] Track dropped events - -### Metrics Collection -- [ ] Enable Filebeat monitoring -- [ ] Export metrics endpoint -- [ ] Track events sent/failed -- [ ] Monitor resource usage -- [ ] Add performance counters - -### Status Reporting -- [ ] Implement detailed status in status.sh -- [ ] Show connection state -- [ ] Display queue status -- [ ] Report recent errors -- [ ] Show throughput metrics - -## Phase 8: Advanced Features (Priority 3) - -### Processors -- [ ] Add field renaming processor -- [ ] Configure drop_event conditions -- [ ] Add rate limiting processor -- [ ] Include fingerprinting for deduplication -- [ ] Add custom field enrichment - -### Multiline Handling -- [ ] Configure patterns for common languages -- [ ] Java stack trace handling -- [ ] Python traceback handling -- [ ] Go panic handling -- [ ] Custom pattern support via environment - -### Field Management -- [ ] Configure field inclusion/exclusion -- [ ] Add custom fields via environment -- [ ] Set up field type conversions -- [ ] Add timestamp parsing -- [ ] Configure field aliasing - -## Phase 9: Testing (Priority 3) - -### Unit Testing -- [ ] Test configuration generation -- [ ] Verify volume mounts -- [ ] Test environment variable substitution -- [ ] Validate filtering logic -- [ ] Test error conditions - -### Integration Testing -- [ ] Test with logserver template -- [ ] Verify Docker log collection -- [ ] Test system log collection -- [ ] Validate SSL connectivity -- [ ] Test reconnection scenarios -- [ ] Verify buffering during outages - -### Load Testing -- [ ] Test with high log volume -- [ ] Measure resource usage -- [ ] Test queue overflow handling -- [ ] Verify rate limiting -- [ ] Benchmark throughput - -## Phase 10: Documentation (Priority 3) - -### User Documentation -- [ ] Create README.txt for dropshell -- [ ] Document all configuration options -- [ ] Add troubleshooting guide -- [ ] Create quick start guide -- [ ] Add FAQ section - -### Configuration Examples -- [ ] Minimal configuration example -- [ ] High-volume configuration -- [ ] Secure SSL configuration -- [ ] Filtered configuration -- [ ] Custom paths configuration - -### Integration Guides -- [ ] Integration with logserver -- [ ] Docker Compose examples -- [ ] Kubernetes DaemonSet example -- [ ] Swarm mode configuration -- [ ] Custom application integration - -## Phase 11: Production Readiness (Priority 4) - -### Security Hardening -- [ ] Run as non-root user where possible -- [ ] Minimize container capabilities -- [ ] Add secrets management -- [ ] Configure log sanitization -- [ ] Add audit logging - -### Updates & Maintenance -- [ ] Add update notification -- [ ] Create upgrade script -- [ ] Add configuration migration -- [ ] Document breaking changes -- [ ] Create rollback procedure - -### Compatibility -- [ ] Test with different Filebeat versions -- [ ] Verify Docker API compatibility -- [ ] Test on different Linux distributions -- [ ] Validate with various log formats -- [ ] Ensure Logstash version compatibility - -## Notes - -### Design Principles -1. **Minimal configuration**: Just needs LOGSERVER_HOST to work -2. **Docker API access**: Use Docker API for driver-independent log collection -3. **Automatic discovery**: Find all container logs without manual configuration -4. **Reliability first**: Never lose logs, buffer locally if needed -5. **Low overhead**: Minimal resource usage on host -6. **Non-intrusive**: No changes to existing containers needed -7. **Driver flexibility**: Allow containers to use any logging driver (especially `local`) - -### Key Requirements -- Must work with zero configuration beyond server address -- Must use Docker API input, not file-based collection -- Must support all Docker logging drivers (local, json-file, etc.) -- Must handle Docker socket permissions properly -- Must be resilient to network failures -- Must not impact host performance significantly -- Must preserve configuration on uninstall - -### Testing Checklist -- [ ] Validates with dropshell test-template -- [ ] Connects to logserver successfully -- [ ] Collects Docker logs automatically -- [ ] Collects system logs properly -- [ ] Handles server downtime gracefully -- [ ] Reconnects automatically -- [ ] Resource usage stays within limits -- [ ] Uninstall preserves configuration \ No newline at end of file diff --git a/logclient/_volumes.sh b/logclient/_volumes.sh deleted file mode 100755 index 32c80e5..0000000 --- a/logclient/_volumes.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash -# Define volume items for logclient container -# These are used across backup, restore, create, and destroy operations - -get_logclient_volumes() { - echo "volume:datavolume:${DATA_VOLUME}" "volume:configvolume:${CONFIG_VOLUME}" -} \ No newline at end of file diff --git a/logclient/backup.sh b/logclient/backup.sh deleted file mode 100755 index 32d3939..0000000 --- a/logclient/backup.sh +++ /dev/null @@ -1,32 +0,0 @@ -#!/bin/bash -source "${AGENT_PATH}/common.sh" -source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh" - -_check_required_env_vars "CONTAINER_NAME" - -echo "Backing up ${CONTAINER_NAME} volumes..." - -# Stop the container to ensure data consistency -bash ./stop.sh || true - -# Backup volumes -BACKUP_DIR="${CONFIG_PATH}/backups/$(date +%Y%m%d_%H%M%S)" -mkdir -p "$BACKUP_DIR" - -# Export volumes -for volume in $(get_logclient_volumes); do - volume_name=$(echo $volume | cut -d: -f3) - echo "Backing up volume: $volume_name" - docker run --rm -v "$volume_name:/source:ro" -v "$BACKUP_DIR:/backup" alpine \ - tar -czf "/backup/${volume_name}.tar.gz" -C /source . -done - -# Backup configuration -cp -r "${CONFIG_PATH}" "$BACKUP_DIR/config_backup" - -echo "Backup completed to: $BACKUP_DIR" - -# Restart the container -bash ./start.sh - -echo "Container restarted" \ No newline at end of file diff --git a/logclient/config/.template_info.env b/logclient/config/.template_info.env deleted file mode 100644 index d82e5e0..0000000 --- a/logclient/config/.template_info.env +++ /dev/null @@ -1,16 +0,0 @@ -# Template identifier - MUST match the directory name -TEMPLATE=logclient - -# Requirements -REQUIRES_HOST_ROOT=false # No root access on host needed -REQUIRES_DOCKER=true # Docker is required -REQUIRES_DOCKER_ROOT=false # Docker root privileges not specifically needed - -# Docker image settings -IMAGE_REGISTRY="docker.elastic.co" -IMAGE_REPO="beats/filebeat" -IMAGE_TAG="7.17.23" - -# Volume definitions -DATA_VOLUME=logclient_data -CONFIG_VOLUME=logclient_config \ No newline at end of file diff --git a/logclient/config/service.env b/logclient/config/service.env index 0e4f4c2..bb02899 100644 --- a/logclient/config/service.env +++ b/logclient/config/service.env @@ -1,34 +1,16 @@ -# Service identification -CONTAINER_NAME=logclient-filebeat +# Log Client Configuration (Promtail) +CONTAINER_NAME=logclient # Server settings (REQUIRED by dropshell) SSH_USER="root" -# Docker image tag override (optional) -IMAGE_TAG="7.17.23" - # REQUIRED: Log server connection LOGSERVER_HOST= -LOGSERVER_PORT=5044 +LOGSERVER_PORT=3100 -# REQUIRED: API Key Authentication -API_KEY="" # Get from logserver admin using generate-api-key.sh +# REQUIRED: Authentication (get from log server admin) +LOKI_USER=logclient +LOKI_PASSWORD= -# Performance tuning -BULK_MAX_SIZE=2048 # Maximum batch size -WORKER_THREADS=1 # Number of worker threads -QUEUE_SIZE=4096 # Internal queue size -MAX_BACKOFF=60s # Maximum retry backoff - -# Filtering -EXCLUDE_CONTAINERS="" # Comma-separated container names to exclude -INCLUDE_CONTAINERS="" # Only include these containers (if set) -EXCLUDE_LABELS="" # Exclude containers with these labels -INCLUDE_LABELS="" # Only include containers with these labels - -# Additional log paths -CUSTOM_LOG_PATHS="" # Comma-separated additional paths to monitor - -# Resource limits -MAX_CPU=50 # Maximum CPU usage percentage -MAX_MEMORY=200MB # Maximum memory usage \ No newline at end of file +# Optional: Set a custom hostname label (defaults to actual hostname) +# HOSTNAME_LABEL= \ No newline at end of file diff --git a/logclient/destroy.sh b/logclient/destroy.sh deleted file mode 100755 index 5243f22..0000000 --- a/logclient/destroy.sh +++ /dev/null @@ -1,22 +0,0 @@ -#!/bin/bash -source "${AGENT_PATH}/common.sh" -source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh" - -_check_required_env_vars "CONTAINER_NAME" - -echo "WARNING: This will permanently delete all data for ${CONTAINER_NAME}" -echo "This action cannot be undone!" -echo "" - -# Since scripts must be non-interactive, only proceed if explicitly called -bash ./stop.sh || true -_remove_container "$CONTAINER_NAME" || true - -# Remove all volumes -for volume in $(get_logclient_volumes); do - volume_name=$(echo $volume | cut -d: -f3) - echo "Removing volume: $volume_name" - docker volume rm "$volume_name" 2>/dev/null || true -done - -echo "Service and all data destroyed" \ No newline at end of file diff --git a/logclient/docker-compose.yml b/logclient/docker-compose.yml new file mode 100644 index 0000000..e79cdda --- /dev/null +++ b/logclient/docker-compose.yml @@ -0,0 +1,19 @@ +version: '3.8' + +services: + promtail: + image: grafana/promtail:latest + container_name: ${CONTAINER_NAME} + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - /var/log:/var/log:ro + - /var/lib/docker/containers:/var/lib/docker/containers:ro + - ${CONFIG_PATH}/promtail.yaml:/etc/promtail/config.yml:ro + - promtail_positions:/tmp/positions + command: -config.file=/etc/promtail/config.yml + environment: + - HOSTNAME=${HOSTNAME_LABEL:-$(hostname)} + restart: unless-stopped + +volumes: + promtail_positions: \ No newline at end of file diff --git a/logclient/install.sh b/logclient/install.sh index fd479da..0af42fe 100755 --- a/logclient/install.sh +++ b/logclient/install.sh @@ -2,52 +2,39 @@ source "${AGENT_PATH}/common.sh" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -# Check required environment variables -_check_required_env_vars "CONTAINER_NAME" "IMAGE_REGISTRY" "IMAGE_REPO" "IMAGE_TAG" "LOGSERVER_HOST" "LOGSERVER_PORT" "API_KEY" +_check_required_env_vars "CONTAINER_NAME" "LOGSERVER_HOST" "LOGSERVER_PORT" "LOKI_PASSWORD" -# Validate API key -if [ -z "$API_KEY" ]; then - echo "" - echo "ERROR: API_KEY is not configured" - echo "" - echo "To get an API key:" - echo "1. On the logserver, run: ./generate-api-key.sh" - echo "2. Enter this client's hostname when prompted" - echo "3. Copy the generated API_KEY to this client's service.env" - echo "" - _die "Missing API_KEY configuration" -fi - -# Check Docker is available +# Check Docker _check_docker_installed || _die "Docker test failed" # Test connectivity to logserver -echo "Testing connectivity to logserver at ${LOGSERVER_HOST}:${LOGSERVER_PORT}..." -nc -zv "$LOGSERVER_HOST" "$LOGSERVER_PORT" 2>/dev/null || echo "WARNING: Cannot connect to logserver. Will retry when container starts." - -# Pull the Docker image -docker pull "$IMAGE_REGISTRY/$IMAGE_REPO:$IMAGE_TAG" || _die "Failed to pull Filebeat image" +echo "Testing connectivity to log server at ${LOGSERVER_HOST}:${LOGSERVER_PORT}..." +nc -zv "$LOGSERVER_HOST" "$LOGSERVER_PORT" 2>/dev/null || echo "WARNING: Cannot connect to log server. Will retry when container starts." # Stop any existing container -bash ./stop.sh || true +bash ./stop.sh 2>/dev/null || true -# Remove old container -_remove_container "$CONTAINER_NAME" || true - -# Generate Filebeat configuration (with actual hostname) -echo "Generating Filebeat configuration..." -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# Generate configuration +echo "Generating configuration..." export HOSTNAME=$(hostname) bash "$SCRIPT_DIR/scripts/generate-config.sh" || _die "Failed to generate configuration" -# Create volumes using common function -source "$SCRIPT_DIR/_volumes.sh" -echo "Creating volumes..." -create_items $(get_logclient_volumes) +# Start the client +echo "Starting Log Client..." +docker compose up -d || _die "Failed to start" -# Start the new container -bash ./start.sh || _die "Failed to start Filebeat" - -echo "Installation of ${CONTAINER_NAME} complete" -echo "Collecting logs from Docker API and shipping to ${LOGSERVER_HOST}:${LOGSERVER_PORT}" -echo "Using API key authentication" \ No newline at end of file +echo "" +echo "=========================================" +echo "Log Client Installed!" +echo "=========================================" +echo "" +echo "Shipping logs to: ${LOGSERVER_HOST}:${LOGSERVER_PORT}" +echo "Using authentication: ${LOKI_USER:-logclient}" +echo "Hostname label: $(hostname)" +echo "" +echo "Collecting:" +echo " - All Docker container logs" +echo " - System logs (/var/log)" +echo "" +echo "View logs at: http://${LOGSERVER_HOST}:3000" +echo "=========================================" \ No newline at end of file diff --git a/logclient/logs.sh b/logclient/logs.sh new file mode 100755 index 0000000..0ed8106 --- /dev/null +++ b/logclient/logs.sh @@ -0,0 +1,5 @@ +#!/bin/bash +source "${AGENT_PATH}/common.sh" +_check_required_env_vars "CONTAINER_NAME" + +docker logs "${CONTAINER_NAME}" --tail 50 \ No newline at end of file diff --git a/logclient/restore.sh b/logclient/restore.sh deleted file mode 100755 index 6dc594a..0000000 --- a/logclient/restore.sh +++ /dev/null @@ -1,47 +0,0 @@ -#!/bin/bash -source "${AGENT_PATH}/common.sh" -source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh" - -_check_required_env_vars "CONTAINER_NAME" - -if [ -z "$1" ]; then - echo "Usage: $0 " - echo "Available backups:" - ls -la "${CONFIG_PATH}/backups/" 2>/dev/null || echo "No backups found" - exit 1 -fi - -BACKUP_DIR="$1" - -if [ ! -d "$BACKUP_DIR" ]; then - _die "Backup directory not found: $BACKUP_DIR" -fi - -echo "Restoring from backup: $BACKUP_DIR" - -# Stop the container -bash ./stop.sh || true - -# Restore volumes -for volume in $(get_logclient_volumes); do - volume_name=$(echo $volume | cut -d: -f3) - backup_file="$BACKUP_DIR/${volume_name}.tar.gz" - - if [ -f "$backup_file" ]; then - echo "Restoring volume: $volume_name" - docker run --rm -v "$volume_name:/target" -v "$BACKUP_DIR:/backup:ro" alpine \ - sh -c "rm -rf /target/* && tar -xzf /backup/${volume_name}.tar.gz -C /target" - fi -done - -# Restore configuration if exists -if [ -d "$BACKUP_DIR/config_backup" ]; then - cp -r "$BACKUP_DIR/config_backup/"* "${CONFIG_PATH}/" -fi - -echo "Restore completed" - -# Start the container -bash ./start.sh - -echo "Container restarted" \ No newline at end of file diff --git a/logclient/scripts/generate-config.sh b/logclient/scripts/generate-config.sh index 0126c8e..864c68b 100755 --- a/logclient/scripts/generate-config.sh +++ b/logclient/scripts/generate-config.sh @@ -1,163 +1,111 @@ #!/bin/bash -# Generate Filebeat configuration from template -# This script creates a filebeat.yml configuration file with proper authentication +# Generate Promtail configuration for Log Client # Check required variables -if [ -z "$LOGSERVER_HOST" ] || [ -z "$LOGSERVER_PORT" ]; then +if [ -z "$LOGSERVER_HOST" ] || [ -z "$LOGSERVER_PORT" ] || [ -z "$LOKI_PASSWORD" ]; then echo "ERROR: Required environment variables not set" echo " LOGSERVER_HOST: ${LOGSERVER_HOST:-NOT SET}" echo " LOGSERVER_PORT: ${LOGSERVER_PORT:-NOT SET}" + echo " LOKI_PASSWORD: ${LOKI_PASSWORD:-NOT SET}" echo "" echo "Please set these in config/service.env before running install" exit 1 fi -# Determine config directory - use CONFIG_PATH from dropshell or fallback -if [ -n "$CONFIG_PATH" ]; then - CONFIG_DIR="$CONFIG_PATH" -elif [ -d "./config" ]; then - CONFIG_DIR="./config" -else - CONFIG_DIR="." -fi +# Get actual hostname +ACTUAL_HOSTNAME=${HOSTNAME_LABEL:-${HOSTNAME:-$(hostname 2>/dev/null || echo "unknown")}} -# Ensure config directory exists +# Determine config directory +CONFIG_DIR="${CONFIG_PATH:-./config}" mkdir -p "$CONFIG_DIR" -# Set defaults for variables if not set -BULK_MAX_SIZE=${BULK_MAX_SIZE:-2048} -WORKER_THREADS=${WORKER_THREADS:-1} -QUEUE_SIZE=${QUEUE_SIZE:-4096} -MAX_BACKOFF=${MAX_BACKOFF:-60s} +# Generate promtail.yaml configuration +cat > "$CONFIG_DIR/promtail.yaml" << EOF +server: + http_listen_port: 9080 + grpc_listen_port: 0 -# Get actual hostname from the host system -ACTUAL_HOSTNAME=${HOSTNAME:-$(hostname 2>/dev/null || echo "unknown")} +positions: + filename: /tmp/positions/positions.yaml -# Generate filebeat.yml configuration with variable substitution -( -cat << 'TEMPLATE_EOF' -# Filebeat Configuration for LogClient -# Generated by generate-config.sh +clients: + - url: http://${LOKI_USER:-logclient}:${LOKI_PASSWORD}@${LOGSERVER_HOST}:${LOGSERVER_PORT}/loki/api/v1/push + # Authentication via URL (HTTP Basic Auth) -# ======================== Docker Input Configuration ========================= -# Use Docker input to collect logs via Docker API -filebeat.inputs: -- type: docker - enabled: true - # Collect from all containers - containers.ids: - - '*' - # Collect both stdout and stderr - containers.stream: all - # Combine partial log lines - combine_partial: true - # Add Docker metadata - processors: - - add_docker_metadata: - host: "unix:///var/run/docker.sock" +scrape_configs: + # Docker container logs via Docker API + - job_name: docker + docker_sd_configs: + - host: unix:///var/run/docker.sock + refresh_interval: 5s + relabel_configs: + - source_labels: ['__meta_docker_container_name'] + regex: '/(.*)' + target_label: 'container_name' + - source_labels: ['__meta_docker_container_id'] + target_label: 'container_id' + - source_labels: ['__meta_docker_container_image'] + target_label: 'image' + - target_label: 'hostname' + replacement: '${ACTUAL_HOSTNAME}' + - target_label: 'job' + replacement: 'docker' -# ======================== System Logs Configuration ========================== -- type: log - enabled: true - paths: - - /var/log/syslog - - /var/log/messages - exclude_lines: ['^#'] - fields: - log_type: syslog + # System logs + - job_name: syslog + static_configs: + - targets: + - localhost + labels: + job: syslog + hostname: ${ACTUAL_HOSTNAME} + __path__: /var/log/syslog -- type: log - enabled: true - paths: - - /var/log/auth.log - - /var/log/secure - exclude_lines: ['^#'] - fields: - log_type: auth + - job_name: messages + static_configs: + - targets: + - localhost + labels: + job: messages + hostname: ${ACTUAL_HOSTNAME} + __path__: /var/log/messages -# ======================== Processors Configuration =========================== -processors: - - add_host_metadata: - when.not.contains: - tags: forwarded - # Override hostname with actual host's hostname - - add_fields: - target: agent - fields: - hostname: __ACTUAL_HOSTNAME__ - - add_fields: - target: host - fields: - name: __ACTUAL_HOSTNAME__ + - job_name: auth + static_configs: + - targets: + - localhost + labels: + job: auth + hostname: ${ACTUAL_HOSTNAME} + __path__: /var/log/auth.log -# ======================== Output Configuration =============================== -output.logstash: - hosts: ["__LOGSERVER_HOST__:__LOGSERVER_PORT__"] - # SSL/TLS configuration - ssl.enabled: false # Set to true when using TLS - ssl.verification_mode: none # Set to full in production with proper certs + # Docker container JSON logs (backup method) + - job_name: containers + static_configs: + - targets: + - localhost + labels: + job: containers + hostname: ${ACTUAL_HOSTNAME} + __path__: /var/lib/docker/containers/*/*-json.log + pipeline_stages: + - json: + expressions: + output: log + stream: stream + time: time + - timestamp: + source: time + format: RFC3339Nano + - labels: + stream: + - output: + source: output +EOF - # Performance settings - bulk_max_size: __BULK_MAX_SIZE__ - worker: __WORKER_THREADS__ # Must be >= 1 - compression_level: 3 - - # Retry configuration - max_retries: 3 - backoff.init: 1s - backoff.max: __MAX_BACKOFF__ - -# ======================== Global Fields ======================================= -# Add API key as a field to all events -fields: - api_key: "__API_KEY__" -fields_under_root: false - -# ======================== Queue Configuration ================================ -queue.mem: - events: __QUEUE_SIZE__ - flush.min_events: 512 - flush.timeout: 5s - -# ======================== Logging Configuration ============================== -logging.level: info -logging.to_files: true -logging.files: - path: /usr/share/filebeat/data/logs - name: filebeat - keepfiles: 3 - permissions: 0600 - -# ======================== Monitoring ========================================== -monitoring.enabled: false -http.enabled: true -http.host: 0.0.0.0 -http.port: 5066 - -# ======================== File Permissions ==================================== -# Set strict permissions (disabled for Docker) -# filebeat.config.modules.path: ${path.config}/modules.d/*.yml -TEMPLATE_EOF -) | sed -e "s|__LOGSERVER_HOST__|${LOGSERVER_HOST}|g" \ - -e "s|__LOGSERVER_PORT__|${LOGSERVER_PORT}|g" \ - -e "s|__API_KEY__|${API_KEY}|g" \ - -e "s|__BULK_MAX_SIZE__|${BULK_MAX_SIZE}|g" \ - -e "s|__WORKER_THREADS__|${WORKER_THREADS}|g" \ - -e "s|__QUEUE_SIZE__|${QUEUE_SIZE}|g" \ - -e "s|__MAX_BACKOFF__|${MAX_BACKOFF}|g" \ - -e "s|__ACTUAL_HOSTNAME__|${ACTUAL_HOSTNAME}|g" > "$CONFIG_DIR/filebeat.yml" - -echo "Filebeat configuration generated at: $CONFIG_DIR/filebeat.yml" +echo "Promtail configuration generated at: $CONFIG_DIR/promtail.yaml" echo "Configuration:" echo " LOGSERVER_HOST: ${LOGSERVER_HOST}" echo " LOGSERVER_PORT: ${LOGSERVER_PORT}" -echo " API_KEY: ${API_KEY:+[SET]}" -echo " WORKER_THREADS: ${WORKER_THREADS}" - -# Additional warning if API_KEY is not set -if [ -z "$API_KEY" ]; then - echo "" - echo "WARNING: API_KEY is not set - logs may be rejected by the server" - echo "Get an API key from the LogServer admin using generate-api-key.sh" -fi \ No newline at end of file +echo " HOSTNAME: ${ACTUAL_HOSTNAME}" \ No newline at end of file diff --git a/logclient/start.sh b/logclient/start.sh index 8043c0e..6565bbd 100755 --- a/logclient/start.sh +++ b/logclient/start.sh @@ -1,36 +1,6 @@ #!/bin/bash source "${AGENT_PATH}/common.sh" -_check_required_env_vars "CONTAINER_NAME" "IMAGE_REGISTRY" "IMAGE_REPO" "IMAGE_TAG" +_check_required_env_vars "CONTAINER_NAME" -# Check that config file exists -if [ ! -f "${CONFIG_PATH}/filebeat.yml" ]; then - _die "filebeat.yml not found in ${CONFIG_PATH}/filebeat.yml" -fi - -# Create Docker command -DOCKER_RUN_CMD="docker run -d \ - --name $CONTAINER_NAME \ - --restart unless-stopped \ - --user root \ - -v /var/run/docker.sock:/var/run/docker.sock:ro \ - -v /var/log:/var/log:ro \ - -v ${CONFIG_PATH}:/usr/share/filebeat/config:ro \ - -v ${DATA_VOLUME}:/usr/share/filebeat/data \ - -e LOGSERVER_HOST=${LOGSERVER_HOST} \ - -e LOGSERVER_PORT=${LOGSERVER_PORT} \ - -e API_KEY=${API_KEY} \ - -e HOSTNAME=$(hostname) \ - $IMAGE_REGISTRY/$IMAGE_REPO:$IMAGE_TAG \ - filebeat -e -strict.perms=false \ - -c /usr/share/filebeat/config/filebeat.yml" - -if ! _create_and_start_container "$DOCKER_RUN_CMD" "$CONTAINER_NAME"; then - _die "Failed to start container ${CONTAINER_NAME}" -fi - -# Check if the container is running -if ! _is_container_running "$CONTAINER_NAME"; then - _die "Container ${CONTAINER_NAME} is not running" -fi - -echo "Container ${CONTAINER_NAME} started" \ No newline at end of file +docker compose up -d || _die "Failed to start" +echo "Log Client started" \ No newline at end of file diff --git a/logclient/status.sh b/logclient/status.sh index 45bfe3e..c2e3032 100755 --- a/logclient/status.sh +++ b/logclient/status.sh @@ -2,30 +2,10 @@ source "${AGENT_PATH}/common.sh" _check_required_env_vars "CONTAINER_NAME" -# Check if container exists -if ! docker ps -a --format "{{.Names}}" | grep -q "^${CONTAINER_NAME}$"; then - echo "Unknown" - exit 0 -fi - -# Check container state -STATE=$(docker inspect -f '{{.State.Status}}' "$CONTAINER_NAME" 2>/dev/null) -case "$STATE" in - running) - # Additional check: verify connection to logserver - if docker logs "$CONTAINER_NAME" 2>&1 | tail -20 | grep -q "ERROR"; then - echo "Error" - else - echo "Running" - fi - ;; - exited|stopped) - echo "Stopped" - ;; - restarting|paused) - echo "Error" - ;; - *) - echo "Unknown" - ;; -esac \ No newline at end of file +if docker ps | grep -q "${CONTAINER_NAME}"; then + echo "Running" + echo " Shipping logs to: ${LOGSERVER_HOST}:${LOGSERVER_PORT}" + docker ps --format "table {{.Names}}\t{{.Status}}" | grep "${CONTAINER_NAME}" +else + echo "Stopped" +fi \ No newline at end of file diff --git a/logclient/stop.sh b/logclient/stop.sh index 7e9b8c6..798f35f 100755 --- a/logclient/stop.sh +++ b/logclient/stop.sh @@ -2,6 +2,5 @@ source "${AGENT_PATH}/common.sh" _check_required_env_vars "CONTAINER_NAME" -docker stop "$CONTAINER_NAME" 2>/dev/null || true - -echo "Filebeat stopped" \ No newline at end of file +docker compose stop || _die "Failed to stop" +echo "Log Client stopped" \ No newline at end of file diff --git a/logclient/uninstall.sh b/logclient/uninstall.sh index 1df1674..2e60077 100755 --- a/logclient/uninstall.sh +++ b/logclient/uninstall.sh @@ -2,16 +2,5 @@ source "${AGENT_PATH}/common.sh" _check_required_env_vars "CONTAINER_NAME" -# Stop the container -bash ./stop.sh || _die "Failed to stop container" - -# Remove the container -_remove_container "$CONTAINER_NAME" || _die "Failed to remove container" - -# CRITICAL: Never remove data volumes in uninstall.sh! -# Data volumes must be preserved for potential reinstallation -# Configuration and certificates must be preserved - -echo "Uninstallation of ${CONTAINER_NAME} complete" -echo "Note: Configuration and certificates have been preserved." -echo "To remove all data, use destroy.sh" \ No newline at end of file +docker compose down || true +echo "Log Client uninstalled" \ No newline at end of file diff --git a/logserver/DOCUMENTATION.md b/logserver/DOCUMENTATION.md deleted file mode 100644 index 2bd0ce2..0000000 --- a/logserver/DOCUMENTATION.md +++ /dev/null @@ -1,279 +0,0 @@ -# Dropshell LogServer Template - -A comprehensive centralized logging solution using the ELK Stack (Elasticsearch, Logstash, Kibana) for receiving, processing, and visualizing logs from multiple hosts. - -## Overview - -This template deploys a full-featured ELK stack that: -- Receives logs from multiple sources via Beats protocol -- Stores and indexes logs in Elasticsearch -- Provides powerful search and visualization through Kibana -- Supports automatic log parsing and enrichment -- Handles Docker container logs and system logs from clients - -## Architecture - -### Components - -1. **Elasticsearch** (7.17.x) - - Distributed search and analytics engine - - Stores and indexes all log data - - Provides fast full-text search capabilities - - Single-node configuration for simplicity (can be scaled) - -2. **Logstash** (7.17.x) - - Log processing pipeline - - Receives logs from Filebeat clients - - Parses and enriches log data - - Routes logs to appropriate Elasticsearch indices - -3. **Kibana** (7.17.x) - - Web UI for log exploration and visualization - - Create dashboards and alerts - - Real-time log streaming - - Advanced search queries - -## Features - -### Minimum Configuration Design -- Auto-discovery of log formats -- Pre-configured dashboards for common services -- Automatic index lifecycle management -- Built-in parsing for Docker and syslog formats -- Zero-configuration client connectivity - -### Log Processing -- Automatic timestamp extraction -- Docker metadata enrichment (container name, image, labels) -- Syslog parsing with severity levels -- JSON log support -- Multi-line log handling (stacktraces, etc.) -- Grok pattern matching for common formats - -### Security & Performance -- **Mutual TLS (mTLS)** authentication for client connections -- **API key authentication** as an alternative to certificates -- **Per-client authentication** with unique keys/certificates -- **SSL/TLS encryption** for all client connections -- **Basic authentication** for Kibana web access -- **IP whitelisting** for additional security -- Index lifecycle management for storage optimization -- Automatic old log cleanup -- Resource limits to prevent overconsumption - -## Port Configuration - -- **5601**: Kibana Web UI (HTTP/HTTPS with authentication) -- **9200**: Elasticsearch REST API (HTTP) - internal only -- **5044**: Logstash Beats input (TCP/TLS) - authenticated client connections -- **514**: Syslog input (UDP/TCP) - optional, unauthenticated -- **24224**: Fluentd forward input - optional Docker logging driver - -## Storage Requirements - -- **Minimum**: 10GB for basic operation -- **Recommended**: 50GB+ depending on log volume -- **Log Retention**: Default 30 days (configurable) - -## Client Authentication - -### Authentication Methods - -1. **Mutual TLS (mTLS) - Recommended** - - Each client gets a unique certificate signed by the server's CA - - Strongest security with mutual authentication - - Automatic certificate validation - -2. **API Keys** - - Each client gets a unique API key - - Simpler to manage than certificates - - Good for environments where certificate management is difficult - -3. **Basic Auth (Not Recommended)** - - Shared username/password - - Least secure, only for testing - -### Client Configuration - -Clients using the `logclient` template will: -1. Authenticate using provided credentials (cert/key or API key) -2. Establish encrypted TLS connection -3. Ship all Docker container logs -4. Ship system logs (syslog, auth, kernel) -5. Maintain connection with automatic reconnection -6. Buffer logs locally during network outages - -## Dashboard Features - -### Pre-configured Dashboards -- **System Overview**: Overall health and log volume metrics -- **Docker Containers**: Container-specific logs and metrics -- **Error Analysis**: Aggregated error logs from all sources -- **Security Events**: Authentication and access logs -- **Application Logs**: Parsed application-specific logs - -### Search Capabilities -- Full-text search across all logs -- Filter by time range, host, container, severity -- Save and share search queries -- Export search results - -## Resource Requirements - -### Minimum -- CPU: 2 cores -- RAM: 4GB -- Storage: 10GB - -### Recommended -- CPU: 4+ cores -- RAM: 8GB+ -- Storage: 50GB+ SSD - -## Configuration Options - -### Environment Variables (service.env) - -```bash -# Elasticsearch settings -ES_HEAP_SIZE=2g -ES_MAX_MAP_COUNT=262144 - -# Logstash settings -LS_HEAP_SIZE=1g -LS_PIPELINE_WORKERS=2 - -# Kibana settings -KIBANA_PASSWORD=changeme -KIBANA_BASE_PATH=/ - -# Log retention -LOG_RETENTION_DAYS=30 -LOG_MAX_SIZE_GB=50 - -# Authentication Mode -AUTH_MODE=mtls # Options: mtls, apikey, basic -ENABLE_TLS=true - -# mTLS Settings (if AUTH_MODE=mtls) -CA_CERT_PATH=/certs/ca.crt -SERVER_CERT_PATH=/certs/server.crt -SERVER_KEY_PATH=/certs/server.key -CLIENT_CERT_REQUIRED=true - -# API Key Settings (if AUTH_MODE=apikey) -API_KEYS_PATH=/config/api-keys.yml - -# Network Security -ALLOWED_IPS="" # Comma-separated list, empty = all -``` - -## Usage - -### Installation -```bash -dropshell install logserver -``` - -### Generate Client Credentials - -#### For mTLS Authentication: -```bash -# Generate client certificate for a new host -dropshell exec logserver /scripts/generate-client-cert.sh hostname -# This creates hostname.crt and hostname.key files -``` - -#### For API Key Authentication: -```bash -# Generate API key for a new client -dropshell exec logserver /scripts/generate-api-key.sh hostname -# Returns an API key to configure in the client -``` - -### Access Kibana -Navigate to `https://:5601` in your browser. - -Default credentials: -- Username: `elastic` -- Password: `changeme` (change in service.env) - -### View Logs -```bash -dropshell logs logserver -``` - -### Backup -```bash -dropshell backup logserver -``` - -## Troubleshooting - -### Common Issues - -1. **Elasticsearch failing to start** - - Check vm.max_map_count: `sysctl vm.max_map_count` (should be 262144+) - - Verify sufficient memory available - -2. **No logs appearing in Kibana** - - Check Logstash is receiving data: port 5044 should be open - - Verify client connectivity - - Check index patterns in Kibana - -3. **High memory usage** - - Adjust heap sizes in service.env - - Configure index lifecycle management - - Reduce retention period - -## Integration - -This template is designed to work seamlessly with the `logclient` template. Simply: -1. Deploy this logserver -2. Deploy logclient on each host you want to monitor -3. Configure logclient with the logserver address -4. Logs will automatically start flowing - -## Security Considerations - -1. **Authentication Setup** - - Use mTLS for production environments - - Generate unique credentials for each client - - Rotate certificates/keys regularly - - Store credentials securely - -2. **Network Security** - - Always use TLS encryption for client connections - - Configure IP whitelisting when possible - - Use firewall rules to restrict access - - Consider VPN or private networks - -3. **Access Control** - - Change default Kibana password immediately - - Create read-only users for viewing logs - - Implement role-based access control (RBAC) - - Audit access logs regularly - -4. **Data Protection** - - Regular backups of Elasticsearch indices - - Encrypt data at rest (optional) - - Monitor disk usage to prevent data loss - - Implement log retention policies - -## Maintenance - -### Daily Tasks -- Monitor disk usage -- Check for failed log shipments -- Review error dashboards - -### Weekly Tasks -- Verify all clients are reporting -- Check index health -- Review and optimize slow queries - -### Monthly Tasks -- Update ELK stack components -- Archive old indices -- Review retention policies -- Performance tuning based on usage patterns \ No newline at end of file diff --git a/logserver/README.md b/logserver/README.md index d44fdfb..6162796 100644 --- a/logserver/README.md +++ b/logserver/README.md @@ -1,50 +1,73 @@ -# LogServer +# Log Server (Loki + Grafana) -Centralized logging with ELK Stack (Elasticsearch, Logstash, Kibana). +Central log server using Grafana + Loki. MUCH simpler than ELK/Kibana! ## Quick Start -1. **System Setup** -```bash -sudo sysctl -w vm.max_map_count=262144 -``` - -2. **Configure** -Edit `config/service.env`: -- Set `SERVER_PUBLICBASEURL` to your actual server URL -- Set `KIBANA_USERNAME` to your preferred username -- Change `KIBANA_USER_PASSWORD` from default - -3. **Install** +1. **Install** ```bash dropshell install logserver ``` -4. **Generate Client Keys** +2. **Access Web UI** +- URL: `http://:3000` +- Login: `admin` / `changeme` (set in service.env) + +3. **View Logs** +- Click **Dashboards** (4 squares icon) +- Click **"Central Logs"** +- All logs from all servers appear! + +## For Client Servers + +On each server that should send logs here: + +1. Edit `logclient/config/service.env`: ```bash -./generate-api-key.sh -# Enter hostname when prompted -# Save the API key for client configuration +LOGSERVER_HOST= +LOGSERVER_PORT=3100 +LOKI_USER=logclient # Default username +LOKI_PASSWORD= # From server's service.env ``` -5. **Setup Kibana** (first time only) +2. Install the client: ```bash -./setup-kibana.sh +dropshell install logclient ``` -6. **Access Kibana** -- URL: `http://:5601` -- Username: Set in `service.env` (KIBANA_USERNAME, default: `admin`) -- Password: Set in `service.env` (KIBANA_USER_PASSWORD) -- Click "Discover" → View your logs! +## Features +- **Super simple interface** - Just one dashboard +- **Lightweight** - Uses 10x less resources than ELK +- **Real-time** - Logs appear instantly +- **Multi-server** - See all servers in one place +- **Automatic labels** - Filter by hostname, container, etc. + +## Security +- Loki endpoint requires authentication (HTTP Basic Auth) +- Default: `logclient` / `changeme` +- **IMPORTANT**: Change `LOKI_PASSWORD` in service.env! ## Ports -- `5601` - Kibana Web UI -- `5044` - Log ingestion (Filebeat) +- `3000` - Grafana Web UI +- `3100` - Loki API (authenticated) -## Files -- `config/service.env` - Configuration -- `config/api-keys.yml` - Client API keys -- `generate-api-key.sh` - Add new clients +## How to Use -See [DOCUMENTATION.md](DOCUMENTATION.md) for full details. \ No newline at end of file +### Filter by Server +- Use the "Server" dropdown at the top +- Or click any `hostname` label + +### Filter by Container +- Click any `container_name` label + +### Search Text +- Use the search box in each panel + +### Time Range +- Top-right corner - adjust as needed + +## Why This Instead of ELK? +- **10x simpler** - One clean dashboard vs Kibana complexity +- **10x smaller** - ~200MB RAM vs 2-3GB for ELK +- **Zero configuration** - Just works +- **Fast setup** - No index patterns, mappings, or complexity \ No newline at end of file diff --git a/logserver/SETUP.md b/logserver/SETUP.md deleted file mode 100644 index 187d715..0000000 --- a/logserver/SETUP.md +++ /dev/null @@ -1,100 +0,0 @@ -# LogServer Quick Setup Guide - -## Prerequisites -- Docker and Docker Compose installed -- 4GB+ RAM, 10GB+ disk space -- Port 5601 (Kibana) and 5044 (Logstash) available - -## Initial Setup - -### 1. System Configuration -```bash -# Required for Elasticsearch -sudo sysctl -w vm.max_map_count=262144 -echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf -``` - -### 2. Configure Server -Edit `config/service.env`: -```bash -# Change default password -KIBANA_PASSWORD=your-secure-password -``` - -### 3. Install -```bash -dropshell install logserver -``` - -## Generate Client API Keys - -Run the interactive key generator: -```bash -./generate-api-key.sh -``` - -Follow the prompts: -1. Enter hostname for each client -2. Script generates secure API key -3. Shows configuration to copy to client -4. Repeat for additional clients - -## Access Kibana - -1. Open browser: `http://your-server-ip:5601` -2. Login: `elastic` / `your-secure-password` -3. Create index pattern: `filebeat-*` -4. View logs in Discover tab - -## Add Log Clients - -On each client machine: -```bash -# Get API key from server admin (they run ./generate-api-key.sh) - -# Edit logclient/config/service.env: -LOGSERVER_HOST=your-server-ip -LOGSERVER_PORT=5044 -API_KEY=your-api-key-here - -# Install and start -dropshell install logclient -``` - -## Verify Setup - -```bash -# Check server status -dropshell status logserver - -# View server logs -dropshell logs logserver - -# Test client connection (from client) -docker logs logclient-filebeat | grep "connection" -``` - -## Troubleshooting - -**Elasticsearch won't start**: Check `vm.max_map_count` is 262144+ - -**No logs in Kibana**: -- Verify client can reach server port 5044 -- Check API key is correct in client's service.env -- Verify API key exists in server's api-keys.yml -- Refresh index pattern in Kibana - -**High memory usage**: Adjust heap sizes in `service.env`: -```bash -ES_HEAP_SIZE=1g # Reduce from 2g -LS_HEAP_SIZE=512m # Reduce from 1g -``` - -## Security Checklist - -- [ ] Changed default Kibana password -- [ ] Generated unique API key per client -- [ ] API keys stored securely -- [ ] Firewall allows only necessary ports (5601, 5044) -- [ ] Regular backup configured -- [ ] Reviewed api-keys.yml for old/unused keys \ No newline at end of file diff --git a/logserver/SIMPLE_GUIDE.md b/logserver/SIMPLE_GUIDE.md deleted file mode 100644 index e35e91d..0000000 --- a/logserver/SIMPLE_GUIDE.md +++ /dev/null @@ -1,84 +0,0 @@ -# Simple Guide to Viewing Logs in Kibana - -## First Time Setup -Run this once after installing LogServer: -```bash -./setup-kibana.sh -``` - -## Viewing Logs - The Easy Way - -### 1. Open Kibana -Go to: `http://:5601` - -### 2. Login -Use the username and password from your service.env - -### 3. Click "Discover" -It's in the left menu (looks like a compass icon) - -### 4. You're Done! -Your logs are now visible. That's it! - -## Simple Controls - -### See Recent Logs Only -- Top-right corner: Click the time picker -- Choose "Last 15 minutes" or "Last 1 hour" - -### Filter by Container -- Find any log entry -- Next to `container_name`: click the `+` button -- Now you only see logs from that container - -### Filter by Server -- Next to `host.name`: click the `+` button -- Now you only see logs from that host - -### Search for Text -- Top search bar: Type any word -- Press Enter -- Shows only logs containing that word - -### Live Updates -- Top-right: Click "Refresh" -- Choose "Every 5 seconds" -- Logs update automatically - -### Remove Filters -- Look for filter pills under the search bar -- Click the `x` on any filter to remove it - -## Common Searches - -**Show errors only:** -``` -error OR ERROR OR Error -``` - -**Show warnings and errors:** -``` -error OR ERROR OR warn OR WARN -``` - -**Show specific container:** -``` -container_name: "myapp" -``` - -**Show multiple containers:** -``` -container_name: ("app1" OR "app2") -``` - -## Tips - -1. **Too many columns?** Click "container_name" and "message" in the left sidebar to show just those - -2. **Want raw logs?** Click the ">" arrow next to any log entry to expand it - -3. **Export logs?** Click "Share" → "CSV Reports" → "Generate CSV" - -4. **Time zone wrong?** Click your profile icon → "Advanced Settings" → search "timezone" - -That's all you need to know! Kibana has many advanced features, but for basic log viewing and searching, these commands are sufficient. \ No newline at end of file diff --git a/logserver/TODO.md b/logserver/TODO.md deleted file mode 100644 index 2b73a35..0000000 --- a/logserver/TODO.md +++ /dev/null @@ -1,244 +0,0 @@ -# LogServer Template - Implementation TODO - -## Phase 1: Core Infrastructure (Priority 1) - -### Configuration Files -- [ ] Create `config/.template_info.env` with template metadata -- [ ] Create `config/service.env` with user-configurable settings -- [ ] Define all required environment variables (ports, passwords, heap sizes) -- [ ] Set appropriate default values for zero-config experience - -### Docker Compose Setup -- [ ] Create `docker-compose.yml` with ELK stack services -- [ ] Configure Elasticsearch single-node setup -- [ ] Configure Logstash with Beats input pipeline -- [ ] Configure Kibana with Elasticsearch connection -- [ ] Set up proper networking between services -- [ ] Define named volumes for data persistence -- [ ] Configure health checks for each service - -### Required Scripts -- [ ] Implement `install.sh` - Pull images, create volumes, start services -- [ ] Implement `uninstall.sh` - Stop and remove containers (preserve volumes!) -- [ ] Implement `start.sh` - Start all ELK services with docker-compose -- [ ] Implement `stop.sh` - Gracefully stop all services -- [ ] Implement `status.sh` - Check health of all three services - -## Phase 2: Logstash Configuration (Priority 1) - -### Input Configuration -- [ ] Configure Beats input on port 5044 with TLS/SSL -- [ ] Set up mutual TLS (mTLS) authentication -- [ ] Configure client certificate validation -- [ ] Add API key authentication option -- [ ] Implement IP whitelisting -- [ ] Add Syslog input on port 514 (UDP/TCP) - unauthenticated -- [ ] Add Docker Fluentd input on port 24224 (optional) - -### Filter Pipeline -- [ ] Create Docker log parser (extract container metadata) -- [ ] Create Syslog parser (RFC3164 and RFC5424) -- [ ] Add JSON parser for structured logs -- [ ] Implement multiline pattern for stack traces -- [ ] Add timestamp extraction and normalization -- [ ] Create field enrichment (add host metadata) -- [ ] Implement conditional routing based on log type - -### Output Configuration -- [ ] Configure Elasticsearch output with index patterns -- [ ] Set up index templates for different log types -- [ ] Configure index lifecycle management (ILM) - -## Phase 3: Elasticsearch Setup (Priority 1) - -### System Configuration -- [ ] Set appropriate heap size defaults (ES_HEAP_SIZE) -- [ ] Configure vm.max_map_count requirement check -- [ ] Set up single-node discovery settings -- [ ] Configure data persistence volume -- [ ] Set up index templates for: - - [ ] Docker logs (docker-*) - - [ ] System logs (syslog-*) - - [ ] Application logs (app-*) - - [ ] Error logs (errors-*) - -### Index Management -- [ ] Configure ILM policies for log rotation -- [ ] Set retention period (default 30 days) -- [ ] Configure max index size limits -- [ ] Set up automatic cleanup of old indices -- [ ] Create snapshot repository configuration - -## Phase 4: Kibana Configuration (Priority 2) - -### Initial Setup -- [ ] Configure Kibana with Elasticsearch URL -- [ ] Set up basic authentication -- [ ] Configure server base path -- [ ] Set appropriate memory limits - -### Pre-built Dashboards -- [ ] Create System Overview dashboard -- [ ] Create Docker Containers dashboard -- [ ] Create Error Analysis dashboard -- [ ] Create Security Events dashboard -- [ ] Create Host Metrics dashboard - -### Saved Searches -- [ ] Error logs across all sources -- [ ] Authentication events -- [ ] Container lifecycle events -- [ ] Slow queries/performance issues -- [ ] Critical system events - -### Index Patterns -- [ ] Configure docker-* pattern -- [ ] Configure syslog-* pattern -- [ ] Configure app-* pattern -- [ ] Configure filebeat-* pattern - -## Phase 5: Optional Scripts (Priority 2) - -### Operational Scripts -- [ ] Implement `logs.sh` - Show logs from all ELK services -- [ ] Implement `backup.sh` - Snapshot Elasticsearch indices -- [ ] Implement `restore.sh` - Restore from snapshots -- [ ] Implement `destroy.sh` - Complete removal including volumes -- [ ] Implement `ports.sh` - Display all exposed ports -- [ ] Implement `ssh.sh` - Shell into specific container - -### Helper Scripts -- [ ] Create `_volumes.sh` for volume management helpers -- [ ] Add health check script for all services -- [ ] Create performance tuning script -- [ ] Add certificate generation script for SSL - -## Phase 6: Security Features (Priority 1 - CRITICAL) - -### Certificate Authority Setup -- [ ] Create CA certificate and key for signing client certs -- [ ] Generate server certificate for Logstash -- [ ] Create certificate generation script for clients -- [ ] Set up certificate storage structure -- [ ] Implement certificate rotation mechanism - -### mTLS Authentication -- [ ] Configure Logstash for mutual TLS -- [ ] Set up client certificate validation -- [ ] Create client certificate generation script -- [ ] Implement certificate revocation list (CRL) -- [ ] Add certificate expiry monitoring - -### API Key Authentication -- [ ] Create API key generation script -- [ ] Configure Logstash to accept API keys -- [ ] Implement API key storage (encrypted) -- [ ] Add API key rotation mechanism -- [ ] Create API key revocation process - -### Network Security -- [ ] Implement IP whitelisting in Logstash -- [ ] Configure firewall rules -- [ ] Set up rate limiting -- [ ] Add connection throttling -- [ ] Implement DDoS protection - -### Kibana Security -- [ ] Configure Kibana HTTPS -- [ ] Set up basic authentication -- [ ] Create user management scripts -- [ ] Implement session management -- [ ] Add audit logging - -## Phase 7: Performance & Optimization (Priority 3) - -### Resource Management -- [ ] Configure CPU limits for each service -- [ ] Set memory limits appropriately -- [ ] Add swap handling configuration -- [ ] Configure JVM options files -- [ ] Add performance monitoring - -### Optimization -- [ ] Configure pipeline workers -- [ ] Set batch sizes for optimal throughput -- [ ] Configure queue sizes -- [ ] Add caching configuration -- [ ] Optimize index refresh intervals - -## Phase 8: Testing & Documentation (Priority 3) - -### Testing -- [ ] Test installation process -- [ ] Test uninstall (verify volume preservation) -- [ ] Test log ingestion from sample client -- [ ] Test all dashboard functionality -- [ ] Test backup and restore procedures -- [ ] Load test with high log volume -- [ ] Test failover and recovery - -### Documentation -- [ ] Create README.txt for dropshell format -- [ ] Document all configuration options -- [ ] Add troubleshooting guide -- [ ] Create quick start guide -- [ ] Document upgrade procedures -- [ ] Add performance tuning guide - -## Phase 9: Integration Testing (Priority 3) - -### With LogClient -- [ ] Test automatic discovery -- [ ] Verify log flow from client to server -- [ ] Test reconnection scenarios -- [ ] Verify all log types are parsed correctly -- [ ] Test SSL communication -- [ ] Measure end-to-end latency - -### Compatibility Testing -- [ ] Test with different Docker versions -- [ ] Test on various Linux distributions -- [ ] Verify with different log formats -- [ ] Test with high-volume producers -- [ ] Validate resource usage - -## Phase 10: Production Readiness (Priority 4) - -### Monitoring & Alerting -- [ ] Add Elasticsearch monitoring -- [ ] Configure disk space alerts -- [ ] Set up index health monitoring -- [ ] Add performance metrics collection -- [ ] Create alert rules in Kibana - -### Maintenance Features -- [ ] Add automatic update check -- [ ] Create maintenance mode -- [ ] Add data export functionality -- [ ] Create migration scripts -- [ ] Add configuration validation - -## Notes - -### Design Principles -1. **Minimum configuration**: Should work with just `dropshell install logserver` -2. **Data safety**: Never delete volumes in uninstall.sh -3. **Non-interactive**: All scripts must run without user input -4. **Idempotent**: Scripts can be run multiple times safely -5. **Clear feedback**: Provide clear status and error messages - -### Dependencies -- Docker and Docker Compose -- Sufficient system resources (4GB+ RAM recommended) -- Network connectivity for clients -- Persistent storage for logs - -### Testing Checklist -- [ ] All required scripts present and executable -- [ ] Template validates with dropshell test-template -- [ ] Services start and connect properly -- [ ] Logs flow from client to Kibana -- [ ] Data persists across container restarts -- [ ] Uninstall preserves data volumes -- [ ] Resource limits are enforced -- [ ] Error handling works correctly \ No newline at end of file diff --git a/logserver/_volumes.sh b/logserver/_volumes.sh deleted file mode 100755 index ee9801b..0000000 --- a/logserver/_volumes.sh +++ /dev/null @@ -1,9 +0,0 @@ -#!/bin/bash -# Define volume items for logserver container -# These are used across backup, restore, create, and destroy operations - -get_logserver_volumes() { - echo "volume:elasticsearch_data:${CONTAINER_NAME}_elasticsearch_data" \ - "volume:logstash_data:${CONTAINER_NAME}_logstash_data" \ - "volume:kibana_data:${CONTAINER_NAME}_kibana_data" -} \ No newline at end of file diff --git a/logserver/backup.sh b/logserver/backup.sh deleted file mode 100755 index 5ffca67..0000000 --- a/logserver/backup.sh +++ /dev/null @@ -1,37 +0,0 @@ -#!/bin/bash -source "${AGENT_PATH}/common.sh" -source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh" - -_check_required_env_vars "CONTAINER_NAME" - -echo "Backing up ${CONTAINER_NAME} ELK stack volumes..." -echo "Note: This may take a while for large log databases" - -# Stop the containers to ensure data consistency -bash ./stop.sh || true - -# Backup volumes -BACKUP_DIR="${CONFIG_PATH}/backups/$(date +%Y%m%d_%H%M%S)" -mkdir -p "$BACKUP_DIR" - -# Export volumes -for volume in $(get_logserver_volumes); do - volume_name=$(echo $volume | cut -d: -f3) - echo "Backing up volume: $volume_name" - docker run --rm -v "$volume_name:/source:ro" -v "$BACKUP_DIR:/backup" alpine \ - tar -czf "/backup/${volume_name}.tar.gz" -C /source . -done - -# Backup configuration -cp -r "${CONFIG_PATH}" "$BACKUP_DIR/config_backup" - -# Backup docker-compose.yml -cp docker-compose.yml "$BACKUP_DIR/" 2>/dev/null || true - -echo "Backup completed to: $BACKUP_DIR" -echo "Size: $(du -sh $BACKUP_DIR | cut -f1)" - -# Restart the containers -bash ./start.sh - -echo "ELK stack restarted" \ No newline at end of file diff --git a/logserver/config/.template_info.env b/logserver/config/.template_info.env deleted file mode 100644 index 2c60810..0000000 --- a/logserver/config/.template_info.env +++ /dev/null @@ -1,17 +0,0 @@ -# Template identifier - MUST match the directory name -TEMPLATE=logserver - -# Requirements -REQUIRES_HOST_ROOT=false # No root access on host needed -REQUIRES_DOCKER=true # Docker is required -REQUIRES_DOCKER_ROOT=false # Docker root privileges not specifically needed - -# Docker compose used for ELK stack -USES_DOCKER_COMPOSE=true - -# Volume definitions for persistence -DATA_VOLUME="${CONTAINER_NAME}_elasticsearch_data" -LOGSTASH_VOLUME="${CONTAINER_NAME}_logstash_data" -KIBANA_VOLUME="${CONTAINER_NAME}_kibana_data" -CERTS_VOLUME="${CONTAINER_NAME}_certs" -CONFIG_VOLUME="${CONTAINER_NAME}_config" \ No newline at end of file diff --git a/logserver/config/api-keys.yml b/logserver/config/api-keys.yml deleted file mode 100644 index 40b9c0a..0000000 --- a/logserver/config/api-keys.yml +++ /dev/null @@ -1,6 +0,0 @@ -# API Keys for LogServer Authentication -# Format: hostname:api_key -# Generated by generate-api-key.sh - -api_keys: - video: a7798c63c2ac439b5ba20f3bf8bf27b5361231cdcbdc4fc9d7af715308fdf707 diff --git a/logserver/config/dashboards/central-logs.json b/logserver/config/dashboards/central-logs.json new file mode 100644 index 0000000..015d37c --- /dev/null +++ b/logserver/config/dashboards/central-logs.json @@ -0,0 +1,154 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": "-- Grafana --", + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "gnetId": null, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [ + { + "datasource": "Loki", + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "gridPos": { + "h": 3, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 4, + "options": { + "content": "## Central Log Viewer\n\n**Quick Filters:** Click any label (hostname, container_name, job) to filter | **Search:** Use the search box above each panel | **Time Range:** Top right corner", + "mode": "markdown" + }, + "pluginVersion": "7.5.7", + "title": "", + "type": "text" + }, + { + "datasource": "Loki", + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 3 + }, + "id": 2, + "options": { + "dedupStrategy": "none", + "enableLogDetails": true, + "prettifyLogMessage": false, + "showCommonLabels": false, + "showLabels": true, + "showTime": true, + "sortOrder": "Descending", + "wrapLogMessage": true + }, + "targets": [ + { + "expr": "{job=\"docker\"}", + "legendFormat": "{{hostname}} - {{container_name}}", + "refId": "A" + } + ], + "title": "Docker Container Logs (All Servers)", + "type": "logs" + }, + { + "datasource": "Loki", + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 13 + }, + "id": 3, + "options": { + "dedupStrategy": "none", + "enableLogDetails": true, + "prettifyLogMessage": false, + "showCommonLabels": false, + "showLabels": true, + "showTime": true, + "sortOrder": "Descending", + "wrapLogMessage": true + }, + "targets": [ + { + "expr": "{job=~\"syslog|auth\"}", + "legendFormat": "{{hostname}} - {{job}}", + "refId": "A" + } + ], + "title": "System Logs (All Servers)", + "type": "logs" + } + ], + "refresh": "10s", + "schemaVersion": 27, + "style": "dark", + "tags": ["logs", "central"], + "templating": { + "list": [ + { + "allValue": ".*", + "current": { + "selected": true, + "text": ["All"], + "value": ["$__all"] + }, + "datasource": "Loki", + "definition": "label_values(hostname)", + "description": "Filter by server", + "error": null, + "hide": 0, + "includeAll": true, + "label": "Server", + "multi": true, + "name": "hostname", + "options": [], + "query": "label_values(hostname)", + "refresh": 1, + "regex": "", + "skipUrlSync": false, + "sort": 1, + "type": "query" + } + ] + }, + "time": { + "from": "now-15m", + "to": "now" + }, + "timepicker": { + "refresh_intervals": ["5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h"] + }, + "timezone": "", + "title": "Central Logs", + "uid": "central-logs", + "version": 0 +} \ No newline at end of file diff --git a/logserver/config/dashboards/simple-logs.json b/logserver/config/dashboards/simple-logs.json new file mode 100644 index 0000000..8325052 --- /dev/null +++ b/logserver/config/dashboards/simple-logs.json @@ -0,0 +1,69 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": "-- Grafana --", + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "gnetId": null, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [ + { + "datasource": "Loki", + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 20, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 2, + "options": { + "dedupStrategy": "none", + "enableLogDetails": true, + "prettifyLogMessage": false, + "showCommonLabels": false, + "showLabels": true, + "showTime": true, + "sortOrder": "Descending", + "wrapLogMessage": true + }, + "targets": [ + { + "expr": "{job=~\"docker|syslog|auth\"}", + "refId": "A" + } + ], + "title": "All Logs", + "type": "logs" + } + ], + "schemaVersion": 27, + "style": "dark", + "tags": ["logs"], + "templating": { + "list": [] + }, + "time": { + "from": "now-15m", + "to": "now" + }, + "timepicker": {}, + "timezone": "", + "title": "Simple Logs", + "uid": "simple-logs", + "version": 0 +} \ No newline at end of file diff --git a/logserver/config/grafana-dashboard.yml b/logserver/config/grafana-dashboard.yml new file mode 100644 index 0000000..35c89ff --- /dev/null +++ b/logserver/config/grafana-dashboard.yml @@ -0,0 +1,12 @@ +apiVersion: 1 + +providers: + - name: 'default' + orgId: 1 + folder: '' + type: file + disableDeletion: false + updateIntervalSeconds: 10 + allowUiUpdates: true + options: + path: /var/lib/grafana/dashboards \ No newline at end of file diff --git a/logserver/config/grafana-datasources.yml b/logserver/config/grafana-datasources.yml new file mode 100644 index 0000000..c680e97 --- /dev/null +++ b/logserver/config/grafana-datasources.yml @@ -0,0 +1,9 @@ +apiVersion: 1 + +datasources: + - name: Loki + type: loki + access: proxy + url: http://loki:3100 + isDefault: true + editable: false \ No newline at end of file diff --git a/logserver/config/logstash.conf b/logserver/config/logstash.conf deleted file mode 100644 index 1804e68..0000000 --- a/logserver/config/logstash.conf +++ /dev/null @@ -1,142 +0,0 @@ -# Logstash Configuration for LogServer -# Handles Beats input with API key authentication - -input { - # Beats input for Filebeat clients - beats { - port => 5044 - ssl => false # Set to true for production with proper certificates - - # API key authentication handled via filter below - } - - # Optional: Syslog input for direct syslog shipping - tcp { - port => 514 - type => "syslog" - } - - udp { - port => 514 - type => "syslog" - } -} - -filter { - # API Key validation - check if client provided a valid key - # The API key should be in the [fields][api_key] field from Filebeat - if [fields][api_key] { - # Load and validate API key - ruby { - init => " - require 'yaml' - @api_keys = {} - - # Load API keys from file - begin - if File.exist?('/usr/share/logstash/config/api-keys.yml') - config = YAML.load_file('/usr/share/logstash/config/api-keys.yml') - if config && config['api_keys'] - config['api_keys'].each do |hostname, key| - @api_keys[key.to_s.strip] = hostname.to_s.strip - end - end - end - rescue => e - @logger.error('Failed to load API keys', :error => e.message) - end - " - code => " - api_key = event.get('[fields][api_key]') - - if api_key && @api_keys.has_key?(api_key) - # Valid API key - add hostname to event - event.set('[@metadata][client_hostname]', @api_keys[api_key]) - event.set('[@metadata][authenticated]', true) - else - # Invalid API key - event.set('[@metadata][authenticated]', false) - event.tag('_authfailure') - end - " - } - - # Drop unauthorized events - if "_authfailure" in [tags] { - drop { } - } - } else { - # No API key provided - mark as unauthenticated - # You can choose to drop these or allow them based on your security requirements - mutate { - add_tag => [ "no_api_key" ] - } - # Uncomment to require API keys for all connections: - # drop { } - } - - # Parse Docker logs - if [docker] { - # Docker metadata is already parsed by Filebeat - mutate { - add_field => { - "container_name" => "%{[docker][container][name]}" - "container_id" => "%{[docker][container][id]}" - "container_image" => "%{[docker][container][image]}" - } - } - } - - # Parse syslog - if [type] == "syslog" { - grok { - match => { - "message" => "%{SYSLOGLINE}" - } - } - date { - match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] - } - } - - # Parse JSON logs if they exist - if [message] =~ /^\{.*\}$/ { - json { - source => "message" - target => "json_message" - } - } - - # Add timestamp if not present - if ![timestamp] { - mutate { - add_field => { "timestamp" => "%{@timestamp}" } - } - } - - # Clean up metadata - mutate { - remove_field => [ "@version", "beat", "offset", "prospector" ] - } -} - -output { - # Send to Elasticsearch with authentication - elasticsearch { - hosts => ["elasticsearch:9200"] - user => "elastic" - password => "${ELASTIC_PASSWORD:changeme}" - - # Use different indices based on input type - index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" - - # Manage index templates - manage_template => true - template_overwrite => true - } - - # Optional: Debug output (comment out in production) - # stdout { - # codec => rubydebug - # } -} \ No newline at end of file diff --git a/logserver/config/logstash.yml b/logserver/config/logstash.yml deleted file mode 100644 index d967e2a..0000000 --- a/logserver/config/logstash.yml +++ /dev/null @@ -1,29 +0,0 @@ -# Logstash Configuration Settings -# This file contains Logstash settings (not pipeline configuration) - -# Node name -node.name: "logstash" - -# Pipeline settings -pipeline.workers: 2 -pipeline.batch.size: 125 -pipeline.batch.delay: 50 -pipeline.ecs_compatibility: disabled - -# HTTP API settings -http.host: "0.0.0.0" -http.port: 9600 - -# Monitoring -monitoring.enabled: false - -# Queue settings -queue.type: memory -queue.max_bytes: 1gb - -# Path settings are handled by Docker volumes -# path.data: /usr/share/logstash/data -# path.logs: /usr/share/logstash/logs - -# Log level -log.level: info \ No newline at end of file diff --git a/logserver/config/loki.yaml b/logserver/config/loki.yaml new file mode 100644 index 0000000..0b49c49 --- /dev/null +++ b/logserver/config/loki.yaml @@ -0,0 +1,51 @@ +auth_enabled: false + +server: + http_listen_port: 3100 + grpc_listen_port: 9096 + +ingester: + wal: + enabled: true + dir: /loki/wal + lifecycler: + address: 127.0.0.1 + ring: + kvstore: + store: inmemory + replication_factor: 1 + +schema_config: + configs: + - from: 2020-10-24 + store: boltdb-shipper + object_store: filesystem + schema: v11 + index: + prefix: index_ + period: 24h + +storage_config: + boltdb_shipper: + active_index_directory: /loki/boltdb-shipper-active + cache_location: /loki/boltdb-shipper-cache + cache_ttl: 24h + filesystem: + directory: /loki/chunks + +compactor: + working_directory: /loki/boltdb-shipper-compactor + +limits_config: + enforce_metric_name: false + reject_old_samples: true + reject_old_samples_max_age: 168h + max_entries_limit_per_query: 5000 + retention_period: 168h # 7 days + +chunk_store_config: + max_look_back_period: 168h + +table_manager: + retention_deletes_enabled: true + retention_period: 168h \ No newline at end of file diff --git a/logserver/config/nginx.conf b/logserver/config/nginx.conf new file mode 100644 index 0000000..6fa6cd1 --- /dev/null +++ b/logserver/config/nginx.conf @@ -0,0 +1,30 @@ +events { + worker_connections 1024; +} + +http { + upstream loki { + server loki:3100; + } + + server { + listen 80; + + # Require authentication for all requests + auth_basic "Loki Authentication"; + auth_basic_user_file /etc/nginx/.htpasswd; + + # Proxy all requests to Loki + location / { + proxy_pass http://loki; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + # Important for Loki push endpoint + client_max_body_size 100M; + client_body_buffer_size 10M; + } + } +} \ No newline at end of file diff --git a/logserver/config/service.env b/logserver/config/service.env index 0049585..c481a28 100644 --- a/logserver/config/service.env +++ b/logserver/config/service.env @@ -1,46 +1,20 @@ -# Service identification +# Log Server Configuration (Loki + Grafana) CONTAINER_NAME=logserver # Server settings (REQUIRED by dropshell) SSH_USER="root" -# Elasticsearch settings -ES_VERSION=7.17.23 -ES_HEAP_SIZE=2g -ES_MAX_MAP_COUNT=262144 - -# Logstash settings -LS_VERSION=7.17.23 -LS_HEAP_SIZE=1g -LS_PIPELINE_WORKERS=2 - -# Kibana settings -KIBANA_VERSION=7.17.23 - -# Authentication (IMPORTANT: Change these!) -ELASTIC_PASSWORD=changeme # Password for 'elastic' superuser (internal use) -KIBANA_USERNAME=admin # Your login username for Kibana -KIBANA_USER_PASSWORD=changeme # Your login password for Kibana - # Ports -KIBANA_PORT=5601 -LOGSTASH_BEATS_PORT=5044 -LOGSTASH_SYSLOG_PORT=514 +WEB_PORT=3000 # Grafana web UI +LOKI_PORT=3100 # Loki API (for clients to send logs) -# Server configuration -SERVER_PUBLICBASEURL=http://localhost:5601 # Change to your server's actual URL +# Authentication for Grafana +ADMIN_USER=admin +ADMIN_PASSWORD=changeme -# Log retention -LOG_RETENTION_DAYS=30 -LOG_MAX_SIZE_GB=50 +# Authentication for Loki (clients must use these) +LOKI_USER=logclient +LOKI_PASSWORD=changeme -# Authentication -ENABLE_TLS=true -API_KEYS_FILE=${CONFIG_PATH}/api-keys.yml - -# Network Security -ALLOWED_IPS="" # Comma-separated list, empty = all - -# Resource limits -MAX_CPU_PERCENT=80 -MAX_MEMORY=4GB \ No newline at end of file +# Log retention (days) +LOG_RETENTION=7 \ No newline at end of file diff --git a/logserver/config/validate_api_key.rb b/logserver/config/validate_api_key.rb deleted file mode 100644 index 4d77ed3..0000000 --- a/logserver/config/validate_api_key.rb +++ /dev/null @@ -1,43 +0,0 @@ -# Ruby script for Logstash to validate API keys -# This is a simplified validation - in production, use proper authentication - -require 'yaml' - -def register(params) - @api_keys_file = params["api_keys_file"] -end - -def filter(event) - # Get the API key from the event - api_key = event.get("[api_key]") || event.get("[@metadata][api_key]") - - # If no API key, pass through (for backwards compatibility) - # In production, you should reject events without valid keys - if api_key.nil? || api_key.empty? - # For now, allow events without API keys - # event.cancel # Uncomment to require API keys - return [event] - end - - # Load API keys from file - begin - if File.exist?(@api_keys_file) - config = YAML.load_file(@api_keys_file) - valid_keys = config['api_keys'].values if config && config['api_keys'] - - # Check if the provided key is valid - if valid_keys && valid_keys.include?(api_key) - # Valid key - let the event through - event.set("[@metadata][authenticated]", true) - else - # Invalid key - drop the event - event.cancel - end - end - rescue => e - # Log error but don't crash - event.set("[@metadata][auth_error]", e.message) - end - - return [event] -end \ No newline at end of file diff --git a/logserver/destroy.sh b/logserver/destroy.sh deleted file mode 100755 index 0906253..0000000 --- a/logserver/destroy.sh +++ /dev/null @@ -1,26 +0,0 @@ -#!/bin/bash -source "${AGENT_PATH}/common.sh" -source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh" - -_check_required_env_vars "CONTAINER_NAME" - -echo "WARNING: This will PERMANENTLY DELETE all ELK stack data!" -echo "This includes all logs, dashboards, and configurations" -echo "This action cannot be undone!" -echo "" - -# Since scripts must be non-interactive, only proceed if explicitly called -# Stop all containers -docker compose down || true - -# Remove all volumes -for volume in $(get_logserver_volumes); do - volume_name=$(echo $volume | cut -d: -f3) - echo "Removing volume: $volume_name" - docker volume rm "$volume_name" 2>/dev/null || true -done - -# Also remove the Docker network if it exists -docker network rm "${CONTAINER_NAME}_elk" 2>/dev/null || true - -echo "ELK stack and all data destroyed" \ No newline at end of file diff --git a/logserver/docker-compose.yml b/logserver/docker-compose.yml index 61ffe62..09efaac 100644 --- a/logserver/docker-compose.yml +++ b/logserver/docker-compose.yml @@ -1,80 +1,57 @@ +version: '3.8' + services: - elasticsearch: - image: docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION:-7.17.23} - container_name: ${CONTAINER_NAME}_elasticsearch + # Grafana - Simple Web UI + grafana: + image: grafana/grafana:latest + container_name: ${CONTAINER_NAME}_grafana environment: - - discovery.type=single-node - - "ES_JAVA_OPTS=-Xms${ES_HEAP_SIZE:-2g} -Xmx${ES_HEAP_SIZE:-2g}" - - xpack.security.enabled=true - - xpack.security.authc.api_key.enabled=true - - ELASTIC_PASSWORD=${ELASTIC_PASSWORD:-${KIBANA_PASSWORD:-changeme}} - - xpack.monitoring.enabled=false - - cluster.routing.allocation.disk.threshold_enabled=false + - GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin} + - GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-changeme} + - GF_USERS_ALLOW_SIGN_UP=false volumes: - - elasticsearch_data:/usr/share/elasticsearch/data + - grafana_data:/var/lib/grafana + - ${CONFIG_PATH}/grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yaml:ro + - ${CONFIG_PATH}/grafana-dashboard.yml:/etc/grafana/provisioning/dashboards/dashboard.yaml:ro + - ${CONFIG_PATH}/dashboards:/var/lib/grafana/dashboards:ro ports: - - "127.0.0.1:9200:9200" - networks: - - elk + - "${WEB_PORT:-3000}:3000" restart: unless-stopped - ulimits: - memlock: - soft: -1 - hard: -1 - nofile: - soft: 65536 - hard: 65536 - - logstash: - image: docker.elastic.co/logstash/logstash:${LS_VERSION:-7.17.23} - container_name: ${CONTAINER_NAME}_logstash - environment: - - "LS_JAVA_OPTS=-Xms${LS_HEAP_SIZE:-1g} -Xmx${LS_HEAP_SIZE:-1g}" - - "xpack.monitoring.enabled=false" - - ELASTIC_PASSWORD=${ELASTIC_PASSWORD:-${KIBANA_PASSWORD:-changeme}} - - CONTAINER_NAME=${CONTAINER_NAME} - command: logstash -f /usr/share/logstash/config/logstash.conf - volumes: - - ${CONFIG_PATH}:/usr/share/logstash/config - - logstash_data:/usr/share/logstash/data - ports: - - "${LOGSTASH_BEATS_PORT:-5044}:5044" - - "${LOGSTASH_SYSLOG_PORT:-514}:514/tcp" - - "${LOGSTASH_SYSLOG_PORT:-514}:514/udp" - networks: - - elk depends_on: - - elasticsearch - restart: unless-stopped - - kibana: - image: docker.elastic.co/kibana/kibana:${KIBANA_VERSION:-7.17.23} - container_name: ${CONTAINER_NAME}_kibana - environment: - - ELASTICSEARCH_HOSTS=http://elasticsearch:9200 - - ELASTICSEARCH_USERNAME=elastic - - ELASTICSEARCH_PASSWORD=${ELASTIC_PASSWORD:-${KIBANA_PASSWORD:-changeme}} - - XPACK_SECURITY_ENABLED=true - - NODE_OPTIONS=--openssl-legacy-provider - - SERVER_PUBLICBASEURL=${SERVER_PUBLICBASEURL:-http://localhost:5601} - volumes: - - kibana_data:/usr/share/kibana/data - ports: - - "${KIBANA_PORT:-5601}:5601" + - loki networks: - - elk - depends_on: - - elasticsearch + - loki-net + + # Loki - Log storage (not exposed directly) + loki: + image: grafana/loki:latest + container_name: ${CONTAINER_NAME}_loki + volumes: + - loki_data:/loki + - ${CONFIG_PATH}/loki.yaml:/etc/loki/local-config.yaml:ro + command: -config.file=/etc/loki/local-config.yaml restart: unless-stopped + networks: + - loki-net + + # Nginx - Authentication proxy for Loki + nginx: + image: nginx:alpine + container_name: ${CONTAINER_NAME}_nginx + volumes: + - ${CONFIG_PATH}/nginx.conf:/etc/nginx/nginx.conf:ro + - ${CONFIG_PATH}/.htpasswd:/etc/nginx/.htpasswd:ro + ports: + - "${LOKI_PORT:-3100}:80" + restart: unless-stopped + depends_on: + - loki + networks: + - loki-net networks: - elk: - driver: bridge + loki-net: volumes: - elasticsearch_data: - name: ${CONTAINER_NAME}_elasticsearch_data - logstash_data: - name: ${CONTAINER_NAME}_logstash_data - kibana_data: - name: ${CONTAINER_NAME}_kibana_data \ No newline at end of file + grafana_data: + loki_data: \ No newline at end of file diff --git a/logserver/generate-api-key.sh b/logserver/generate-api-key.sh deleted file mode 100755 index 6b8983f..0000000 --- a/logserver/generate-api-key.sh +++ /dev/null @@ -1,172 +0,0 @@ -#!/bin/bash - -# Interactive API Key Generation Script for LogServer -# This script generates secure API keys and adds them to api-keys.yml - -# Determine where to put the api-keys.yml file -determine_api_keys_location() { - # 1. If api-keys.yml already exists in current folder, use it - if [ -f "./api-keys.yml" ]; then - echo "./api-keys.yml" - return 0 - fi - - # 2. If service.env exists in current folder, put keys here - if [ -f "./service.env" ]; then - echo "./api-keys.yml" - return 0 - fi - - # 3. If config folder exists, put keys there - if [ -d "./config" ]; then - echo "./config/api-keys.yml" - return 0 - fi - - # No valid location found - return 1 -} - -# Try to determine location -if API_KEYS_FILE=$(determine_api_keys_location); then - : # Location found, continue -else - echo -e "${RED}Error: Cannot determine where to place api-keys.yml${NC}" - echo "" - echo "This script must be run from one of these locations:" - echo " 1. A deployed service directory (contains service.env)" - echo " 2. The logserver template directory (contains config/ folder)" - echo " 3. A directory with existing api-keys.yml file" - echo "" - echo "Current directory: $(pwd)" - echo "Contents: $(ls -la 2>/dev/null | head -5)" - exit 1 -fi - -# Colors for output -RED='\033[0;31m' -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -NC='\033[0m' # No Color - -# Generate a secure random API key -generate_key() { - openssl rand -hex 32 2>/dev/null || cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1 -} - -# Initialize api-keys.yml if it doesn't exist -init_api_keys_file() { - if [ ! -f "$API_KEYS_FILE" ]; then - # Create directory if needed - local dir=$(dirname "$API_KEYS_FILE") - if [ ! -d "$dir" ]; then - mkdir -p "$dir" - echo -e "${GREEN}Created directory: $dir${NC}" - fi - - echo "# API Keys for LogServer Authentication" > "$API_KEYS_FILE" - echo "# Format: hostname:api_key" >> "$API_KEYS_FILE" - echo "# Generated by generate-api-key.sh" >> "$API_KEYS_FILE" - echo "" >> "$API_KEYS_FILE" - echo "api_keys:" >> "$API_KEYS_FILE" - echo -e "${GREEN}Created new api-keys.yml file at: $API_KEYS_FILE${NC}" - else - echo -e "${GREEN}Using existing api-keys.yml at: $API_KEYS_FILE${NC}" - fi -} - -# Check if hostname already has a key -check_existing_key() { - local hostname=$1 - if grep -q "^ ${hostname}:" "$API_KEYS_FILE" 2>/dev/null; then - return 0 - fi - return 1 -} - -# Add key to api-keys.yml -add_key_to_file() { - local hostname=$1 - local api_key=$2 - echo " ${hostname}: ${api_key}" >> "$API_KEYS_FILE" -} - -# Main script -echo -e "${GREEN}=== LogServer API Key Generator ===${NC}" -echo "" - -# Initialize file if needed -init_api_keys_file - -# Interactive mode -while true; do - echo -e "${YELLOW}Enter hostname for the client (or 'done' to finish):${NC}" - read -p "> " hostname - - if [ "$hostname" = "done" ] || [ -z "$hostname" ]; then - break - fi - - # Validate hostname - allow simple names, must start with alphanumeric - if [[ ! "$hostname" =~ ^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$ ]]; then - echo -e "${RED}Invalid hostname format. Use only letters, numbers, dots, dashes, and underscores.${NC}" - echo -e "${RED}Hostname must start and end with a letter or number.${NC}" - continue - fi - - # Check if key already exists - if check_existing_key "$hostname"; then - echo -e "${YELLOW}Key already exists for ${hostname}${NC}" - read -p "Generate new key? (y/n): " overwrite - if [ "$overwrite" != "y" ]; then - continue - fi - # Remove old key - sed -i "/^ ${hostname}:/d" "$API_KEYS_FILE" - fi - - # Generate new key - api_key=$(generate_key) - - # Add to file - add_key_to_file "$hostname" "$api_key" - - echo -e "${GREEN}✓ Generated API key for ${hostname}${NC}" - echo "" - echo "Configuration for ${hostname}:" - echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" - echo "Add to client's service.env:" - echo "" - echo "LOGSERVER_HOST=$(hostname -I | awk '{print $1}')" - echo "LOGSERVER_PORT=5044" - echo "API_KEY=${api_key}" - echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" - echo "" - - # Option to add more - read -p "Add another client? (y/n): " add_more - if [ "$add_more" != "y" ]; then - break - fi -done - -# Show summary -echo "" -echo -e "${GREEN}=== Summary ===${NC}" -echo "API keys file: $API_KEYS_FILE" -echo "Total clients configured: $(grep -c "^ " "$API_KEYS_FILE" 2>/dev/null || echo 0)" -echo "" -echo "To view all keys: cat $API_KEYS_FILE" -echo "To revoke a key: Edit $API_KEYS_FILE and remove the line" -echo "" - -# Show location-specific restart instructions -if [[ "$API_KEYS_FILE" == "./api-keys.yml" ]] && [ -f "./service.env" ]; then - # We're in a deployed service directory - echo -e "${YELLOW}Remember to restart the service to apply changes:${NC}" - echo " dropshell restart logserver" -else - # We're in the template directory - echo -e "${YELLOW}Note: Deploy this template to use these keys:${NC}" - echo " dropshell install logserver" -fi \ No newline at end of file diff --git a/logserver/install.sh b/logserver/install.sh index 6806c06..d21bc88 100755 --- a/logserver/install.sh +++ b/logserver/install.sh @@ -2,128 +2,48 @@ source "${AGENT_PATH}/common.sh" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -# Check required environment variables -_check_required_env_vars "CONTAINER_NAME" "ES_VERSION" "LS_VERSION" "KIBANA_VERSION" +_check_required_env_vars "CONTAINER_NAME" -# Check Docker and Docker Compose are available +# Check Docker _check_docker_installed || _die "Docker test failed" -docker compose version >/dev/null 2>&1 || _die "Docker Compose is not installed (requires Docker Compose V2)" - -# Check vm.max_map_count for Elasticsearch -current_max_map_count=$(sysctl -n vm.max_map_count 2>/dev/null || echo 0) -if [ "$current_max_map_count" -lt 262144 ]; then - echo "WARNING: vm.max_map_count is too low ($current_max_map_count)" - echo "Elasticsearch requires at least 262144" - echo "Please run: sudo sysctl -w vm.max_map_count=262144" - echo "And add to /etc/sysctl.conf to persist" - _die "System configuration needs adjustment" -fi - -# Check available memory -available_mem=$(free -m | awk '/^Mem:/{print $7}') -if [ "$available_mem" -lt 3000 ]; then - echo "WARNING: Low available memory (${available_mem}MB)" - echo "ELK stack requires at least 3-4GB free memory for proper operation" - echo "Services may take longer to start or fail to start" - echo "" -fi +docker compose version >/dev/null 2>&1 || _die "Docker Compose V2 is required" # Stop any existing containers -bash ./stop.sh || true +bash ./stop.sh 2>/dev/null || true -# Remove old containers -docker compose down --remove-orphans 2>/dev/null || true +# Create config directory +mkdir -p "${CONFIG_PATH}/dashboards" -# Pull the Docker images -echo "Pulling ELK stack images..." -docker pull docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION} || _die "Failed to pull Elasticsearch" -docker pull docker.elastic.co/logstash/logstash:${LS_VERSION} || _die "Failed to pull Logstash" -docker pull docker.elastic.co/kibana/kibana:${KIBANA_VERSION} || _die "Failed to pull Kibana" +# Copy configuration files +cp "$SCRIPT_DIR/config/"*.yaml "$SCRIPT_DIR/config/"*.yml "$SCRIPT_DIR/config/"*.conf "${CONFIG_PATH}/" 2>/dev/null || true +cp "$SCRIPT_DIR/config/dashboards/"*.json "${CONFIG_PATH}/dashboards/" 2>/dev/null || true -# Create volumes using common function -source "$SCRIPT_DIR/_volumes.sh" -echo "Creating volumes..." -create_items $(get_logserver_volumes) - -# Ensure config directory exists -mkdir -p "${CONFIG_PATH}" - -# Initialize API keys file if it doesn't exist -if [ ! -f "${CONFIG_PATH}/api-keys.yml" ]; then - echo "No API keys configured yet." - echo "Run ./generate-api-key.sh to add client keys" - echo "api_keys:" > "${CONFIG_PATH}/api-keys.yml" +# Generate htpasswd file for Loki authentication +echo "Generating authentication file..." +# Use openssl to generate htpasswd (available on most systems) +if command -v openssl >/dev/null 2>&1; then + # Generate password hash + PASS_HASH=$(openssl passwd -apr1 "${LOKI_PASSWORD:-changeme}") + echo "${LOKI_USER:-logclient}:$PASS_HASH" > "${CONFIG_PATH}/.htpasswd" +elif command -v htpasswd >/dev/null 2>&1; then + # Use htpasswd if available + htpasswd -cb "${CONFIG_PATH}/.htpasswd" "${LOKI_USER:-logclient}" "${LOKI_PASSWORD:-changeme}" +else + echo "WARNING: Cannot generate password file - no openssl or htpasswd found" + echo "Using basic auth with plain text (NOT SECURE):" + echo "${LOKI_USER:-logclient}:${LOKI_PASSWORD:-changeme}" > "${CONFIG_PATH}/.htpasswd" fi -# Copy Logstash configurations if they don't exist -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# Start the stack +echo "Starting Log Server..." +docker compose up -d || _die "Failed to start" -if [ ! -f "${CONFIG_PATH}/logstash.conf" ]; then - if [ -f "$SCRIPT_DIR/config/logstash.conf" ]; then - cp "$SCRIPT_DIR/config/logstash.conf" "${CONFIG_PATH}/logstash.conf" - echo "Copied Logstash pipeline configuration to ${CONFIG_PATH}" - else - echo "WARNING: logstash.conf not found in template" - fi -fi - -if [ ! -f "${CONFIG_PATH}/logstash.yml" ]; then - if [ -f "$SCRIPT_DIR/config/logstash.yml" ]; then - cp "$SCRIPT_DIR/config/logstash.yml" "${CONFIG_PATH}/logstash.yml" - echo "Copied Logstash settings to ${CONFIG_PATH}" - else - echo "WARNING: logstash.yml not found in template" - fi -fi - -# Start the ELK stack -echo "Starting ELK stack..." -docker compose up -d --build || _die "Failed to start ELK stack" - -# Wait for services to be ready with polling -echo "Waiting for services to start (this can take 2-3 minutes on first run)..." -MAX_WAIT=240 # Maximum 4 minutes +# Wait for services +echo -n "Waiting for services to start..." +MAX_WAIT=60 WAITED=0 while [ $WAITED -lt $MAX_WAIT ]; do - # Check if all services are running - if bash ./status.sh 2>/dev/null | grep -q "Running"; then - echo " All services are up!" - break - fi - - # Show progress with time elapsed - if [ $((WAITED % 10)) -eq 0 ] && [ $WAITED -gt 0 ]; then - echo -n " [${WAITED}s]" - else - echo -n "." - fi - - sleep 2 - WAITED=$((WAITED + 2)) -done -echo "" - -if [ $WAITED -ge $MAX_WAIT ]; then - echo "Warning: Services are still starting. This is normal on first run." - echo "Current status:" - bash ./status.sh || true - echo "" - echo "You can check the logs with:" - echo " docker logs ${CONTAINER_NAME}_elasticsearch" - echo " docker logs ${CONTAINER_NAME}_logstash" - echo " docker logs ${CONTAINER_NAME}_kibana" - echo "" - echo "The services will continue starting in the background." -fi - -# Create custom user -echo "Setting up custom user '${KIBANA_USERNAME:-admin}'..." -echo -n "Waiting for Elasticsearch API..." - -# First wait for Elasticsearch to be ready -WAITED=0 -while [ $WAITED -lt 60 ]; do - if docker exec ${CONTAINER_NAME}_elasticsearch curl -s -u elastic:${ELASTIC_PASSWORD} http://localhost:9200/_cluster/health 2>/dev/null | grep -q '"status":"yellow"\|"status":"green"'; then + if curl -s "http://localhost:${WEB_PORT:-3000}/api/health" 2>/dev/null | grep -q "ok"; then echo " Ready!" break fi @@ -131,56 +51,28 @@ while [ $WAITED -lt 60 ]; do sleep 2 WAITED=$((WAITED + 2)) done +echo "" -if [ $WAITED -lt 60 ]; then - # Now create the user - docker exec ${CONTAINER_NAME}_elasticsearch bash -c " - - result=\$(curl -s -X POST -u elastic:${ELASTIC_PASSWORD} \ - -H 'Content-Type: application/json' \ - http://localhost:9200/_security/user/${KIBANA_USERNAME:-admin} \ - -d '{ - \"password\" : \"${KIBANA_USER_PASSWORD:-changeme}\", - \"roles\" : [ \"superuser\" ], - \"full_name\" : \"Admin User\" - }' 2>/dev/null) - - if echo \"\$result\" | grep -q '\"created\":true'; then - echo \"User created successfully\" - else - echo \"User already exists (this is fine)\" - fi - " -else - echo "Warning: Elasticsearch API not ready after 60 seconds" -fi - -echo "Installation of ${CONTAINER_NAME} complete" echo "" echo "=========================================" -echo "Kibana UI: ${SERVER_PUBLICBASEURL:-http://$(hostname -I | awk '{print $1}'):${KIBANA_PORT}}" -echo "" -echo "Login with your custom user:" -echo " Username: ${KIBANA_USERNAME:-admin}" -echo " Password: ${KIBANA_USER_PASSWORD:-changeme}" -echo "" -echo "Or the superuser:" -echo " Username: elastic" -echo " Password: ${ELASTIC_PASSWORD:-changeme}" +echo "Log Server Installed!" echo "=========================================" echo "" -# Only show reminder if using default values -if [ "${ELASTIC_PASSWORD}" = "changeme" ] || [ -z "${SERVER_PUBLICBASEURL}" ] || [ "${SERVER_PUBLICBASEURL}" = "http://localhost:5601" ]; then - echo "REMINDER: Update service.env with:" - if [ "${ELASTIC_PASSWORD}" = "changeme" ]; then - echo " - A secure password in ELASTIC_PASSWORD" - fi - if [ -z "${SERVER_PUBLICBASEURL}" ] || [ "${SERVER_PUBLICBASEURL}" = "http://localhost:5601" ]; then - echo " - Your actual server IP/domain in SERVER_PUBLICBASEURL" - fi - echo "" -fi -echo "Logstash listening on port ${LOGSTASH_BEATS_PORT} for Filebeat clients" +echo "Web UI: http://$(hostname -I | awk '{print $1}'):${WEB_PORT:-3000}" +echo "Login: ${ADMIN_USER:-admin} / ${ADMIN_PASSWORD:-changeme}" echo "" -echo "To add client authentication:" -echo " ./generate-api-key.sh" \ No newline at end of file +echo "TO VIEW LOGS:" +echo "1. Click 'Dashboards' (4 squares icon)" +echo "2. Click 'Central Logs'" +echo "3. See all logs from all servers!" +echo "" +echo "FOR CLIENTS TO SEND LOGS HERE:" +echo "Server: $(hostname -I | awk '{print $1}')" +echo "Port: ${LOKI_PORT:-3100}" +echo "Username: ${LOKI_USER:-logclient}" +echo "Password: ${LOKI_PASSWORD:-changeme}" +echo "" +echo "IMPORTANT: Change LOKI_PASSWORD in service.env!" +echo "" +echo "Install 'logclient' on other servers" +echo "=========================================" \ No newline at end of file diff --git a/logserver/restore.sh b/logserver/restore.sh deleted file mode 100755 index b734068..0000000 --- a/logserver/restore.sh +++ /dev/null @@ -1,56 +0,0 @@ -#!/bin/bash -source "${AGENT_PATH}/common.sh" -source "$(dirname "${BASH_SOURCE[0]}")/_volumes.sh" - -_check_required_env_vars "CONTAINER_NAME" - -if [ -z "$1" ]; then - echo "Usage: $0 " - echo "Available backups:" - ls -la "${CONFIG_PATH}/backups/" 2>/dev/null || echo "No backups found" - exit 1 -fi - -BACKUP_DIR="$1" - -if [ ! -d "$BACKUP_DIR" ]; then - _die "Backup directory not found: $BACKUP_DIR" -fi - -echo "Restoring ELK stack from backup: $BACKUP_DIR" -echo "WARNING: This will overwrite all current data!" - -# Stop the containers -bash ./stop.sh || true - -# Restore volumes -for volume in $(get_logserver_volumes); do - volume_name=$(echo $volume | cut -d: -f3) - backup_file="$BACKUP_DIR/${volume_name}.tar.gz" - - if [ -f "$backup_file" ]; then - echo "Restoring volume: $volume_name" - # Clear existing data and restore - docker run --rm -v "$volume_name:/target" -v "$BACKUP_DIR:/backup:ro" alpine \ - sh -c "rm -rf /target/* && tar -xzf /backup/${volume_name}.tar.gz -C /target" - else - echo "Warning: Backup file not found for $volume_name" - fi -done - -# Restore configuration if exists -if [ -d "$BACKUP_DIR/config_backup" ]; then - cp -r "$BACKUP_DIR/config_backup/"* "${CONFIG_PATH}/" -fi - -# Restore docker-compose.yml if exists -if [ -f "$BACKUP_DIR/docker-compose.yml" ]; then - cp "$BACKUP_DIR/docker-compose.yml" . -fi - -echo "Restore completed" - -# Start the containers -bash ./start.sh - -echo "ELK stack restarted with restored data" \ No newline at end of file diff --git a/logserver/scripts/setup-user.sh b/logserver/scripts/setup-user.sh deleted file mode 100755 index eb709b7..0000000 --- a/logserver/scripts/setup-user.sh +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/bash - -# Script to create a custom user for Kibana login -# This runs after Elasticsearch is up and creates the user defined in service.env - -# Wait for Elasticsearch to be ready -echo "Waiting for Elasticsearch to be ready..." -until curl -s -u elastic:${ELASTIC_PASSWORD} http://localhost:9200/_cluster/health | grep -q '"status":"yellow"\|"status":"green"'; do - sleep 5 -done - -echo "Creating user '${KIBANA_USERNAME}'..." - -# Create the custom user with superuser role -curl -X POST -u elastic:${ELASTIC_PASSWORD} \ - -H "Content-Type: application/json" \ - http://localhost:9200/_security/user/${KIBANA_USERNAME} \ - -d '{ - "password" : "'${KIBANA_USER_PASSWORD}'", - "roles" : [ "superuser" ], - "full_name" : "Admin User", - "email" : "admin@example.com" - }' - -if [ $? -eq 0 ]; then - echo "" - echo "User '${KIBANA_USERNAME}' created successfully!" - echo "You can now log in to Kibana with:" - echo " Username: ${KIBANA_USERNAME}" - echo " Password: ${KIBANA_USER_PASSWORD}" -else - echo "Note: User might already exist or there was an error" -fi \ No newline at end of file diff --git a/logserver/setup-kibana.sh b/logserver/setup-kibana.sh deleted file mode 100755 index 8e83f9b..0000000 --- a/logserver/setup-kibana.sh +++ /dev/null @@ -1,108 +0,0 @@ -#!/bin/bash - -# Setup Kibana for simple log viewing -# This creates index patterns and saved searches for easy log access - -source "${AGENT_PATH}/common.sh" -_check_required_env_vars "CONTAINER_NAME" "ELASTIC_PASSWORD" "KIBANA_PORT" - -KIBANA_URL="http://localhost:${KIBANA_PORT}" -AUTH="elastic:${ELASTIC_PASSWORD}" - -echo "Setting up Kibana for simple log viewing..." -echo "" - -# Wait for Kibana to be ready -echo -n "Waiting for Kibana to be ready..." -MAX_WAIT=60 -WAITED=0 -while [ $WAITED -lt $MAX_WAIT ]; do - if docker exec ${CONTAINER_NAME}_kibana curl -s -u "$AUTH" "${KIBANA_URL}/api/status" 2>/dev/null | grep -q '"level":"available"'; then - echo " Ready!" - break - fi - echo -n "." - sleep 2 - WAITED=$((WAITED + 2)) -done - -if [ $WAITED -ge $MAX_WAIT ]; then - echo "" - echo "ERROR: Kibana is not ready after ${MAX_WAIT} seconds" - exit 1 -fi - -# Create index pattern for Filebeat -echo "Creating Filebeat index pattern..." -docker exec ${CONTAINER_NAME}_kibana curl -s -X POST \ - -u "$AUTH" \ - -H "Content-Type: application/json" \ - -H "kbn-xsrf: true" \ - "${KIBANA_URL}/api/saved_objects/index-pattern/filebeat-*" \ - -d '{ - "attributes": { - "title": "filebeat-*", - "timeFieldName": "@timestamp", - "fields": "[]" - } - }' > /dev/null 2>&1 - -# Set as default index pattern -docker exec ${CONTAINER_NAME}_kibana curl -s -X POST \ - -u "$AUTH" \ - -H "Content-Type: application/json" \ - -H "kbn-xsrf: true" \ - "${KIBANA_URL}/api/kibana/settings" \ - -d '{ - "changes": { - "defaultIndex": "filebeat-*" - } - }' > /dev/null 2>&1 - -# Create a simple saved search for Docker logs -echo "Creating saved searches..." -docker exec ${CONTAINER_NAME}_kibana curl -s -X POST \ - -u "$AUTH" \ - -H "Content-Type: application/json" \ - -H "kbn-xsrf: true" \ - "${KIBANA_URL}/api/saved_objects/search" \ - -d '{ - "attributes": { - "title": "Docker Container Logs", - "description": "View all Docker container logs", - "columns": ["container_name", "message"], - "sort": ["@timestamp", "desc"], - "kibanaSavedObjectMeta": { - "searchSourceJSON": "{\"index\":\"filebeat-*\",\"query\":{\"match_all\":{}},\"filter\":[{\"meta\":{\"index\":\"filebeat-*\",\"negate\":false,\"disabled\":false,\"alias\":null,\"type\":\"exists\",\"key\":\"container_name\",\"value\":\"exists\"},\"exists\":{\"field\":\"container_name\"}}]}" - } - } - }' > /dev/null 2>&1 - -echo "" -echo "=========================================" -echo "Kibana Setup Complete!" -echo "=========================================" -echo "" -echo "QUICK START GUIDE:" -echo "" -echo "1. Open Kibana: ${SERVER_PUBLICBASEURL:-http://$(hostname -I | awk '{print $1}'):${KIBANA_PORT}}" -echo "" -echo "2. Login with:" -echo " Username: ${KIBANA_USERNAME:-elastic}" -echo " Password: [your password]" -echo "" -echo "3. TO VIEW LOGS SIMPLY:" -echo " a) Click 'Discover' in the left menu" -echo " b) Time range is in top-right (set to 'Last 15 minutes' or 'Today')" -echo " c) Your logs will appear below" -echo "" -echo "4. TO FILTER LOGS:" -echo " - By container: Click '+' next to any 'container_name' value" -echo " - By host: Click '+' next to any 'host.name' value" -echo " - Search box: Type keywords to search all logs" -echo "" -echo "5. TO VIEW LIVE LOGS:" -echo " - Click the 'Refresh' button in top-right" -echo " - Set it to refresh every 5 seconds" -echo "" -echo "=========================================" \ No newline at end of file diff --git a/logserver/start.sh b/logserver/start.sh index 43a8363..148d230 100755 --- a/logserver/start.sh +++ b/logserver/start.sh @@ -2,16 +2,5 @@ source "${AGENT_PATH}/common.sh" _check_required_env_vars "CONTAINER_NAME" -echo "Starting ELK stack..." -docker compose up -d || _die "Failed to start ELK stack" - -# Wait for services to be ready -echo "Waiting for services to start..." -sleep 5 - -# Check if services are running -if docker compose ps | grep -q "Up"; then - echo "ELK stack started successfully" -else - _die "Failed to start ELK stack services" -fi \ No newline at end of file +docker compose up -d || _die "Failed to start" +echo "Log Server started" \ No newline at end of file diff --git a/logserver/status.sh b/logserver/status.sh index 180a913..170ef72 100755 --- a/logserver/status.sh +++ b/logserver/status.sh @@ -2,38 +2,11 @@ source "${AGENT_PATH}/common.sh" _check_required_env_vars "CONTAINER_NAME" -# For verbose mode (when called directly, not from install script) -if [ "$1" = "-v" ] || [ "$1" = "--verbose" ]; then - VERBOSE=true -else - VERBOSE=false -fi - -# Check if containers exist -if ! docker ps -a --format "{{.Names}}" | grep -q "^${CONTAINER_NAME}_"; then - echo "Unknown" - exit 0 -fi - -# Check individual service status using docker directly (more reliable) -es_running=$(docker ps --format "{{.Names}}" | grep -c "^${CONTAINER_NAME}_elasticsearch$" || echo 0) -ls_running=$(docker ps --format "{{.Names}}" | grep -c "^${CONTAINER_NAME}_logstash$" || echo 0) -kb_running=$(docker ps --format "{{.Names}}" | grep -c "^${CONTAINER_NAME}_kibana$" || echo 0) - -if [ "$VERBOSE" = "true" ]; then - echo "Elasticsearch: $([ $es_running -eq 1 ] && echo "Running" || echo "Not running")" - echo "Logstash: $([ $ls_running -eq 1 ] && echo "Running" || echo "Not running")" - echo "Kibana: $([ $kb_running -eq 1 ] && echo "Running" || echo "Not running")" -fi - -# Return overall status -if [ "$es_running" -eq 1 ] && [ "$ls_running" -eq 1 ] && [ "$kb_running" -eq 1 ]; then +if docker ps | grep -q "${CONTAINER_NAME}_grafana"; then echo "Running" -elif [ "$es_running" -eq 0 ] && [ "$ls_running" -eq 0 ] && [ "$kb_running" -eq 0 ]; then - echo "Stopped" + echo " Web UI: http://$(hostname -I | awk '{print $1}'):${WEB_PORT:-3000}" + echo " Loki API: http://$(hostname -I | awk '{print $1}'):${LOKI_PORT:-3100}" + docker ps --format "table {{.Names}}\t{{.Status}}" | grep "${CONTAINER_NAME}_" else - # At least one service is having issues - if [ "$VERBOSE" = "false" ]; then - echo "Starting" # More accurate than "Error" during startup - fi + echo "Stopped" fi \ No newline at end of file diff --git a/logserver/stop.sh b/logserver/stop.sh index b65613c..b9d081d 100755 --- a/logserver/stop.sh +++ b/logserver/stop.sh @@ -2,7 +2,5 @@ source "${AGENT_PATH}/common.sh" _check_required_env_vars "CONTAINER_NAME" -echo "Stopping ELK stack..." -docker compose stop || true - -echo "ELK stack stopped" \ No newline at end of file +docker compose stop || _die "Failed to stop" +echo "Log Server stopped" \ No newline at end of file diff --git a/logserver/test-auth.sh b/logserver/test-auth.sh deleted file mode 100755 index f505fb1..0000000 --- a/logserver/test-auth.sh +++ /dev/null @@ -1,54 +0,0 @@ -#!/bin/bash - -# Test script to verify API key authentication is working -# This sends test log messages with and without API keys to verify authentication - -source "${AGENT_PATH}/common.sh" -_check_required_env_vars "CONTAINER_NAME" "LOGSTASH_BEATS_PORT" - -echo "=== Testing Logstash API Key Authentication ===" -echo "" - -# Check if service is running -if ! bash ./status.sh | grep -q "Running"; then - echo "Error: LogServer is not running. Please start it first." - exit 1 -fi - -# Get server IP -SERVER_IP=$(hostname -I | awk '{print $1}') - -echo "Testing authentication on ${SERVER_IP}:${LOGSTASH_BEATS_PORT}" -echo "" - -# Test 1: Send without API key (should be tagged or dropped based on config) -echo "Test 1: Sending log without API key..." -echo '{"message":"Test log without API key","timestamp":"'$(date -Iseconds)'"}' | \ - nc -w 2 ${SERVER_IP} ${LOGSTASH_BEATS_PORT} 2>/dev/null || true - -# Test 2: Send with invalid API key (should be dropped) -echo "Test 2: Sending log with invalid API key..." -echo '{"message":"Test log with invalid key","fields":{"api_key":"invalid-key-12345"},"timestamp":"'$(date -Iseconds)'"}' | \ - nc -w 2 ${SERVER_IP} ${LOGSTASH_BEATS_PORT} 2>/dev/null || true - -# Test 3: Send with valid API key (if one exists) -if [ -f "${CONFIG_PATH}/api-keys.yml" ]; then - # Extract first API key from the file - API_KEY=$(grep -E "^ [^:]+: " "${CONFIG_PATH}/api-keys.yml" | head -1 | awk '{print $2}') - if [ -n "$API_KEY" ]; then - echo "Test 3: Sending log with valid API key..." - echo '{"message":"Test log with valid key","fields":{"api_key":"'${API_KEY}'"},"timestamp":"'$(date -Iseconds)'"}' | \ - nc -w 2 ${SERVER_IP} ${LOGSTASH_BEATS_PORT} 2>/dev/null || true - else - echo "Test 3: Skipped - no API keys found" - fi -else - echo "Test 3: Skipped - api-keys.yml not found" -fi - -echo "" -echo "Tests sent. Check Logstash logs to verify authentication:" -echo " docker logs ${CONTAINER_NAME}_logstash --tail 20" -echo "" -echo "Valid logs should appear in Elasticsearch." -echo "Invalid logs should be dropped." \ No newline at end of file diff --git a/logserver/uninstall.sh b/logserver/uninstall.sh index 3a46ee5..3868521 100755 --- a/logserver/uninstall.sh +++ b/logserver/uninstall.sh @@ -2,15 +2,5 @@ source "${AGENT_PATH}/common.sh" _check_required_env_vars "CONTAINER_NAME" -# Stop the containers -bash ./stop.sh || _die "Failed to stop containers" - -# Remove the containers -docker compose down --remove-orphans || _die "Failed to remove containers" - -# CRITICAL: Never remove data volumes in uninstall.sh! -# Data volumes must be preserved for potential reinstallation -# Only destroy.sh should remove volumes, and it must be explicit - -echo "Uninstallation of ${CONTAINER_NAME} complete" -echo "Note: Data volumes have been preserved. To remove all data, use destroy.sh" \ No newline at end of file +docker compose down || true +echo "Log Server uninstalled (data preserved)" \ No newline at end of file diff --git a/simple-logs/README.md b/simple-logs/README.md new file mode 100644 index 0000000..9788a4b --- /dev/null +++ b/simple-logs/README.md @@ -0,0 +1,61 @@ +# Simple Logs + +A dead-simple log viewer using Grafana + Loki. Much easier than ELK/Kibana! + +## Features +- **ALL logs in one place** - Docker containers + system logs +- **Simple interface** - Just one dashboard, no complex setup +- **Fast & lightweight** - Uses 10x less resources than ELK +- **Real-time** - Logs appear instantly +- **No configuration** - Works out of the box + +## Quick Start + +1. **Install** +```bash +dropshell install simple-logs +``` + +2. **Access** +- Go to: `http://:3000` +- Login: `admin` / `admin` (change on first login) + +3. **View Logs** +- Click the **Dashboards** icon (4 squares) on left +- Click **"Simple Logs"** +- That's it! All your logs are there + +## How to Use + +### Filter by Container +Click any `container_name` label to see only that container's logs + +### Search Text +Use the search box at the top to find specific text + +### Change Time Range +Top-right corner - set to "Last 5 minutes" for recent logs + +### View Details +Click any log line to expand and see full details + +## What Gets Logged +- All Docker container logs +- System logs (/var/log/syslog) +- Authentication logs (/var/log/auth.log) + +## Ports +- `3000` - Grafana Web UI +- `3100` - Loki API (internal) + +## Why This Instead of ELK? +- **10x simpler** - One dashboard vs Kibana's complexity +- **10x smaller** - ~200MB RAM vs 2-3GB for ELK +- **Instant setup** - No index patterns, no field mappings +- **Just works** - No configuration needed + +## Advanced: Custom Queries +In the logs panel, you can use LogQL queries: +- `{container_name="myapp"}` - Specific container +- `{job="syslog"} |= "error"` - System errors +- `{job="docker"} |~ "(?i)error|warn"` - All Docker warnings/errors \ No newline at end of file diff --git a/simple-logs/config/dashboards/simple-logs.json b/simple-logs/config/dashboards/simple-logs.json new file mode 100644 index 0000000..8325052 --- /dev/null +++ b/simple-logs/config/dashboards/simple-logs.json @@ -0,0 +1,69 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": "-- Grafana --", + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "gnetId": null, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [ + { + "datasource": "Loki", + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 20, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 2, + "options": { + "dedupStrategy": "none", + "enableLogDetails": true, + "prettifyLogMessage": false, + "showCommonLabels": false, + "showLabels": true, + "showTime": true, + "sortOrder": "Descending", + "wrapLogMessage": true + }, + "targets": [ + { + "expr": "{job=~\"docker|syslog|auth\"}", + "refId": "A" + } + ], + "title": "All Logs", + "type": "logs" + } + ], + "schemaVersion": 27, + "style": "dark", + "tags": ["logs"], + "templating": { + "list": [] + }, + "time": { + "from": "now-15m", + "to": "now" + }, + "timepicker": {}, + "timezone": "", + "title": "Simple Logs", + "uid": "simple-logs", + "version": 0 +} \ No newline at end of file diff --git a/simple-logs/config/grafana-dashboard.yml b/simple-logs/config/grafana-dashboard.yml new file mode 100644 index 0000000..35c89ff --- /dev/null +++ b/simple-logs/config/grafana-dashboard.yml @@ -0,0 +1,12 @@ +apiVersion: 1 + +providers: + - name: 'default' + orgId: 1 + folder: '' + type: file + disableDeletion: false + updateIntervalSeconds: 10 + allowUiUpdates: true + options: + path: /var/lib/grafana/dashboards \ No newline at end of file diff --git a/simple-logs/config/grafana-datasources.yml b/simple-logs/config/grafana-datasources.yml new file mode 100644 index 0000000..c680e97 --- /dev/null +++ b/simple-logs/config/grafana-datasources.yml @@ -0,0 +1,9 @@ +apiVersion: 1 + +datasources: + - name: Loki + type: loki + access: proxy + url: http://loki:3100 + isDefault: true + editable: false \ No newline at end of file diff --git a/simple-logs/config/loki.yaml b/simple-logs/config/loki.yaml new file mode 100644 index 0000000..7c4bb6a --- /dev/null +++ b/simple-logs/config/loki.yaml @@ -0,0 +1,36 @@ +auth_enabled: false + +server: + http_listen_port: 3100 + grpc_listen_port: 9096 + +common: + path_prefix: /loki + storage: + filesystem: + chunks_directory: /loki/chunks + rules_directory: /loki/rules + replication_factor: 1 + ring: + instance_addr: 127.0.0.1 + kvstore: + store: inmemory + +schema_config: + configs: + - from: 2020-10-24 + store: boltdb-shipper + object_store: filesystem + schema: v11 + index: + prefix: index_ + period: 24h + +ruler: + alertmanager_url: http://localhost:9093 + +limits_config: + retention_period: 168h # 7 days + enforce_metric_name: false + reject_old_samples: true + reject_old_samples_max_age: 168h \ No newline at end of file diff --git a/simple-logs/config/promtail.yaml b/simple-logs/config/promtail.yaml new file mode 100644 index 0000000..4407455 --- /dev/null +++ b/simple-logs/config/promtail.yaml @@ -0,0 +1,67 @@ +server: + http_listen_port: 9080 + grpc_listen_port: 0 + +positions: + filename: /tmp/positions/positions.yaml + +clients: + - url: http://loki:3100/loki/api/v1/push + +scrape_configs: + # Docker container logs + - job_name: docker + static_configs: + - targets: + - localhost + labels: + job: docker + __path__: /var/lib/docker/containers/*/*-json.log + pipeline_stages: + - json: + expressions: + stream: stream + time: time + log: log + - timestamp: + source: time + format: RFC3339Nano + - labels: + stream: + - regex: + expression: '(?P[0-9a-f]{12})' + source: filename + - labels: + container_id: + + # System logs + - job_name: syslog + static_configs: + - targets: + - localhost + labels: + job: syslog + __path__: /var/log/syslog + host: ${HOSTNAME} + + - job_name: auth + static_configs: + - targets: + - localhost + labels: + job: auth + __path__: /var/log/auth.log + host: ${HOSTNAME} + + # Docker events via Docker socket + - job_name: docker_events + docker_sd_configs: + - host: unix:///var/run/docker.sock + refresh_interval: 5s + relabel_configs: + - source_labels: [__meta_docker_container_name] + target_label: container_name + - source_labels: [__meta_docker_container_id] + target_label: container_id + - source_labels: [__meta_docker_container_image] + target_label: image \ No newline at end of file diff --git a/simple-logs/config/service.env b/simple-logs/config/service.env new file mode 100644 index 0000000..4d786a2 --- /dev/null +++ b/simple-logs/config/service.env @@ -0,0 +1,15 @@ +# Simple Log Viewer Configuration +CONTAINER_NAME=simple-logs + +# Server settings (REQUIRED by dropshell) +SSH_USER="root" + +# Web UI Port +WEB_PORT=3000 + +# Log retention (days) +LOG_RETENTION=7 + +# Authentication (optional - leave empty for no auth) +AUTH_USERNAME="" +AUTH_PASSWORD="" \ No newline at end of file diff --git a/simple-logs/docker-compose.yml b/simple-logs/docker-compose.yml new file mode 100644 index 0000000..70c40fd --- /dev/null +++ b/simple-logs/docker-compose.yml @@ -0,0 +1,54 @@ +version: '3.8' + +services: + # Grafana - Simple Web UI for viewing logs + grafana: + image: grafana/grafana:latest + container_name: ${CONTAINER_NAME}_grafana + environment: + - GF_SECURITY_ADMIN_USER=${AUTH_USERNAME:-admin} + - GF_SECURITY_ADMIN_PASSWORD=${AUTH_PASSWORD:-admin} + - GF_AUTH_ANONYMOUS_ENABLED=${AUTH_USERNAME:+false} + - GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer + - GF_INSTALL_PLUGINS= + volumes: + - grafana_data:/var/lib/grafana + - ${CONFIG_PATH}/grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yaml:ro + - ${CONFIG_PATH}/grafana-dashboard.yml:/etc/grafana/provisioning/dashboards/dashboard.yaml:ro + - ${CONFIG_PATH}/dashboards:/var/lib/grafana/dashboards:ro + ports: + - "${WEB_PORT:-3000}:3000" + restart: unless-stopped + depends_on: + - loki + + # Loki - Lightweight log storage (100x simpler than Elasticsearch) + loki: + image: grafana/loki:2.9.0 + container_name: ${CONTAINER_NAME}_loki + volumes: + - loki_data:/loki + - ${CONFIG_PATH}/loki.yaml:/etc/loki/local-config.yaml:ro + ports: + - "3100:3100" + command: -config.file=/etc/loki/local-config.yaml + restart: unless-stopped + + # Promtail - Collects all logs (Docker + System) + promtail: + image: grafana/promtail:2.9.0 + container_name: ${CONTAINER_NAME}_promtail + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - /var/log:/var/log:ro + - ${CONFIG_PATH}/promtail.yaml:/etc/promtail/config.yml:ro + - promtail_positions:/tmp/positions + command: -config.file=/etc/promtail/config.yml + restart: unless-stopped + depends_on: + - loki + +volumes: + grafana_data: + loki_data: + promtail_positions: \ No newline at end of file diff --git a/simple-logs/install.sh b/simple-logs/install.sh new file mode 100755 index 0000000..0e35ab3 --- /dev/null +++ b/simple-logs/install.sh @@ -0,0 +1,62 @@ +#!/bin/bash +source "${AGENT_PATH}/common.sh" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +_check_required_env_vars "CONTAINER_NAME" + +# Check Docker +_check_docker_installed || _die "Docker test failed" +docker compose version >/dev/null 2>&1 || _die "Docker Compose V2 is required" + +# Stop any existing containers +bash ./stop.sh 2>/dev/null || true + +# Create config directory +mkdir -p "${CONFIG_PATH}/dashboards" + +# Copy configuration files +cp "$SCRIPT_DIR/config/"*.yaml "$SCRIPT_DIR/config/"*.yml "${CONFIG_PATH}/" 2>/dev/null || true +cp "$SCRIPT_DIR/config/dashboards/"*.json "${CONFIG_PATH}/dashboards/" 2>/dev/null || true + +# Start the stack +echo "Starting Simple Logs stack..." +docker compose up -d || _die "Failed to start" + +# Wait for Grafana to be ready +echo -n "Waiting for Grafana to start..." +MAX_WAIT=60 +WAITED=0 +while [ $WAITED -lt $MAX_WAIT ]; do + if curl -s "http://localhost:${WEB_PORT:-3000}/api/health" 2>/dev/null | grep -q "ok"; then + echo " Ready!" + break + fi + echo -n "." + sleep 2 + WAITED=$((WAITED + 2)) +done +echo "" + +echo "=========================================" +echo "Simple Logs Installed!" +echo "=========================================" +echo "" +echo "Access at: http://$(hostname -I | awk '{print $1}'):${WEB_PORT:-3000}" +echo "" +if [ -n "${AUTH_USERNAME}" ]; then + echo "Login: ${AUTH_USERNAME} / ${AUTH_PASSWORD}" +else + echo "Login: admin / admin" + echo "(You'll be asked to change password on first login)" +fi +echo "" +echo "TO VIEW LOGS:" +echo "1. Click 'Dashboards' icon (4 squares) on left" +echo "2. Click 'Simple Logs'" +echo "3. Your logs appear immediately!" +echo "" +echo "TO FILTER:" +echo "- Click any label to filter by it" +echo "- Use the search box for text search" +echo "- Time range selector is in top-right" +echo "=========================================" \ No newline at end of file diff --git a/simple-logs/logs.sh b/simple-logs/logs.sh new file mode 100755 index 0000000..ea504dc --- /dev/null +++ b/simple-logs/logs.sh @@ -0,0 +1,12 @@ +#!/bin/bash +source "${AGENT_PATH}/common.sh" +_check_required_env_vars "CONTAINER_NAME" + +echo "=== Grafana Logs ===" +docker logs "${CONTAINER_NAME}_grafana" --tail 20 +echo "" +echo "=== Loki Logs ===" +docker logs "${CONTAINER_NAME}_loki" --tail 20 +echo "" +echo "=== Promtail Logs ===" +docker logs "${CONTAINER_NAME}_promtail" --tail 20 \ No newline at end of file diff --git a/simple-logs/start.sh b/simple-logs/start.sh new file mode 100755 index 0000000..12c40b2 --- /dev/null +++ b/simple-logs/start.sh @@ -0,0 +1,6 @@ +#!/bin/bash +source "${AGENT_PATH}/common.sh" +_check_required_env_vars "CONTAINER_NAME" + +docker compose up -d || _die "Failed to start Simple Logs" +echo "Simple Logs started" \ No newline at end of file diff --git a/simple-logs/status.sh b/simple-logs/status.sh new file mode 100755 index 0000000..ef18f39 --- /dev/null +++ b/simple-logs/status.sh @@ -0,0 +1,11 @@ +#!/bin/bash +source "${AGENT_PATH}/common.sh" +_check_required_env_vars "CONTAINER_NAME" + +if docker ps | grep -q "${CONTAINER_NAME}_grafana"; then + echo "Running" + echo " Grafana: http://$(hostname -I | awk '{print $1}'):${WEB_PORT:-3000}" + docker ps --format "table {{.Names}}\t{{.Status}}" | grep "${CONTAINER_NAME}_" +else + echo "Stopped" +fi \ No newline at end of file diff --git a/simple-logs/stop.sh b/simple-logs/stop.sh new file mode 100755 index 0000000..8b30519 --- /dev/null +++ b/simple-logs/stop.sh @@ -0,0 +1,6 @@ +#!/bin/bash +source "${AGENT_PATH}/common.sh" +_check_required_env_vars "CONTAINER_NAME" + +docker compose stop || _die "Failed to stop Simple Logs" +echo "Simple Logs stopped" \ No newline at end of file diff --git a/simple-logs/uninstall.sh b/simple-logs/uninstall.sh new file mode 100755 index 0000000..ab01fe8 --- /dev/null +++ b/simple-logs/uninstall.sh @@ -0,0 +1,6 @@ +#!/bin/bash +source "${AGENT_PATH}/common.sh" +_check_required_env_vars "CONTAINER_NAME" + +docker compose down || true +echo "Simple Logs uninstalled (data preserved)" \ No newline at end of file