dropshell/TEMPLATES.md

# Dropshell Template Guide

This guide explains how to create templates for Dropshell - a system management tool for deploying and managing services on remote servers via Docker containers.

## Template Structure

Every template must follow this directory structure:

```
template-name/
├── config/
│   ├── .template_info.env    # REQUIRED: Template metadata
│   └── service.env            # REQUIRED: Default service configuration
├── install.sh                 # REQUIRED: Installation script
├── uninstall.sh              # REQUIRED: Uninstallation script
├── start.sh                  # OPTIONAL: Start the service
├── stop.sh                   # OPTIONAL: Stop the service
├── status.sh                 # OPTIONAL: Check service status
├── logs.sh                   # OPTIONAL: View service logs
├── backup.sh                 # OPTIONAL: Backup service data
├── restore.sh                # OPTIONAL: Restore service data
├── destroy.sh                # OPTIONAL: Destroy service and data
├── ssh.sh                    # OPTIONAL: SSH into service container
├── ports.sh                  # OPTIONAL: List exposed ports
└── README.txt                # OPTIONAL: Template documentation
```

## Important Script Requirements

**All template scripts MUST be non-interactive.** Scripts should never prompt for user input or require any manual interaction. This is critical for automated deployments and remote management.

- **install.sh, uninstall.sh**: Must complete without any user prompts
- **Data preservation**: Use backup.sh/restore.sh for data management instead of prompting during uninstall
- **Configuration**: All settings must come from environment variables or config files
- **Confirmations**: Never use `read`, `select`, or any interactive prompts
- **Errors**: Use `_die` from common.sh to handle errors and exit cleanly

## Critical Data Preservation Rules

**NEVER DESTROY DATA VOLUMES IN UNINSTALL.SH**

This is a fundamental principle of Dropshell templates:

1. **uninstall.sh MUST preserve all data volumes** - Only remove containers, never volumes
2. **Data volumes contain irreplaceable user data** - Timestamps, logs, databases, configurations
3. **Only destroy.sh should remove volumes** - And it must be a separate, explicit action
4. **Volume removal is IRREVERSIBLE** - Once deleted, data cannot be recovered

### Correct Implementation

**uninstall.sh** - Preserves data:
```bash
#!/bin/bash
source "${AGENT_PATH}/common.sh"
_check_required_env_vars "CONTAINER_NAME"

# Stop and remove container only
bash ./stop.sh || _die "Failed to stop container"
_remove_container "$CONTAINER_NAME" || _die "Failed to remove container"

# DO NOT remove volumes here!
# Data volumes are preserved for potential reinstallation

echo "Uninstallation complete. Data volumes preserved."
echo "To remove data permanently, run: ./destroy.sh"
```

**destroy.sh** - Explicitly removes everything:
```bash
#!/bin/bash
source "${AGENT_PATH}/common.sh"
_check_required_env_vars "CONTAINER_NAME" "DATA_VOLUME"

echo "WARNING: This will PERMANENTLY DELETE all data for ${CONTAINER_NAME}"
echo "This action cannot be undone!"

# Since scripts must be non-interactive, only proceed if explicitly called
bash ./stop.sh
_remove_container "$CONTAINER_NAME"
_remove_volume "$DATA_VOLUME"  # Only destroy.sh removes volumes

echo "Service and all data destroyed"
```

### Why This Matters

- **User Trust**: Users expect uninstall to be reversible by reinstalling
- **Data Safety**: Application data often contains critical business information
- **Timestamps**: Historical data and audit logs are irreplaceable
- **Recovery**: Allows users to reinstall without losing their work
- **Best Practice**: Follows industry standards for application management

## Essential Files

### 1. config/.template_info.env

Template metadata file that defines the template and its requirements. This file is managed by the template author and should NOT be modified by users:

```bash
# Template identifier - MUST match the directory name
TEMPLATE=template-name

# Requirements
REQUIRES_HOST_ROOT=false      # Whether root access on host is needed
REQUIRES_DOCKER=true          # Whether Docker is required
REQUIRES_DOCKER_ROOT=false    # Whether Docker root privileges are needed

# Docker image settings
IMAGE_REGISTRY="docker.io"
IMAGE_REPO="vendor/image"
IMAGE_TAG="latest"

# Volume definitions (if needed)
DATA_VOLUME="${CONTAINER_NAME}_data"
CONFIG_VOLUME="${CONTAINER_NAME}_config"
```

**Important**: This file is replaced when the template is updated. Users should never edit this file.

### 2. config/service.env

Default service configuration that **will be modified per installation**. When a user creates a service from this template, they edit this file to customize the deployment:

```bash
# Service identification (REQUIRED)
CONTAINER_NAME=service-name

# Server settings (REQUIRED by dropshell)
SSH_USER="root"  # CRITICAL: This must always be defined!

# Docker settings (optional)
IMAGE_TAG="latest"

# Service-specific variables
# (Add any configuration specific to your service)
```

**CRITICAL REQUIREMENT**: The `SSH_USER` variable MUST be defined in every service.env file. Dropshell uses this internally for server management. Without it, the template will fail with "SSH_USER variable not defined" error.

**Important**: This is the file users edit to customize each service instance. Different deployments of the same template will have different `service.env` files.

### How Environment Variables Work

When Dropshell runs any template script (install.sh, start.sh, etc.), it:

1. **First loads** all variables from `config/.template_info.env`
2. **Then loads** all variables from `config/service.env` (which can override template defaults)
3. **Exports** all these variables to the environment
4. **Executes** the requested script with these variables available

This means:
- Scripts can use any variable defined in either file directly (e.g., `$CONTAINER_NAME`, `$IMAGE_TAG`)
- Variables in `service.env` override those in `.template_info.env` if they have the same name
- Users customize deployments by editing `service.env`, never `.template_info.env`

### 3. install.sh

Installation script that sets up the service:

```bash
#!/bin/bash
source "${AGENT_PATH}/common.sh"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Check required environment variables
_check_required_env_vars "CONTAINER_NAME" "IMAGE_REGISTRY" "IMAGE_REPO" "IMAGE_TAG"

# Check Docker is available
_check_docker_installed || _die "Docker test failed"

# Pull the Docker image
docker pull "$IMAGE_REGISTRY/$IMAGE_REPO:$IMAGE_TAG" || _die "Failed to pull image"

# Stop any existing container
bash ./stop.sh || _die "Failed to stop container"

# Remove old container
_remove_container "$CONTAINER_NAME" || _die "Failed to remove container"

# Start the new container
bash ./start.sh || _die "Failed to start container"

echo "Installation of ${CONTAINER_NAME} complete"
```

### 4. uninstall.sh

Uninstallation script to remove the service (must be non-interactive and MUST preserve data):

```bash
#!/bin/bash
source "${AGENT_PATH}/common.sh"
_check_required_env_vars "CONTAINER_NAME"

# Stop the container
bash ./stop.sh || _die "Failed to stop container"

# Remove the container
_remove_container "$CONTAINER_NAME" || _die "Failed to remove container"

# CRITICAL: Never remove data volumes in uninstall.sh!
# Data volumes must be preserved for potential reinstallation
# Only destroy.sh should remove volumes, and it must be explicit

echo "Uninstallation of ${CONTAINER_NAME} complete"
echo "Note: Data volumes have been preserved. To remove all data, use destroy.sh"
```

## Optional Command Scripts

### start.sh
Starts the Docker container with appropriate configuration:

```bash
#!/bin/bash
source "${AGENT_PATH}/common.sh"
_check_required_env_vars "CONTAINER_NAME" "IMAGE_REGISTRY" "IMAGE_REPO" "IMAGE_TAG"

docker run -d \
    --name "$CONTAINER_NAME" \
    --restart unless-stopped \
    -v "$DATA_VOLUME:/data" \
    "$IMAGE_REGISTRY/$IMAGE_REPO:$IMAGE_TAG"
```

### stop.sh
Stops the running container:

```bash
#!/bin/bash
source "${AGENT_PATH}/common.sh"
_check_required_env_vars "CONTAINER_NAME"

docker stop "$CONTAINER_NAME" 2>/dev/null || true
```

### status.sh
Reports the service status.

**Expected Output Format:**
- Must output a single line with one of these exact status values:
  - `Running` - Service is active and operational
  - `Stopped` - Service is stopped but configured
  - `Error` - Service is in an error state
  - `Unknown` - Status cannot be determined

The output is parsed by Dropshell for the `list` command, so it must be exactly one of these values with no additional text.

```bash
#!/bin/bash
source "${AGENT_PATH}/common.sh"
_check_required_env_vars "CONTAINER_NAME"

if docker ps --format "table {{.Names}}" | grep -q "^${CONTAINER_NAME}$"; then
    echo "Running"
else
    echo "Stopped"
fi
```

For more complex status checks:

```bash
#!/bin/bash
source "${AGENT_PATH}/common.sh"
_check_required_env_vars "CONTAINER_NAME"

# Check if container exists
if ! docker ps -a --format "{{.Names}}" | grep -q "^${CONTAINER_NAME}$"; then
    echo "Unknown"
    exit 0
fi

# Check container state
STATE=$(docker inspect -f '{{.State.Status}}' "$CONTAINER_NAME" 2>/dev/null)
case "$STATE" in
    running)
        # Additional health check if needed
        if docker inspect -f '{{.State.Health.Status}}' "$CONTAINER_NAME" 2>/dev/null | grep -q "unhealthy"; then
            echo "Error"
        else
            echo "Running"
        fi
        ;;
    exited|stopped)
        echo "Stopped"
        ;;
    restarting|paused)
        echo "Error"
        ;;
    *)
        echo "Unknown"
        ;;
esac
```

### logs.sh
Shows container logs:

```bash
#!/bin/bash
source "${AGENT_PATH}/common.sh"
_check_required_env_vars "CONTAINER_NAME"

docker logs "$CONTAINER_NAME" "$@"
```

### backup.sh / restore.sh
Handle data backup and restoration for services with persistent data.

## Common Functions (from common.sh)

Templates have access to these utility functions:

- `_check_required_env_vars VAR1 VAR2 ...` - Verify required variables are set
- `_die "message"` - Exit with error message
- `_check_docker_installed` - Verify Docker is available
- `_remove_container NAME` - Safely remove a container
- `create_items ITEM1 ITEM2 ...` - Create volumes or networks
- `_remove_volume NAME` - Remove a Docker volume

## Environment Variables

Templates receive these variables at runtime:

- `AGENT_PATH` - Path to the Dropshell agent directory
- `CONFIG_PATH` - Path to the service configuration directory
- `SCRIPT_DIR` - Directory containing the template scripts
- All variables from `.template_info.env` and `service.env`

## Validation Requirements

Templates must pass these validation checks:

1. **Required files exist**: `config/.template_info.env`, `config/service.env`, `install.sh`, `uninstall.sh`
2. **Scripts are executable**: All `.sh` files must have execute permissions
3. **TEMPLATE variable matches**: The `TEMPLATE` variable in `.template_info.env` must match the directory name
4. **Valid environment files**: Both `.env` files must be parseable
5. **SSH_USER defined**: The `SSH_USER` variable MUST be present in `service.env`

## Best Practices

1. **Always use `_check_required_env_vars`** at the start of scripts to validate inputs
2. **Handle errors gracefully** with `|| _die "Error message"`
3. **Use Docker volumes** for persistent data that survives container recreation
4. **NEVER remove data volumes in uninstall.sh** - Only destroy.sh should remove volumes
5. **Document configuration** in README.txt for users
6. **Support idempotency** - scripts should handle being run multiple times
7. **Separate concerns** - uninstall.sh removes service, destroy.sh removes data
8. **Use standard naming** - volumes as `${CONTAINER_NAME}_data`, configs as `${CONTAINER_NAME}_config`
9. **Test thoroughly** - use `dropshell test-template <path>` to validate
10. **Preserve user data** - Always prioritize data safety over convenience

## Examples

### Example 1: Basic Web Server (Nginx)

A minimal template for deploying an Nginx web server:

```
nginx-server/
├── config/
│   ├── .template_info.env
│   └── service.env
├── install.sh
├── uninstall.sh
├── start.sh
├── stop.sh
└── logs.sh
```

`.template_info.env`:
```bash
TEMPLATE=nginx-server
REQUIRES_DOCKER=true
REQUIRES_DOCKER_ROOT=false
IMAGE_REGISTRY="docker.io"
IMAGE_REPO="nginx"
CONTENT_VOLUME="${CONTAINER_NAME}_content"
```

`service.env`:
```bash
CONTAINER_NAME=nginx-server
SSH_USER="root"
IMAGE_TAG="alpine"
HTTP_PORT=8080
```

`start.sh`:
```bash
#!/bin/bash
source "${AGENT_PATH}/common.sh"
_check_required_env_vars "CONTAINER_NAME" "IMAGE_REGISTRY" "IMAGE_REPO" "IMAGE_TAG" "HTTP_PORT"

docker run -d \
    --name "$CONTAINER_NAME" \
    --restart unless-stopped \
    -p "${HTTP_PORT}:80" \
    -v "${CONTENT_VOLUME}:/usr/share/nginx/html:ro" \
    "$IMAGE_REGISTRY/$IMAGE_REPO:$IMAGE_TAG"
```

### Example 2: Stateful Application with Persistent Storage

A more complex template for a database or application requiring data persistence, configuration files, and network settings:

```
postgres-db/
├── config/
│   ├── .template_info.env
│   ├── service.env
│   └── postgres.conf        # Additional config files
├── install.sh
├── uninstall.sh
├── start.sh
├── stop.sh
├── status.sh
├── backup.sh                # Data backup
├── restore.sh               # Data restoration
├── destroy.sh               # Complete cleanup including volumes
├── logs.sh
└── README.txt
```

`.template_info.env`:
```bash
TEMPLATE=postgres-db
REQUIRES_DOCKER=true
REQUIRES_DOCKER_ROOT=true    # Needed for volume management
IMAGE_REGISTRY="docker.io"
IMAGE_REPO="postgres"

# Multiple volumes for different purposes
DATA_VOLUME="${CONTAINER_NAME}_data"
BACKUP_VOLUME="${CONTAINER_NAME}_backup"
CONFIG_VOLUME="${CONTAINER_NAME}_config"
```

`service.env`:
```bash
CONTAINER_NAME=postgres-db
SSH_USER="root"
IMAGE_TAG="15-alpine"
DB_PORT=5432
POSTGRES_PASSWORD="changeme"
POSTGRES_USER="admin"
POSTGRES_DB="myapp"
```

`backup.sh`:
```bash
#!/bin/bash
source "${AGENT_PATH}/common.sh"
_check_required_env_vars "CONTAINER_NAME" "BACKUP_VOLUME" "POSTGRES_USER" "POSTGRES_DB"

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="backup_${TIMESTAMP}.sql"

echo "Creating backup ${BACKUP_FILE}..."
docker exec "$CONTAINER_NAME" pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" > "/tmp/${BACKUP_FILE}"
docker cp "/tmp/${BACKUP_FILE}" "${CONTAINER_NAME}:/${BACKUP_VOLUME}/${BACKUP_FILE}"
rm "/tmp/${BACKUP_FILE}"

echo "Backup completed: ${BACKUP_FILE}"
```

`destroy.sh`:
```bash
#!/bin/bash
source "${AGENT_PATH}/common.sh"
_check_required_env_vars "CONTAINER_NAME" "DATA_VOLUME" "BACKUP_VOLUME" "CONFIG_VOLUME"

echo "WARNING: This will destroy all data for ${CONTAINER_NAME}"
read -p "Are you sure? (yes/no): " confirm

if [ "$confirm" = "yes" ]; then
    bash ./stop.sh
    _remove_container "$CONTAINER_NAME"
    _remove_volume "$DATA_VOLUME"
    _remove_volume "$BACKUP_VOLUME"
    _remove_volume "$CONFIG_VOLUME"
    echo "Service and all data destroyed"
else
    echo "Cancelled"
fi
```

## Docker Compose Templates

When creating templates that use Docker Compose:

### Important: Always Build on Install

When using `docker compose` with custom images (defined with `build:` in your compose file), **always use the `--build` flag** in your `install.sh`:

```bash
# CORRECT - Forces rebuild of images during installation
docker compose up -d --build

# WRONG - May use stale cached images
docker compose up -d
```

This ensures that:
- Custom images are rebuilt with the latest code changes
- No stale images are used from previous installations
- The service always starts with the most recent image version

### Example Docker Compose install.sh

```bash
#!/bin/bash
source "${AGENT_PATH}/common.sh"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

_check_required_env_vars "CONTAINER_NAME"
_check_docker_installed || _die "Docker test failed"

# Stop any existing containers
docker compose down || true

# Start with rebuild to ensure fresh images
docker compose up -d --build || _die "Failed to start services"

echo "Installation of ${CONTAINER_NAME} complete"
```

### Docker Compose Best Practices

1. **Always include `--build`** in install.sh when using custom images
2. **Use `.env` files** for configuration that Docker Compose can read
3. **Define service names** consistently with `CONTAINER_NAME`
4. **Handle cleanup** properly in stop.sh and uninstall.sh using `docker compose down`
5. **Use named volumes** for persistent data that matches Dropshell conventions

## Testing Templates

After creating a template, validate it:

```bash
dropshell test-template /path/to/template
```

This checks all requirements and reports any issues.