simple-object-server/README.md

# Simple Object Storage

A simple object storage system that stores files with metadata and provides a REST API for access.


## Features

- Store files with metadata (label:tag pairs and custom fields)
- Retrieve files by hash or label:tag combination
- Check if a file exists by hash or label:tag
- Delete files by hash
- List all stored objects
- Automatic file deduplication using content hashing
- Support for large file uploads (up to 6GB)
- High-performance HTTP server with async request handling
- Configurable storage location and server settings
- Token-based authentication for write operations
- CORS support for web applications
- Rate limiting for security


## Installation

### Quick Install (Pre-built Binaries)

Download and install both the server and hash utility:
```bash
wget -q https://getbin.xyz/simple-object-server-install:latest -O- | bash
```

This installs:
- `simple-object-server` - The main server binary
- `sos-hash` - Utility for generating bcrypt hashes for authentication tokens

The binaries are installed to `~/.local/bin` (or `/usr/local/bin` if run as root).

### Running with Docker

```bash
docker run -d \
  -p 8080:8080 \
  -v /path/to/storage:/data/storage \
  -v /path/to/sos_config.json:/data/sos_config.json:ro \
  --name object-server \
  gitea.jde.nz/public/simple-object-server
```

This will:
- Map port 8080 from the container to your host
- Mount your storage directory to `/data` in the container
- Mount your config directory to the container
- Run the service in detached mode

Ensure that the `storage_path` set in the `sos_config.json` is `/data/storage`, the path inside the container.

### Using Docker Compose

Alternatively, you can use Docker Compose. Create a `docker-compose.yml` file:

```yaml
version: '3'
services:
  object-server:
    image: gitea.jde.nz/public/simple-object-server
    ports:
      - "8080:8080"
    volumes:
      - /path/to/storage:/data/storage
      - /path/to/sos_config.json:/data/sos_config.json:ro
    restart: unless-stopped
```

Then run:
```bash
docker-compose up -d
```
## Manual Install

To install the latest version of simple-object-server, run:
```bash
curl https://getbin.xyz/simple-object-server-install | bash
```

## Configuration

The server can be configured by creating a JSON configuration file at `~/.config/simple-object-server/sos_config.json`. Default values are shown below (everything but write tokens), suitable for running in Docker.

### Secure Token Configuration

**IMPORTANT**: The server configuration must contain bcrypt hashes, NOT plaintext tokens. Clients send plaintext tokens, server stores hashes.

#### Step-by-Step Token Setup

1. **Generate a secure random token** (keep this secret - this is what clients will use):
   ```bash
   # Generate a strong random token
   TOKEN=$(openssl rand -base64 32)
   echo "Save this token for client use: $TOKEN"
   ```

2. **Hash the token for server configuration** using the `sos-hash` utility:
   ```bash
   # If you installed via the quick install method, use:
   sos-hash
   Enter token to hash: [paste your token here]
   
   # Or pipe it directly
   echo "$TOKEN" | sos-hash -q
   
   # Or generate both token and hash at once
   sos-hash --generate
   # This outputs both the plaintext token (for clients) and hash (for config)
   
   # If building from source, use:
   ./output/hash_token
   ```

3. **Put the HASH (not the token) in your server configuration**:

```json
{
    "host": "0.0.0.0",
    "port": 80,
    "storage_path": "/data/storage",
    "write_tokens": [
        "$2b$12$7d5c2e5f4a3b1e9f8c7b6a5d4e3f2a1b9c8d7e6f5a4b3c2d1e9f8a7b6c5d4e3f"
        // This is the HASH, not the plaintext token!
    ],
    "cors": {
        "allowed_origins": ["*"],
        "allowed_methods": ["GET", "PUT", "POST", "DELETE", "OPTIONS"],
        "allowed_headers": ["Authorization", "Content-Type"],
        "allow_credentials": false
    },
   "rate_limiting": {
        "auth_rate_limit": 5,        // Maximum number of auth attempts
        "auth_window_seconds": 300   // Time window in seconds (5 minutes)
    }
}
```

#### Complete Example

```bash
# 1. Generate a secure token
TOKEN=$(openssl rand -base64 32)
echo "Client token: $TOKEN"
# Output: Client token: 3ezzqHF9UNcIokHK5AAC1098eaTLLcd5hW2FbOAHP4Q=

# 2. Hash it for the server config (using installed sos-hash)
HASH=$(echo "$TOKEN" | sos-hash -q)
echo "Server hash: $HASH"
# Output: Server hash: $2b$12$...long hash string...

# 3. Put the HASH in sos_config.json (NOT the token!)
# 4. Clients use the TOKEN (NOT the hash!) in API calls:
curl -H "Authorization: Bearer $TOKEN" ...
```

#### Security Notes

- **Never store plaintext tokens** in configuration files
- **Server config gets the hash**: The bcrypt hash goes in `sos_config.json`
- **Clients use the plaintext token**: API calls use `Bearer <plaintext-token>`
- **Use strong tokens**: At least 32 characters of random data
- **Rotate tokens regularly**: Generate new tokens periodically
- **Cost factor**: Default is 12, increase for higher security (each increment doubles the computation time)

## Building

To build output/simple-object-server for the current architecture run:
```bash
./build.sh
```

To build and test the docker image gitea.jde.nz/public/simple-object-server:test, run:
```bash
./test.sh
```

To build for multiple architectures, and publish the combined docker image to gitea.jde.nz, run:
```bash
./publish.sh
```

## API Endpoints

### Upload a File

```
PUT /upload
```

Parameters:
- `file`: The file to upload
- `metadata`: JSON object containing:
  - `labeltags`: Array of strings in "label:tag" format (required)
  - Additional custom fields (optional)

Example:
```bash
curl -X PUT \
  -H "Authorization: Bearer your-token" \
  -F "file=@example.txt" \
  -F 'metadata={"labeltags":["test:latest","project:alpha"],"description":"Example file"}' \
  http://localhost:8080/upload
```

### PUT /update

Update the metadata for an existing object. The request must be authenticated with a valid write token. The request body must be JSON and include both a `hash` (the object's hash) and a `metadata` object.

**Request:**
- Method: PUT
- URL: /update
- Headers:
  - Authorization: Bearer <WRITE_TOKEN>
  - Content-Type: application/json
- Body:
```json
{
  "hash": "<object_hash>",
  "metadata": {
    "key1": "value1",
    "key2": "value2"
  }
}
```

**Response:**
- 200 OK on success:
```json
{
  "result": "success",
  "hash": "<object_hash>"
}
```
- 400 Bad Request if missing fields or invalid JSON
- 404 Not Found if the object does not exist
- 401/403 if authentication fails

**Example:**
```sh
curl -X PUT http://localhost:8080/update \
  -H "Authorization: Bearer <WRITE_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"hash": "abc123", "metadata": {"foo": "bar"}}'
```

### Get a File

```
GET /{hash}
GET /{label}:{tag}
GET /object/{hash}
GET /object/{label}:{tag}
```

Example:
```bash
curl http://localhost:8080/abc123 > abc123.txt
curl http://localhost:8080/test:latest > test.sh
curl http://localhost:8080/object/abc123 > abc123.txt
```

### Get Object Metadata

```
GET /meta/{hash}
GET /meta/{label}:{tag}
```

Example:
```bash
curl http://localhost:8080/meta/abc123
curl http://localhost:8080/meta/test:latest
```

### Get Hash for Label:Tag

```
GET /hash/{label}:{tag}
```

Example:
```bash
curl http://localhost:8080/hash/test:latest
```

### Get Version for Label:Tag

```
GET /version/{label}:{tag}
```

Example:
```bash
curl http://localhost:8080/version/test:latest
```

### Check if a File Exists

```
GET /exists/{hash}
GET /exists/{label}:{tag}
```

Example:
```bash
curl http://localhost:8080/exists/abc123
curl http://localhost:8080/exists/test:latest
```

### Delete a File

```
GET /deleteobject?hash={hash}
```

Example:
```bash
curl -H "Authorization: Bearer your-token" http://localhost:8080/deleteobject?hash=abc123
```

### List All Objects

```
GET /dir
```

Example:
```bash
curl http://localhost:8080/dir
```

### Status Endpoint

```
GET /status
```

Example:
```bash
curl http://localhost:8080/status
```

## Database Schema

The system uses SQLite to store metadata about uploaded files. The database schema is as follows:

```sql
CREATE TABLE objects (
    hash TEXT PRIMARY KEY,
    labeltags TEXT NOT NULL,  -- JSON array of label:tag pairs
    metadata TEXT NOT NULL    -- JSON object with additional metadata
);
```

## Testing

The repository includes comprehensive test scripts:
- `test.sh`: Complete test suite including basic functionality, metadata preservation, tag versioning, rate limiting, and 1GB file upload tests
- `testing/test.sh`: Direct test script for integration testing
- `testing/test_1GB_file_upload.sh`: Standalone 1GB file upload test

To run the full test suite:
```bash
./test.sh
```

To run individual tests:
```bash
./testing/test.sh
./testing/test_1GB_file_upload.sh
```

## License

This project is licensed under the MIT License - see the LICENSE file for details.