Simple Object Storage

A simple object storage system that stores files with metadata and provides a REST API for access.

Install

To install the latest version of simple-object-server, run:

curl https://getbin.xyz/simple-object-server-install | bash

Features

• Store files with metadata (label:tag pairs and custom fields)
• Retrieve files by hash or label:tag combination
• Check if a file exists by hash or label:tag
• Delete files by hash
• List all stored objects
• Automatic file deduplication using content hashing
• Support for large file uploads
• Configurable storage location and server settings
• Token-based authentication for write operations

Building

To build output/simple-object-server for the current architecture run:

./build.sh

To build and test the docker image gitea.jde.nz/public/simple-object-server:test, run:

./test.sh

To build for multiple architectures, and publish the combined docker image to gitea.jde.nz, run:

./publish.sh

Running with Docker

docker run -d \
  -p 8080:8080 \
  -v /path/to/storage:/data/storage \
  -v /path/to/sos_config.json:/data/sos_config.json:ro \
  --name object-server \
  gitea.jde.nz/public/simple-object-server

This will:

• Map port 8080 from the container to your host
• Mount your storage directory to /data in the container
• Mount your config directory to the container
• Run the service in detached mode

Ensure that the storage_path set in the sos_config.json is /data/storage, the path inside the container.

Using Docker Compose

Alternatively, you can use Docker Compose. Create a docker-compose.yml file:

version: '3'
services:
  object-server:
    image: gitea.jde.nz/public/simple-object-server
    ports:
      - "8080:8080"
    volumes:
      - /path/to/storage:/data/storage
      - /path/to/sos_config.json:/data/sos_config.json:ro
    restart: unless-stopped

Then run:

docker-compose up -d

Configuration

The server can be configured by creating a JSON configuration file at ~/.config/simple-object-server/sos_config.json. Default values are shown below (everything but write tokens), suitable for running in Docker.

{
    "host": "0.0.0.0",
    "port": 80,
    "storage_path": "/data/storage",
    "write_tokens": ["your-secret-token"],
    "cors": {
        "allowed_origins": ["*"],
        "allowed_methods": ["GET", "PUT", "POST", "DELETE", "OPTIONS"],
        "allowed_headers": ["Authorization", "Content-Type"],
        "allow_credentials": false
    },
   "rate_limiting": {
        "auth_rate_limit": 5,        // Maximum number of auth attempts
        "auth_window_seconds": 300   // Time window in seconds (5 minutes)
    }
}

API Endpoints

Upload a File

PUT /upload

Parameters:

• file: The file to upload
• metadata: JSON object containing:
  • labeltags: Array of strings in "label:tag" format (required)
  • Additional custom fields (optional)

Example:

curl -X PUT \
  -H "Authorization: Bearer your-token" \
  -F "file=@example.txt" \
  -F 'metadata={"labeltags":["test:latest","project:alpha"],"description":"Example file"}' \
  http://localhost:8080/upload

PUT /update

Update the metadata for an existing object. The request must be authenticated with a valid write token. The request body must be JSON and include both a hash (the object's hash) and a metadata object.

Request:

• Method: PUT
• URL: /update
• Headers:
  • Authorization: Bearer <WRITE_TOKEN>
  • Content-Type: application/json
• Body:

{
  "hash": "<object_hash>",
  "metadata": {
    "key1": "value1",
    "key2": "value2"
  }
}

Response:

• 200 OK on success:

{
  "result": "success",
  "hash": "<object_hash>"
}

• 400 Bad Request if missing fields or invalid JSON
• 404 Not Found if the object does not exist
• 401/403 if authentication fails

Example:

curl -X PUT http://localhost:8080/update \
  -H "Authorization: Bearer <WRITE_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"hash": "abc123", "metadata": {"foo": "bar"}}'

Get a File

GET /{hash}
GET /{label}:{tag}
GET /object/{hash}
GET /object/{label}:{tag}

Example:

curl http://localhost:8080/abc123 > abc123.txt
curl http://localhost:8080/test:latest > test.sh
curl http://localhost:8080/object/abc123 > abc123.txt

Get Object Metadata

GET /meta/{hash}
GET /meta/{label}:{tag}

Example:

curl http://localhost:8080/meta/abc123
curl http://localhost:8080/meta/test:latest

Get Hash for Label:Tag

GET /hash/{label}:{tag}

Example:

curl http://localhost:8080/hash/test:latest

Get Version for Label:Tag

GET /version/{label}:{tag}

Example:

curl http://localhost:8080/version/test:latest

Check if a File Exists

GET /exists/{hash}
GET /exists/{label}:{tag}

Example:

curl http://localhost:8080/exists/abc123
curl http://localhost:8080/exists/test:latest

Delete a File

GET /deleteobject?hash={hash}

Example:

curl -H "Authorization: Bearer your-token" http://localhost:8080/deleteobject?hash=abc123

List All Objects

GET /dir

Example:

curl http://localhost:8080/dir

Status Endpoint

GET /status

Example:

curl http://localhost:8080/status

Database Schema

The system uses SQLite to store metadata about uploaded files. The database schema is as follows:

CREATE TABLE objects (
    hash TEXT PRIMARY KEY,
    labeltags TEXT NOT NULL,  -- JSON array of label:tag pairs
    metadata TEXT NOT NULL    -- JSON object with additional metadata
);

Testing

The repository includes two test scripts:

• test.sh: Basic functionality tests
• test_1GB_file_upload.sh: Tests uploading and downloading a 1GB file

To run the tests:

./test.sh
./test_1GB_file_upload.sh

License

This project is licensed under the MIT License - see the LICENSE file for details.