Simple Object Storage

A simple object storage system that stores files with metadata and provides a REST API for access.

Features

• Store files with metadata (labels, tags, and custom fields)
• Retrieve files by hash or label:tag combination
• Check if a file exists by hash or label:tag
• Delete files by hash
• List all stored objects
• Automatic file deduplication using content hashing
• Support for large file uploads
• Configurable storage location and server settings
• Token-based authentication for write operations

Building

mkdir build
cd build
cmake ..
make

Configuration

The server can be configured by creating a JSON configuration file at ~/.config/simple_object_storage/config.json. Here's an example configuration:

{
    "host": "localhost",
    "port": 8080,
    "storage_path": "/path/to/storage",
    "write_tokens": ["your-secret-token"]
}

API Endpoints

Upload a File

PUT /upload

Parameters:

• file: The file to upload
• metadata: JSON object containing:
  • labels: Array of strings (required)
  • tags: Array of strings (required)
  • Additional custom fields (optional)

Example:

curl -X PUT \
  -H "Authorization: Bearer your-token" \
  -F "file=@example.txt" \
  -F 'metadata={"labels":["test"],"tags":["latest"],"description":"Example file"}' \
  http://localhost:8080/upload

Get a File

GET /object/{hash}
GET /object/{label}:{tag}

Example:

curl http://localhost:8080/object/abc123
curl http://localhost:8080/object/test:latest

Check if a File Exists

GET /exists/{hash}
GET /exists/{label}:{tag}

Example:

curl http://localhost:8080/exists/abc123
curl http://localhost:8080/exists/test:latest

Delete a File

GET /deleteobject?hash={hash}

Example:

curl -H "Authorization: Bearer your-token" http://localhost:8080/deleteobject?hash=abc123

List All Objects

GET /list

Example:

curl http://localhost:8080/list

Database Schema

The system uses SQLite to store metadata about uploaded files. The database schema is as follows:

CREATE TABLE objects (
    hash TEXT PRIMARY KEY,
    labels TEXT NOT NULL,  -- JSON array of labels
    tags TEXT NOT NULL,    -- JSON array of tags
    metadata TEXT NOT NULL -- JSON object with additional metadata
);

Testing

The repository includes two test scripts:

• test.sh: Basic functionality tests
• test_1GB_file_upload.sh: Tests uploading and downloading a 1GB file

To run the tests:

./test.sh
./test_1GB_file_upload.sh

License

This project is licensed under the MIT License - see the LICENSE file for details.