Bug fixing
This commit is contained in:
Your Name
193
README.md
193
README.md
@@ -1,95 +1,138 @@
|
||||
# Simple Object Storage
|
||||
|
||||
## Introduction
|
||||
A simple object storage system that stores files with metadata and provides a REST API for access.
|
||||
|
||||
Simple Object Storage is a very simple C++ webserver
|
||||
which provides a store of tagged binary objects (the objects can be large),
|
||||
which are available over http.
|
||||
## Features
|
||||
|
||||
Read access is public.
|
||||
Write access is controlled by tokens.
|
||||
- Store files with metadata (labels, tags, and custom fields)
|
||||
- Retrieve files by hash or label:tag combination
|
||||
- Check if a file exists by hash or label:tag
|
||||
- Delete files by hash
|
||||
- List all stored objects
|
||||
- Automatic file deduplication using content hashing
|
||||
- Support for large file uploads
|
||||
- Configurable storage location and server settings
|
||||
- Token-based authentication for write operations
|
||||
|
||||
Public read actions:
|
||||
## Building
|
||||
|
||||
### Retrieve Object
|
||||
- Objects are accessed via a label and tag, or via their hash. For example:
|
||||
- `wget http://localhost:8123/object/squashkiwi:latest`
|
||||
- `wget http://localhost:8123/object/4528400792837739857`
|
||||
|
||||
### Check Object Existence
|
||||
- Check if an object exists by label:tag or hash:
|
||||
- `curl http://localhost:8123/exists/squashkiwi:latest`
|
||||
- `curl http://localhost:8123/exists/4528400792837739857`
|
||||
|
||||
### Retrieve Hash
|
||||
- Get the hash for a given label and tag:
|
||||
- `curl http://localhost:8123/hash/squashkiwi:latest`
|
||||
- Response format: `{"result":"success","hash":"4528400792837739857"}`
|
||||
|
||||
### List Store Contents
|
||||
- Get a full list of {label:tag,hash} entries:
|
||||
- `curl http://localhost:8123/dir`
|
||||
- Response format: `{"result":"success","entries":[{"label_tag":"example:latest","hash":"4528400792837739857"}]}`
|
||||
|
||||
### Retrieve Metadata
|
||||
- Get all metadata for a tag:
|
||||
- `curl http://localhost:8123/meta/squashkiwi:latest`
|
||||
- `curl http://localhost:8123/meta/4528400792837739857`
|
||||
- Response format: `{"result":"success","metadata":{"description":"Example file","tags":["test","example"],"custom_field":"custom value"}}`
|
||||
|
||||
### Service Status Check
|
||||
- Quick status check:
|
||||
- `curl http://localhost:8123/status`
|
||||
- Response format: `{"result":"success","status":"ok"}`
|
||||
|
||||
## Write actions (require authentication):
|
||||
|
||||
### Upload Object
|
||||
- Upload a file with metadata (via HTTP PUT):
|
||||
```bash
|
||||
curl -X PUT \
|
||||
-H "Authorization: Bearer YOUR_TOKEN" \
|
||||
-F "file=@/path/to/your/file.txt" \
|
||||
-F 'metadata={"label":"example","tags":["latest","test","example"],"description":"Example file","custom_field":"custom value"}' \
|
||||
"http://localhost:8123/upload"
|
||||
mkdir build
|
||||
cd build
|
||||
cmake ..
|
||||
make
|
||||
```
|
||||
- The object file is uploaded, hashed, added to the registry (if that hash doesn't already exist), and {label:tag,hash} entries are added to the directory index.
|
||||
- Matching tags on older versions are removed.
|
||||
- Response format: `{"result":"success","hash":"4528400792837739857"}`
|
||||
|
||||
### Delete Object
|
||||
- Delete an object and all its tags:
|
||||
- `curl -H "Authorization: Bearer YOUR_TOKEN" http://localhost:8123/deleteobject?hash=4528400792837739857`
|
||||
- Response format: `{"result":"success"}`
|
||||
|
||||
## Configuration
|
||||
- The server is configured via `~/.config/simple_object_storage/config.json` which allows setting:
|
||||
- `write_tokens`: List of valid write access tokens
|
||||
- `object_store_path`: Location for the object store (path on disk)
|
||||
- `host`: Server host (default: "0.0.0.0")
|
||||
- `port`: Server port (default: 8123)
|
||||
|
||||
Example config.json:
|
||||
The server can be configured by creating a JSON configuration file at `~/.config/simple_object_storage/config.json`. Here's an example configuration:
|
||||
|
||||
```json
|
||||
{
|
||||
"write_tokens": ["your-secret-token-1", "your-secret-token-2"],
|
||||
"object_store_path": "/data/storage",
|
||||
"host": "0.0.0.0",
|
||||
"port": 8123
|
||||
"host": "localhost",
|
||||
"port": 8080,
|
||||
"storage_path": "/path/to/storage",
|
||||
"write_tokens": ["your-secret-token"]
|
||||
}
|
||||
```
|
||||
|
||||
## Signal Handling
|
||||
## API Endpoints
|
||||
|
||||
The server handles the following signals:
|
||||
### Upload a File
|
||||
|
||||
- `SIGTERM`/`SIGINT`: Gracefully shuts down the server when received (e.g. from Ctrl+C or system shutdown)
|
||||
- `SIGHUP`: Reloads the server configuration without restarting the service
|
||||
```
|
||||
PUT /upload
|
||||
```
|
||||
|
||||
The server ensures proper cleanup of resources during shutdown, including:
|
||||
- Closing all database connections
|
||||
- Stopping the HTTP server
|
||||
- Cleaning up any open file handles
|
||||
- Properly terminating worker threads
|
||||
Parameters:
|
||||
- `file`: The file to upload
|
||||
- `metadata`: JSON object containing:
|
||||
- `labels`: Array of strings (required)
|
||||
- `tags`: Array of strings (required)
|
||||
- Additional custom fields (optional)
|
||||
|
||||
Dockcross is used to cross-build for both 64-bit x86 and arm64 (combining both into one docker container image).
|
||||
Example:
|
||||
```bash
|
||||
curl -X PUT \
|
||||
-H "Authorization: Bearer your-token" \
|
||||
-F "file=@example.txt" \
|
||||
-F 'metadata={"labels":["test"],"tags":["latest"],"description":"Example file"}' \
|
||||
http://localhost:8080/upload
|
||||
```
|
||||
|
||||
### Get a File
|
||||
|
||||
```
|
||||
GET /object/{hash}
|
||||
GET /object/{label}:{tag}
|
||||
```
|
||||
|
||||
Example:
|
||||
```bash
|
||||
curl http://localhost:8080/object/abc123
|
||||
curl http://localhost:8080/object/test:latest
|
||||
```
|
||||
|
||||
### Check if a File Exists
|
||||
|
||||
```
|
||||
GET /exists/{hash}
|
||||
GET /exists/{label}:{tag}
|
||||
```
|
||||
|
||||
Example:
|
||||
```bash
|
||||
curl http://localhost:8080/exists/abc123
|
||||
curl http://localhost:8080/exists/test:latest
|
||||
```
|
||||
|
||||
### Delete a File
|
||||
|
||||
```
|
||||
GET /deleteobject?hash={hash}
|
||||
```
|
||||
|
||||
Example:
|
||||
```bash
|
||||
curl -H "Authorization: Bearer your-token" http://localhost:8080/deleteobject?hash=abc123
|
||||
```
|
||||
|
||||
### List All Objects
|
||||
|
||||
```
|
||||
GET /list
|
||||
```
|
||||
|
||||
Example:
|
||||
```bash
|
||||
curl http://localhost:8080/list
|
||||
```
|
||||
|
||||
## Database Schema
|
||||
|
||||
The system uses SQLite to store metadata about uploaded files. The database schema is as follows:
|
||||
|
||||
```sql
|
||||
CREATE TABLE objects (
|
||||
hash TEXT PRIMARY KEY,
|
||||
labels TEXT NOT NULL, -- JSON array of labels
|
||||
tags TEXT NOT NULL, -- JSON array of tags
|
||||
metadata TEXT NOT NULL -- JSON object with additional metadata
|
||||
);
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
The repository includes two test scripts:
|
||||
- `test.sh`: Basic functionality tests
|
||||
- `test_1GB_file_upload.sh`: Tests uploading and downloading a 1GB file
|
||||
|
||||
To run the tests:
|
||||
```bash
|
||||
./test.sh
|
||||
./test_1GB_file_upload.sh
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the MIT License - see the LICENSE file for details.
|
||||
|
||||
Reference in New Issue
Block a user