188 lines
3.6 KiB
Markdown
188 lines
3.6 KiB
Markdown
# Simple Object Storage
|
|
|
|
A simple object storage system that stores files with metadata and provides a REST API for access.
|
|
|
|
## Features
|
|
|
|
- Store files with metadata (label:tag pairs and custom fields)
|
|
- Retrieve files by hash or label:tag combination
|
|
- Check if a file exists by hash or label:tag
|
|
- Delete files by hash
|
|
- List all stored objects
|
|
- Automatic file deduplication using content hashing
|
|
- Support for large file uploads
|
|
- Configurable storage location and server settings
|
|
- Token-based authentication for write operations
|
|
|
|
## Building
|
|
|
|
```bash
|
|
mkdir build
|
|
cd build
|
|
cmake ..
|
|
make
|
|
```
|
|
|
|
## Docker
|
|
|
|
You can also run the service using Docker. Here's how:
|
|
|
|
### Building the Docker Image
|
|
|
|
```bash
|
|
docker build -t simple-object-storage .
|
|
```
|
|
|
|
### Running with Docker
|
|
|
|
```bash
|
|
docker run -d \
|
|
-p 8080:8080 \
|
|
-v /path/to/storage:/data/storage \
|
|
-v /path/to/config.json:/data/sos_config.json:ro \
|
|
--name object-storage \
|
|
simple-object-storage
|
|
```
|
|
|
|
This will:
|
|
- Map port 8080 from the container to your host
|
|
- Mount your storage directory to `/data` in the container
|
|
- Mount your config directory to the container
|
|
- Run the service in detached mode
|
|
|
|
Ensure that the `storage_path` set in the `config.json` is `/data/storage`, the path inside the container.
|
|
|
|
### Using Docker Compose
|
|
|
|
Alternatively, you can use Docker Compose. Create a `docker-compose.yml` file:
|
|
|
|
```yaml
|
|
version: '3'
|
|
services:
|
|
object-storage:
|
|
build: .
|
|
ports:
|
|
- "8080:8080"
|
|
volumes:
|
|
- /path/to/storage:/data/storage
|
|
- /path/to/config.json:/data/sos_config.json:ro
|
|
restart: unless-stopped
|
|
```
|
|
|
|
Then run:
|
|
```bash
|
|
docker-compose up -d
|
|
```
|
|
|
|
## Configuration
|
|
|
|
The server can be configured by creating a JSON configuration file at `~/.config/simple_object_storage/config.json`. Here's an example configuration:
|
|
|
|
```json
|
|
{
|
|
"host": "localhost",
|
|
"port": 8080,
|
|
"storage_path": "/path/to/storage",
|
|
"write_tokens": ["your-secret-token"]
|
|
}
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
### Upload a File
|
|
|
|
```
|
|
PUT /upload
|
|
```
|
|
|
|
Parameters:
|
|
- `file`: The file to upload
|
|
- `metadata`: JSON object containing:
|
|
- `labeltags`: Array of strings in "label:tag" format (required)
|
|
- Additional custom fields (optional)
|
|
|
|
Example:
|
|
```bash
|
|
curl -X PUT \
|
|
-H "Authorization: Bearer your-token" \
|
|
-F "file=@example.txt" \
|
|
-F 'metadata={"labeltags":["test:latest","project:alpha"],"description":"Example file"}' \
|
|
http://localhost:8080/upload
|
|
```
|
|
|
|
### Get a File
|
|
|
|
```
|
|
GET /object/{hash}
|
|
GET /object/{label}:{tag}
|
|
```
|
|
|
|
Example:
|
|
```bash
|
|
curl http://localhost:8080/object/abc123
|
|
curl http://localhost:8080/object/test:latest
|
|
```
|
|
|
|
### Check if a File Exists
|
|
|
|
```
|
|
GET /exists/{hash}
|
|
GET /exists/{label}:{tag}
|
|
```
|
|
|
|
Example:
|
|
```bash
|
|
curl http://localhost:8080/exists/abc123
|
|
curl http://localhost:8080/exists/test:latest
|
|
```
|
|
|
|
### Delete a File
|
|
|
|
```
|
|
GET /deleteobject?hash={hash}
|
|
```
|
|
|
|
Example:
|
|
```bash
|
|
curl -H "Authorization: Bearer your-token" http://localhost:8080/deleteobject?hash=abc123
|
|
```
|
|
|
|
### List All Objects
|
|
|
|
```
|
|
GET /list
|
|
```
|
|
|
|
Example:
|
|
```bash
|
|
curl http://localhost:8080/list
|
|
```
|
|
|
|
## Database Schema
|
|
|
|
The system uses SQLite to store metadata about uploaded files. The database schema is as follows:
|
|
|
|
```sql
|
|
CREATE TABLE objects (
|
|
hash TEXT PRIMARY KEY,
|
|
labeltags TEXT NOT NULL, -- JSON array of label:tag pairs
|
|
metadata TEXT NOT NULL -- JSON object with additional metadata
|
|
);
|
|
```
|
|
|
|
## Testing
|
|
|
|
The repository includes two test scripts:
|
|
- `test.sh`: Basic functionality tests
|
|
- `test_1GB_file_upload.sh`: Tests uploading and downloading a 1GB file
|
|
|
|
To run the tests:
|
|
```bash
|
|
./test.sh
|
|
./test_1GB_file_upload.sh
|
|
```
|
|
|
|
## License
|
|
|
|
This project is licensed under the MIT License - see the LICENSE file for details.
|