139 lines
2.6 KiB
Markdown
139 lines
2.6 KiB
Markdown
# Simple Object Storage
|
|
|
|
A simple object storage system that stores files with metadata and provides a REST API for access.
|
|
|
|
## Features
|
|
|
|
- Store files with metadata (labels, tags, and custom fields)
|
|
- Retrieve files by hash or label:tag combination
|
|
- Check if a file exists by hash or label:tag
|
|
- Delete files by hash
|
|
- List all stored objects
|
|
- Automatic file deduplication using content hashing
|
|
- Support for large file uploads
|
|
- Configurable storage location and server settings
|
|
- Token-based authentication for write operations
|
|
|
|
## Building
|
|
|
|
```bash
|
|
mkdir build
|
|
cd build
|
|
cmake ..
|
|
make
|
|
```
|
|
|
|
## Configuration
|
|
|
|
The server can be configured by creating a JSON configuration file at `~/.config/simple_object_storage/config.json`. Here's an example configuration:
|
|
|
|
```json
|
|
{
|
|
"host": "localhost",
|
|
"port": 8080,
|
|
"storage_path": "/path/to/storage",
|
|
"write_tokens": ["your-secret-token"]
|
|
}
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
### Upload a File
|
|
|
|
```
|
|
PUT /upload
|
|
```
|
|
|
|
Parameters:
|
|
- `file`: The file to upload
|
|
- `metadata`: JSON object containing:
|
|
- `labels`: Array of strings (required)
|
|
- `tags`: Array of strings (required)
|
|
- Additional custom fields (optional)
|
|
|
|
Example:
|
|
```bash
|
|
curl -X PUT \
|
|
-H "Authorization: Bearer your-token" \
|
|
-F "file=@example.txt" \
|
|
-F 'metadata={"labels":["test"],"tags":["latest"],"description":"Example file"}' \
|
|
http://localhost:8080/upload
|
|
```
|
|
|
|
### Get a File
|
|
|
|
```
|
|
GET /object/{hash}
|
|
GET /object/{label}:{tag}
|
|
```
|
|
|
|
Example:
|
|
```bash
|
|
curl http://localhost:8080/object/abc123
|
|
curl http://localhost:8080/object/test:latest
|
|
```
|
|
|
|
### Check if a File Exists
|
|
|
|
```
|
|
GET /exists/{hash}
|
|
GET /exists/{label}:{tag}
|
|
```
|
|
|
|
Example:
|
|
```bash
|
|
curl http://localhost:8080/exists/abc123
|
|
curl http://localhost:8080/exists/test:latest
|
|
```
|
|
|
|
### Delete a File
|
|
|
|
```
|
|
GET /deleteobject?hash={hash}
|
|
```
|
|
|
|
Example:
|
|
```bash
|
|
curl -H "Authorization: Bearer your-token" http://localhost:8080/deleteobject?hash=abc123
|
|
```
|
|
|
|
### List All Objects
|
|
|
|
```
|
|
GET /list
|
|
```
|
|
|
|
Example:
|
|
```bash
|
|
curl http://localhost:8080/list
|
|
```
|
|
|
|
## Database Schema
|
|
|
|
The system uses SQLite to store metadata about uploaded files. The database schema is as follows:
|
|
|
|
```sql
|
|
CREATE TABLE objects (
|
|
hash TEXT PRIMARY KEY,
|
|
labels TEXT NOT NULL, -- JSON array of labels
|
|
tags TEXT NOT NULL, -- JSON array of tags
|
|
metadata TEXT NOT NULL -- JSON object with additional metadata
|
|
);
|
|
```
|
|
|
|
## Testing
|
|
|
|
The repository includes two test scripts:
|
|
- `test.sh`: Basic functionality tests
|
|
- `test_1GB_file_upload.sh`: Tests uploading and downloading a 1GB file
|
|
|
|
To run the tests:
|
|
```bash
|
|
./test.sh
|
|
./test_1GB_file_upload.sh
|
|
```
|
|
|
|
## License
|
|
|
|
This project is licensed under the MIT License - see the LICENSE file for details.
|