public/simple-object-server
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

1GB test updated.

Browse Source
This commit is contained in:
Your Name
2025-05-25 11:37:43 +12:00
parent 52dcaada2d
commit ed9725208d
2 changed files with 72 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

52
README.md
View File

@@ -12,52 +12,70 @@ Write access is controlled by tokens.
Public read actions:
### Retrieve Object
- Objects are access via a label and tag, or via their hash. For example:
- Objects are accessed via a label and tag, or via their hash. For example:
- `wget http://localhost:8123/object/squashkiwi:latest`
- `wget http://localhost:8123/object/4528400792837739857`
### Check Object Existence
- Check if an object exists by label:tag or hash:
- `curl http://localhost:8123/exists/squashkiwi:latest`
- `curl http://localhost:8123/exists/4528400792837739857`
### Retrieve Hash
- You can retrieve the hash for a given label and tag with, e.g.:
- Get the hash for a given label and tag:
- `curl http://localhost:8123/hash/squashkiwi:latest`
- Response format: `{"result":"success","hash":"4528400792837739857"}`
### List Store Contents
- you can get a full list of {label:tag,hash} entries (one tag per entry) with:
- Get a full list of {label:tag,hash} entries:
- `curl http://localhost:8123/dir`
- Response format: `{"result":"success","entries":[{"label_tag":"example:latest","hash":"4528400792837739857"}]}`
### Retrieve Metadata
- get all metadata for a tag:
- Get all metadata for a tag:
- `curl http://localhost:8123/meta/squashkiwi:latest`
- Response format: `{"result":"success","metadata":{"description":"Example file","tags":["test","example"],"custom_field":"custom value"}}`
### Service Status Check
- quick status check:
- `curl http://localhost:8123/status`
Write actions:
- Quick status check:
- `curl http://localhost:8123/status`
- Response format: `{"result":"success","status":"ok"}`
## Write actions (require authentication):
### Upload Object
- to upload a file (via http put)
```
- Upload a file with metadata (via HTTP PUT):
```bash
curl -X PUT \
-H "Authorization: Bearer YOUR_TOKEN" \
-F "file=@/path/to/your/file.txt" \
-F 'metadata={"labeltag":"example:latest","description":"Example file","tags":["test","example"],"custom_field":"custom value"}' \
"http://localhost:8123/upload"
```
- the object_file is uploaded, hashed, added to the registry (if that hash doesn't already exist), and {label:tag,hash} entries are added to the directory index.
- matching tags on older versions are removed.
- The object file is uploaded, hashed, added to the registry (if that hash doesn't already exist), and {label:tag,hash} entries are added to the directory index.
- Matching tags on older versions are removed.
- Response format: `{"result":"success","hash":"4528400792837739857"}`
### Delete Object
- to delete an object (and all tags on that object):
- `curl http://localhost:8123/deleteobject?token="WRITE_TOKEN"\&hash="HASH"`
- Delete an object and all its tags:
- `curl -H "Authorization: Bearer YOUR_TOKEN" http://localhost:8123/deleteobject?hash=4528400792837739857`
- Response format: `{"result":"success"}`
## Configuration
- the server is configured via ~/.config/simple_object_storage/config.json which allows setting:
- the list of write access tokens
- the location for the object store (path on disk)
- The server is configured via `~/.config/simple_object_storage/config.json` which allows setting:
- `write_tokens`: List of valid write access tokens
- `object_store_path`: Location for the object store (path on disk)
- `host`: Server host (default: "0.0.0.0")
- `port`: Server port (default: 8123)
Example config.json:
```json
{
"write_tokens": ["your-secret-token-1", "your-secret-token-2"],
"object_store_path": "/data/storage",
"host": "0.0.0.0",
"port": 8123
}
```
Dockcross is used to cross-build for both 64-bit x86 and arm64 (combining both into one docker container image).

46
test_1GB_file_upload.sh
View File

@@ -1,9 +1,13 @@
#!/bin/bash
# Create a temporary test file (100MB)
# Create a temporary test file (1024MB!)
echo "Creating test file..."
dd if=/dev/urandom of=test_file.bin bs=1M count=1024
# Calculate original file hash
echo "Calculating original file hash..."
ORIGINAL_HASH=$(sha256sum test_file.bin | cut -d' ' -f1)
echo "Original hash: $ORIGINAL_HASH"
# read ~/.config/simple_object_storage/config.json
CONFIG_PATH="${HOME}/.config/simple_object_storage/config.json"
@@ -20,14 +24,13 @@ PORT=$(echo $CONFIG | jq -r '.port')
# extract the first write token from the config
WRITE_TOKEN=$(echo $CONFIG | jq -r '.write_tokens[0]')
# Get a write token (assuming a write token 'test_token' is configured)
TOKEN=${WRITE_TOKEN}
# Upload the file
echo "Uploading file..."
RESPONSE=$(curl -X PUT -T test_file.bin "http://${HOST}:${PORT}/upload?token=${TOKEN}&labeltag=test:latest&filename=test_file.bin")
RESPONSE=$(curl -X PUT \
-H "Authorization: Bearer ${WRITE_TOKEN}" \
-F "file=@test_file.bin" \
-F 'metadata={"labeltag":"test:latest","description":"Test file","tags":["test","large"]}' \
"http://${HOST}:${PORT}/upload")
echo "Upload response: $RESPONSE"
# Extract the hash from the response
@@ -44,13 +47,38 @@ echo "Checking if file exists by label:tag..."
EXISTS_TAG=$(curl "http://${HOST}:${PORT}/exists/test:latest")
echo "Exists by label:tag response: $EXISTS_TAG"
# Download the file by hash
echo "Downloading file by hash..."
curl -o downloaded_by_hash.bin "http://${HOST}:${PORT}/object/$HASH"
# Download the file by label:tag
echo "Downloading file by label:tag..."
curl -o downloaded_by_tag.bin "http://${HOST}:${PORT}/object/test:latest"
# Verify downloaded files
echo "Verifying downloaded files..."
HASH_BY_HASH=$(sha256sum downloaded_by_hash.bin | cut -d' ' -f1)
HASH_BY_TAG=$(sha256sum downloaded_by_tag.bin | cut -d' ' -f1)
echo "Original hash: $ORIGINAL_HASH"
echo "Hash of file downloaded by hash: $HASH_BY_HASH"
echo "Hash of file downloaded by tag: $HASH_BY_TAG"
if [ "$ORIGINAL_HASH" = "$HASH_BY_HASH" ] && [ "$ORIGINAL_HASH" = "$HASH_BY_TAG" ]; then
echo "Hash verification successful!"
else
echo "Hash verification failed!"
exit 1
fi
# delete the file!
echo "Deleting file..."
DELETE_RESPONSE=$(curl "http://${HOST}:${PORT}/deleteobject?token=${TOKEN}&hash=${HASH}")
DELETE_RESPONSE=$(curl -H "Authorization: Bearer ${WRITE_TOKEN}" \
"http://${HOST}:${PORT}/deleteobject?hash=${HASH}")
echo "Delete response: $DELETE_RESPONSE"
# Clean up
echo "Cleaning up..."
rm test_file.bin
rm test_file.bin downloaded_by_hash.bin downloaded_by_tag.bin
echo "Test completed."