diff --git a/README.md b/README.md index f260d6b..a5ab022 100644 --- a/README.md +++ b/README.md @@ -12,52 +12,70 @@ Write access is controlled by tokens. Public read actions: ### Retrieve Object -- Objects are access via a label and tag, or via their hash. For example: +- Objects are accessed via a label and tag, or via their hash. For example: - `wget http://localhost:8123/object/squashkiwi:latest` - `wget http://localhost:8123/object/4528400792837739857` +### Check Object Existence +- Check if an object exists by label:tag or hash: - `curl http://localhost:8123/exists/squashkiwi:latest` - `curl http://localhost:8123/exists/4528400792837739857` ### Retrieve Hash -- You can retrieve the hash for a given label and tag with, e.g.: +- Get the hash for a given label and tag: - `curl http://localhost:8123/hash/squashkiwi:latest` + - Response format: `{"result":"success","hash":"4528400792837739857"}` ### List Store Contents -- you can get a full list of {label:tag,hash} entries (one tag per entry) with: +- Get a full list of {label:tag,hash} entries: - `curl http://localhost:8123/dir` + - Response format: `{"result":"success","entries":[{"label_tag":"example:latest","hash":"4528400792837739857"}]}` ### Retrieve Metadata -- get all metadata for a tag: +- Get all metadata for a tag: - `curl http://localhost:8123/meta/squashkiwi:latest` + - Response format: `{"result":"success","metadata":{"description":"Example file","tags":["test","example"],"custom_field":"custom value"}}` ### Service Status Check -- quick status check: - - `curl http://localhost:8123/status` - -Write actions: +- Quick status check: + - `curl http://localhost:8123/status` + - Response format: `{"result":"success","status":"ok"}` +## Write actions (require authentication): ### Upload Object -- to upload a file (via http put) -``` +- Upload a file with metadata (via HTTP PUT): +```bash curl -X PUT \ -H "Authorization: Bearer YOUR_TOKEN" \ -F "file=@/path/to/your/file.txt" \ -F 'metadata={"labeltag":"example:latest","description":"Example file","tags":["test","example"],"custom_field":"custom value"}' \ "http://localhost:8123/upload" ``` - - the object_file is uploaded, hashed, added to the registry (if that hash doesn't already exist), and {label:tag,hash} entries are added to the directory index. - - matching tags on older versions are removed. + - The object file is uploaded, hashed, added to the registry (if that hash doesn't already exist), and {label:tag,hash} entries are added to the directory index. + - Matching tags on older versions are removed. + - Response format: `{"result":"success","hash":"4528400792837739857"}` ### Delete Object -- to delete an object (and all tags on that object): - - `curl http://localhost:8123/deleteobject?token="WRITE_TOKEN"\&hash="HASH"` +- Delete an object and all its tags: + - `curl -H "Authorization: Bearer YOUR_TOKEN" http://localhost:8123/deleteobject?hash=4528400792837739857` + - Response format: `{"result":"success"}` ## Configuration -- the server is configured via ~/.config/simple_object_storage/config.json which allows setting: - - the list of write access tokens - - the location for the object store (path on disk) +- The server is configured via `~/.config/simple_object_storage/config.json` which allows setting: + - `write_tokens`: List of valid write access tokens + - `object_store_path`: Location for the object store (path on disk) + - `host`: Server host (default: "0.0.0.0") + - `port`: Server port (default: 8123) +Example config.json: +```json +{ + "write_tokens": ["your-secret-token-1", "your-secret-token-2"], + "object_store_path": "/data/storage", + "host": "0.0.0.0", + "port": 8123 +} +``` Dockcross is used to cross-build for both 64-bit x86 and arm64 (combining both into one docker container image). diff --git a/test_1GB_file_upload.sh b/test_1GB_file_upload.sh index 8e4fa45..71b8c4a 100755 --- a/test_1GB_file_upload.sh +++ b/test_1GB_file_upload.sh @@ -1,9 +1,13 @@ #!/bin/bash -# Create a temporary test file (100MB) +# Create a temporary test file (1024MB!) echo "Creating test file..." dd if=/dev/urandom of=test_file.bin bs=1M count=1024 +# Calculate original file hash +echo "Calculating original file hash..." +ORIGINAL_HASH=$(sha256sum test_file.bin | cut -d' ' -f1) +echo "Original hash: $ORIGINAL_HASH" # read ~/.config/simple_object_storage/config.json CONFIG_PATH="${HOME}/.config/simple_object_storage/config.json" @@ -20,14 +24,13 @@ PORT=$(echo $CONFIG | jq -r '.port') # extract the first write token from the config WRITE_TOKEN=$(echo $CONFIG | jq -r '.write_tokens[0]') - - -# Get a write token (assuming a write token 'test_token' is configured) -TOKEN=${WRITE_TOKEN} - # Upload the file echo "Uploading file..." -RESPONSE=$(curl -X PUT -T test_file.bin "http://${HOST}:${PORT}/upload?token=${TOKEN}&labeltag=test:latest&filename=test_file.bin") +RESPONSE=$(curl -X PUT \ + -H "Authorization: Bearer ${WRITE_TOKEN}" \ + -F "file=@test_file.bin" \ + -F 'metadata={"labeltag":"test:latest","description":"Test file","tags":["test","large"]}' \ + "http://${HOST}:${PORT}/upload") echo "Upload response: $RESPONSE" # Extract the hash from the response @@ -44,13 +47,38 @@ echo "Checking if file exists by label:tag..." EXISTS_TAG=$(curl "http://${HOST}:${PORT}/exists/test:latest") echo "Exists by label:tag response: $EXISTS_TAG" +# Download the file by hash +echo "Downloading file by hash..." +curl -o downloaded_by_hash.bin "http://${HOST}:${PORT}/object/$HASH" + +# Download the file by label:tag +echo "Downloading file by label:tag..." +curl -o downloaded_by_tag.bin "http://${HOST}:${PORT}/object/test:latest" + +# Verify downloaded files +echo "Verifying downloaded files..." +HASH_BY_HASH=$(sha256sum downloaded_by_hash.bin | cut -d' ' -f1) +HASH_BY_TAG=$(sha256sum downloaded_by_tag.bin | cut -d' ' -f1) + +echo "Original hash: $ORIGINAL_HASH" +echo "Hash of file downloaded by hash: $HASH_BY_HASH" +echo "Hash of file downloaded by tag: $HASH_BY_TAG" + +if [ "$ORIGINAL_HASH" = "$HASH_BY_HASH" ] && [ "$ORIGINAL_HASH" = "$HASH_BY_TAG" ]; then + echo "Hash verification successful!" +else + echo "Hash verification failed!" + exit 1 +fi + # delete the file! echo "Deleting file..." -DELETE_RESPONSE=$(curl "http://${HOST}:${PORT}/deleteobject?token=${TOKEN}&hash=${HASH}") +DELETE_RESPONSE=$(curl -H "Authorization: Bearer ${WRITE_TOKEN}" \ + "http://${HOST}:${PORT}/deleteobject?hash=${HASH}") echo "Delete response: $DELETE_RESPONSE" # Clean up echo "Cleaning up..." -rm test_file.bin +rm test_file.bin downloaded_by_hash.bin downloaded_by_tag.bin echo "Test completed." \ No newline at end of file