diff --git a/CLAUDE.md b/CLAUDE.md index ba14451..eda1ac2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,8 +8,9 @@ Simple Object Server is a C++23 application that provides a REST API for storing - File storage with deduplication using content hashing - Metadata management with label:tag system - Token-based authentication for write operations -- Support for large file uploads +- Support for large file uploads (up to 6GB) - CORS and rate limiting support +- High-performance HTTP handling with Drogon framework ## Common Development Commands @@ -45,9 +46,10 @@ Simple Object Server is a C++23 application that provides a REST API for storing ### Key Components 1. **HTTP Server (src/server.cpp/hpp)** - - Uses httplib for HTTP handling - - Implements all REST endpoints + - Uses Drogon framework for high-performance HTTP handling + - Implements all REST endpoints via HttpController - Handles authentication, CORS, and rate limiting + - Supports large file uploads (up to 6GB) with efficient streaming 2. **Database Layer (src/database.cpp/hpp)** - SQLite-based metadata storage @@ -55,7 +57,8 @@ Simple Object Server is a C++23 application that provides a REST API for storing - Manages label:tag uniqueness and versioning 3. **Request Handlers** - - `put_handler.cpp/hpp`: File upload logic with deduplication + - `HttpController.cpp/hpp`: Drogon HTTP controller for route handling + - `put_handler.cpp/hpp`: File upload logic with deduplication and streaming support - `update_handler.cpp/hpp`: Metadata update functionality 4. **Utilities** @@ -73,6 +76,7 @@ Simple Object Server is a C++23 application that provides a REST API for storing ### Build System - Uses CMake with C++23 standard +- Drogon framework statically linked for high performance - Docker-based build process for consistency across platforms - Static linking for standalone executables - Multi-architecture support (linux/amd64, linux/arm64, windows/amd64) @@ -86,6 +90,7 @@ The test suite (`testing/test.sh`) covers: 4. File deletion 5. Rate limiting functionality 6. MD5 checksum verification +7. Large file upload testing (1GB files) Tests use curl for API interactions and jq for JSON parsing. @@ -97,4 +102,23 @@ The server uses JSON configuration (`sos_config.json`) with these key settings: - `cors`: CORS policy configuration - `rate_limiting`: Auth attempt limits -Default configuration is suitable for Docker deployment with storage at `/data/storage`. \ No newline at end of file +Default configuration is suitable for Docker deployment with storage at `/data/storage`. + +## Architecture Notes + +### HTTP Framework Migration + +The server has been migrated from httplib to Drogon for improved performance and scalability: + +- **Previous**: Used cpp-httplib (single-header library) +- **Current**: Uses Drogon framework with async request handling +- **Benefits**: Better performance, larger file support, thread-safe operation +- **Compatibility**: Full API compatibility maintained during migration + +### Performance Characteristics + +- **Large File Support**: Efficiently handles files up to 6GB +- **Memory Management**: Streaming file processing to minimize memory usage +- **Deduplication**: Content-based deduplication reduces storage requirements +- **Thread Safety**: Singleton pattern with proper mutex protection +- **Security**: Request size limits and rate limiting prevent abuse \ No newline at end of file diff --git a/README.md b/README.md index 3fd56b7..28943ea 100644 --- a/README.md +++ b/README.md @@ -18,9 +18,12 @@ curl https://getbin.xyz/simple-object-server-install | bash - Delete files by hash - List all stored objects - Automatic file deduplication using content hashing -- Support for large file uploads +- Support for large file uploads (up to 6GB) +- High-performance HTTP server with async request handling - Configurable storage location and server settings - Token-based authentication for write operations +- CORS support for web applications +- Rate limiting for security ## Building @@ -278,14 +281,20 @@ CREATE TABLE objects ( ## Testing -The repository includes two test scripts: -- `test.sh`: Basic functionality tests -- `test_1GB_file_upload.sh`: Tests uploading and downloading a 1GB file +The repository includes comprehensive test scripts: +- `test.sh`: Complete test suite including basic functionality, metadata preservation, tag versioning, rate limiting, and 1GB file upload tests +- `testing/test.sh`: Direct test script for integration testing +- `testing/test_1GB_file_upload.sh`: Standalone 1GB file upload test -To run the tests: +To run the full test suite: ```bash ./test.sh -./test_1GB_file_upload.sh +``` + +To run individual tests: +```bash +./testing/test.sh +./testing/test_1GB_file_upload.sh ``` ## License diff --git a/src/put_handler.cpp b/src/put_handler.cpp index 0604930..9802f34 100644 --- a/src/put_handler.cpp +++ b/src/put_handler.cpp @@ -51,7 +51,7 @@ void PutHandler::handle_put_object(const drogon::HttpRequestPtr& req, std::funct if (!contentLengthHeader.empty()) { try { size_t contentLength = std::stoull(contentLengthHeader); - const size_t MAX_UPLOAD_SIZE = 2ULL * 1024 * 1024 * 1024; // 2GB + const size_t MAX_UPLOAD_SIZE = 6ULL * 1024 * 1024 * 1024; // 6GB if (contentLength > MAX_UPLOAD_SIZE) { resp->setStatusCode(drogon::k413RequestEntityTooLarge); nlohmann::json response = {{"result", "error"}, {"error", "File too large"}}; diff --git a/src/server.cpp b/src/server.cpp index cc10561..7bf8e33 100644 --- a/src/server.cpp +++ b/src/server.cpp @@ -146,8 +146,8 @@ bool Server::start() { drogon::app().addListener(config_.host, config_.port); drogon::app().setThreadNum(16); - // Set security limits (allowing for 1GB+ files for testing) - drogon::app().setClientMaxBodySize(2ULL * 1024 * 1024 * 1024); // 2GB max body size + // Set security limits (allowing for large files up to 6GB) + drogon::app().setClientMaxBodySize(6ULL * 1024 * 1024 * 1024); // 6GB max body size drogon::app().setClientMaxMemoryBodySize(100 * 1024 * 1024); // 100MB max memory body (keep this lower) drogon::app().enableGzip(true); // Enable compression