File upload to Tigris from embedded Arduino device

sensa · January 2, 2025, 8:35pm

I have a Python-based dashboard app and Tigris bucket setup. I would like to upload a small (<50kB) file from an Arduino device to the Tigris bucket. Is there recommended, straightforward way to do this. Is a HTTPS POST to the bucket from the Arduino device (in C++) possible?

Thanks!

jmj · January 3, 2025, 3:10am

if you own these devices physically and going to keep the possession of that all the time. You can deliver the access key to those devices after some authentication from your service. Once you have the credentials available on the device - You can make Signed PUT request to Tigris to upload object on to your bucket. (Here I assume you don’t want to use AWS C++ SDK to keep things light). It still uses libcurl and openssl libraries for HTTP calls and signature computation.

#include <iostream>
#include <fstream>
#include <sstream>
#include <iomanip>
#include <string>
#include <curl/curl.h>
#include <openssl/hmac.h>
#include <openssl/sha.h>
#include <openssl/evp.h>

const std::string AWS_REGION = "auto"; 
const std::string AWS_SERVICE = "s3";
const std::string AWS_ACCESS_KEY = "your-access-key";
const std::string AWS_SECRET_KEY = "your-secret-key";
const std::string BUCKET_NAME = "your-bucket-name";
const std::string OBJECT_KEY = "example.txt";
const std::string FILE_PATH = "path/to/your/file.txt"; // local file path

// Helper function to create a date string in the format required in signature computation
std::string get_current_date() {
    time_t now = time(0);
    struct tm *tstruct = gmtime(&now);
    char buf[80];
    strftime(buf, sizeof(buf), "%Y%m%dT%H%M%SZ", tstruct);
    return std::string(buf);
}

// Helper function to create a date string for the credential scope
std::string get_aws_date() {
    time_t now = time(0);
    struct tm *tstruct = gmtime(&now);
    char buf[80];
    strftime(buf, sizeof(buf), "%Y%m%d", tstruct);
    return std::string(buf);
}

// Helper function to generate an HMAC-SHA256 digest
std::string hmac_sha256(const std::string &key, const std::string &data) {
    unsigned char *result;
    unsigned int len = SHA256_DIGEST_LENGTH;
    result = HMAC(EVP_sha256(), key.c_str(), key.length(), (unsigned char*)data.c_str(), data.length(), NULL, &len);
    return std::string((char*)result, len);
}

// Function to generate AWS Signature Version 4
std::string generate_signature(const std::string &date, const std::string &credential_scope, const std::string &signed_headers, const std::string &payload_hash, const std::string &canonical_request, const std::string &request) {
    std::string secret_key = "AWS4" + AWS_SECRET_KEY;
    std::string region_key = hmac_sha256(secret_key, AWS_REGION);
    std::string service_key = hmac_sha256(region_key, AWS_SERVICE);
    std::string signing_key = hmac_sha256(service_key, "aws4_request");

    std::string string_to_sign = "AWS4-HMAC-SHA256\n" + date + "\n" + credential_scope + "\n" + payload_hash;
    std::string string_to_sign_hmac = hmac_sha256(signing_key, string_to_sign);
    return string_to_sign_hmac;
}

std::string get_signed_headers(const std::string &date, const std::string &signed_headers) {
    std::string canonical_headers = "host:s3.amazonaws.com\nx-amz-date:" + date + "\n";
    return canonical_headers;
}

int main() {
    // Get the current date and time in AWS format
    std::string date = get_current_date();
    std::string aws_date = get_aws_date();

    // Construct the canonical request
    std::string canonical_uri = "/" + OBJECT_KEY;
    std::string canonical_querystring = "";
    std::string canonical_headers = get_signed_headers(date, "host;x-amz-date");

    std::string payload_hash = "UNSIGNED-PAYLOAD";
    std::string canonical_request = "PUT\n" + canonical_uri + "\n" + canonical_querystring + "\n" + canonical_headers + "\n" + "host;x-amz-date\n" + payload_hash;

    // Calculate the signature
    std::string credential_scope = aws_date + "/" + AWS_REGION + "/" + AWS_SERVICE + "/aws4_request";
    std::string signature = generate_signature(date, credential_scope, "host;x-amz-date", payload_hash, canonical_request, "s3.amazonaws.com");

    // Build the authorization header
    std::string authorization_header = "AWS4-HMAC-SHA256 Credential=" + AWS_ACCESS_KEY + "/" + credential_scope + ", SignedHeaders=host;x-amz-date, Signature=" + signature;

    // Create the HTTP request using cURL
    CURL *curl = curl_easy_init();
    if (curl) {
        // Set the cURL options
        curl_easy_setopt(curl, CURLOPT_URL, "https://fly.storage.tigris.dev/" + BUCKET_NAME + "/" + OBJECT_KEY);
        curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");

        // Set the headers
        struct curl_slist *headers = NULL;
        headers = curl_slist_append(headers, ("Authorization: " + authorization_header).c_str());
        headers = curl_slist_append(headers, ("x-amz-date: " + date).c_str());
        headers = curl_slist_append(headers, "Content-Type: application/octet-stream");
        curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);

        // Read the file content and set it as the body of the request
        std::ifstream file(FILE_PATH, std::ios::binary);
        std::stringstream file_stream;
        file_stream << file.rdbuf();
        std::string file_content = file_stream.str();
        curl_easy_setopt(curl, CURLOPT_POSTFIELDS, file_content.c_str());

        // Perform the request
        CURLcode res = curl_easy_perform(curl);

        if (res != CURLE_OK) {
            std::cerr << "cURL error: " << curl_easy_strerror(res) << std::endl;
        } else {
            std::cout << "Successfully uploaded the file to S3!" << std::endl;
        }

        // Clean up
        curl_easy_cleanup(curl);
        curl_slist_free_all(headers);
    }

    return 0;
}

(note above code isn’t fully tested)

In case if these devices are not going to be on your possession physically and you don’t want to supply the credentials to these devices. You can analogously think your devices as HTTP browser and follow this approach Browser-Based Uploads Using HTTP POST | Tigris Object Storage Documentation

sensa · January 3, 2025, 9:55pm

Thanks for the feedback, those libraries are not available using the Arduino IDE, but I was able to get it working. This is a remotely connected IOT device that there is not physical access to. You are correct, I want to keep it light so I used a Signed PUT URL and HTTPS calls. I have a secure method for pushing updated Signed URLs to the devices, so this should work. Here is the code for reference:

#include <WiFi.h>
#include <WiFiClientSecure.h>
#include <HTTPClient.h>

const char* ssid = "<WiFiSSID>";
const char* password = "<WiFiPASS>";

const char* serverName = "<Tigris Signed PUT URL>"
const char* root_ca = 
"-----BEGIN CERTIFICATE-----\n"
<HTTPS SERVER CERT>
"-----END CERTIFICATE-----\n";


void setup() {
  Serial.begin(115200);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi...");
  }
  Serial.println("Connected to WiFi");

  sendPutRequest("THIS IS A TEST");
}

void loop() {
  // Nothing to do here
}

void sendPutRequest(const String& content) {
  if (WiFi.status() == WL_CONNECTED) {
    HTTPClient http;
    http.begin(serverName);
    http.addHeader("Content-Type", "text/plain");

    Serial.println("Sending PUT request to: " + String(serverName));
    Serial.println("Payload: " + content);

    int httpResponseCode = http.PUT(content);

    if (httpResponseCode > 0) {
      Serial.print("HTTP Response Code: ");
      Serial.println(httpResponseCode);
      String response = http.getString();
      Serial.println(response);
    } else {
      Serial.print("Error code: ");
      Serial.println(httpResponseCode);
      Serial.println(http.errorToString(httpResponseCode).c_str());
    }
    http.end();
  } else {
    Serial.println("Error in WiFi connection");
  }
}

I started with a text file in this example, but also updated the code to work with binary data.

A follow up question: The Signed PUT URL is for specific file, I see that a Signed PUT URL can be created for a bucket directory. Is it possible then to have the HTTPS PUT call create a new file and assign the filename? If so, how is the filename added to the PUT call?

jmj · January 4, 2025, 7:49am

You can do that via Signed post-policy.

sensa · January 4, 2025, 9:51pm

Thanks, I see this now. I assume this is done through CORS:

https://www.tigrisdata.com/docs/buckets/cors/

jmj · January 5, 2025, 3:53am

Here is the document for post-policy based uploads Browser-Based Uploads Using HTTP POST | Tigris Object Storage Documentation

pre-signed PUT URL based upload restricts you for the object-key and bucket and timeline where as post-policy has those restrictions but it allows you to specify key pattern or skipping the key altogether.

system · January 12, 2025, 3:54am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to sync new upload files URL to my database with Tigris? Questions / Help storage , tigris	3	99	August 24, 2024
Copy one file from AWS S3 bucket to tigris storage , tigris	4	49	November 4, 2024
Can i save multiple files to Tigris as single request? Questions / Help storage , tigris	1	20	January 11, 2025
Tigris Upload Failed to Dispatch Questions / Help storage , tigris	6	180	May 16, 2024
Upload zip file to Tigris Questions / Help storage , tigris	4	188	May 23, 2024

File upload to Tigris from embedded Arduino device

Related topics