Support for multipart presigned URLs

,
1

Hi there,

A while ago I asked if Tigris has support for multipart presigned urls. At that time @himank replied that Tigris did not yet support this feature, but that it would be added to the backlog. Unfortunately the thread is auto-closed so I won’t receive updates on the topic anymore.

So my question is:

  • Is this feature already implemented?
  • If so: is this already updated in the documentation?
  • If not: is there a timeline?

Thanks in advance.

2

Hi @hakoptak - it will be available on or before June 28.

1 Like
3

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

5

Hi @hakoptak - To close out the thread here. This is available on Tigris. I forgot to update here back.

6

Hi @jmj, thanks for the update!

Just checking: Does the multipart upload path also support CNAME buckets? Or is this not yet implemented?

Keep up the good work :flexed_biceps:

1 Like
7

Hi @hakoptak

Yes CNAME flavored pre-signed URLs are supported for MultiPart too.

1 Like
8

Hi @jmj,

I just checked if I could do a multipart upload with presigned urls and it works, provided that I use the full path. That is, a url of the form:

https://fly.storage.tigris.dev/test.sessions.pinggg.cloud/dZQGeQeOKbDhAreAMirIXfNfdfqYIQFxIdeuAqefGojmA/shares/1.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=tid_mNpmYFuMLHfstgsbJEnjDXQLjRSgqXlznLCJdlY_ZIhereWHco%2F20250717%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20250717T124256Z&X-Amz-Expires=15&X-Amz-SignedHeaders=host&partNumber=5&uploadId=f25b464f-a20f-5c81-9e54-ae2cda653887&X-Amz-Signature=69c07950269343f753bb469869c00e81edfc75371cf7ecc4e04a6e3b4eaa4707

However, when I omit the fly.storage.tigris.dev part (because I use a CNAMEd bucket) then I get the error:

<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Resource>/dZQGeQeOKbDhAreAMirIXfNfdfqYIQFxIdeuAqefGojmA/shares/1.pdf</Resource><RequestId>1752756177006561128</RequestId><Key>dZQGeQeOKbDhAreAMirIXfNfdfqYIQFxIdeuAqefGojmA/shares/1.pdf</Key><BucketName>test.sessions.pinggg.cloud</BucketName>
</Error>

Is this normal S3-compatible behaviour?

I expected that I could upload parts similar to a single presigned upload (where I can omit the fly.storage.tigris.dev)…

9

Hi @hakoptak

What is your language you use to generate mp-presigned URL. I can probably provide you code that generates MP presigned URL with the CNAME.

10

Hi @jmj, I am using Golang for the signing logic, but I don’t use any existing package (to have more control over the url parts for some optimization).

By the way: I get an error now when I try to update my bucket settings:

Failed
Failed to verify CNAME for bucket domain

This was working properly before…

11

Hi @hakoptak

We had a configurational bug in portion of our deployment which is now fixed and this error should not persist anymore.

re: custom cname flavored mp-presigned url in Go can you try with this code once

package main

import (
	"bytes"
	"context"
	"fmt"
	"log"
	"math/rand"
	"net/http"
	"strings"
	"time"

	"github.com/aws/aws-sdk-go-v2/config"
	credentials2 "github.com/aws/aws-sdk-go-v2/credentials"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"github.com/aws/aws-sdk-go-v2/service/s3/types"
	"github.com/aws/aws-sdk-go/aws"
)

func main_disabled() {
	bucket := "<your-bucket-name>"
	key := "mp3"
	numParts := 3 // how many parts you're uploading

	ctx := context.Background()

	cfg, err := config.LoadDefaultConfig(
		context.TODO(),
		config.WithRegion("auto"),
		config.WithCredentialsProvider(credentials2.NewStaticCredentialsProvider("tid_<>", "tsec_<>", "")),
	)
	if err != nil {
		log.Fatalf("unable to load SDK config, %v", err)
	}

	cfg.BaseEndpoint = aws.String("<your custom domain>") // for example https://jmj-test.tigrisdata.cloud
	s3Client := s3.NewFromConfig(cfg, func(o *s3.Options) {
		o.UsePathStyle = true
	})

	// Step 1: Initiate multipart upload
	initOutput, err := s3Client.CreateMultipartUpload(ctx, &s3.CreateMultipartUploadInput{
		Bucket: &bucket,
		Key:    &key,
	})
	if err != nil {
		log.Fatalf("failed to initiate multipart upload: %v", err)
	}
	uploadID := initOutput.UploadId
	fmt.Printf("UploadID: %s\n", uploadID)

	// Step 2: Generate presigned URLs for each part
	presigner := s3.NewPresignClient(s3Client)
	completedParts := make([]types.CompletedPart, 3)

	for partNum := 1; partNum <= numParts; partNum++ {
		presignedPart, err := presigner.PresignUploadPart(ctx, &s3.UploadPartInput{
			Bucket:     &bucket,
			Key:        &key,
			UploadId:   uploadID,
			PartNumber: aws.Int32(int32(partNum)),
		}, s3.WithPresignExpires(1*time.Hour))

		if err != nil {
			log.Fatalf("failed to generate presigned URL for part %d: %v", partNum, err)
		}

		fmt.Printf("Presigned URL for part %d: %s\n", partNum, presignedPart.URL)

		// generate 5 MB payload
		objBody := make([]byte, 5*1024*1024)
		for i := range objBody {
			objBody[i] = byte(rand.Intn(256))
		}

		// perform HTTP put to upload the part
		_, etag, err := putPart(presignedPart.URL, objBody)
		completedParts[partNum-1] = types.CompletedPart{
			ETag:       aws.String(etag),
			PartNumber: aws.Int32(int32(partNum)),
		}
	}

	// After the client uploads each part, you'll collect the ETags and then:
	completeMpUploadRes, err := s3Client.CompleteMultipartUpload(ctx, &s3.CompleteMultipartUploadInput{
		Bucket:          aws.String(bucket),
		Key:             aws.String(key),
		UploadId:        uploadID,
		MultipartUpload: &types.CompletedMultipartUpload{Parts: completedParts},
	})
	fmt.Println(completeMpUploadRes)
}

func putPart(url string, data []byte) (int, string, error) {
	req, err := http.NewRequest("PUT", url, bytes.NewReader(data))
	if err != nil {
		return 0, "", fmt.Errorf("failed to create request: %w", err)
	}

	req.Header.Set("Content-Type", "application/octet-stream")

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		return 0, "", fmt.Errorf("PUT request failed: %w", err)
	}
	defer resp.Body.Close()

	etag := resp.Header.Get("ETag")
	etag = strings.ReplaceAll(etag, "\"", "")
	return resp.StatusCode, etag, nil
}
12

Hi @jmj, thanks again for following up. I can confirm that the error has been solved.

Thanks for pointing at UsePathStyle = true. I will implement that later.

13

Hi @jmj, sorry for asking a follow-up question:

During my testing with multipart uploads I find that my dashboard shows a number of objects and size, but there are no objects in that bucket visible. So I guess that these are stale upload parts my system did not complete or abort properly.

Does Tigris offers a way to automatically purge stale uploads after a certain period? It would be great to have this feature and that we could configure this period.

Thanks again!

14

@hakoptak Unfortunately we don’t have right now any way to setup a lifecycle to purge uncommitted uploads. We will soon be having something like this available but as we don’t allow configuring it we also don’t charge for these uncommitted uploads.

15

Hi @himank, thanks for your answer. Could you let me know when you have implemented the purge mechanism? Then I can test is.

And would it be possible to update the docs with the purge working details (e.g. duration) by then?

Thanks again.

16

@hakoptak Yes, I’ll definitely let you know but it will take some time before we get to it but I’ll keep you posted.

17

Hi @himank, thanks for letting me know.

18

Hello @himank, I see that the orphan or stale objects and storage have dissapeared. Does this mean that you’ve already implemented the purge mechanism? Else: what could have caused it?