AWS S3 Migration for Object ACL not working properly

[Eliot]

Hello, I am working on migrating all of my S3 buckets to Tigris as part of a planned, permanent migration off of AWS. Several of my buckets have a lot of object-scoped ACLs. I am starting my migration testing focusing on one of these. The documentation says that if I enable object ACLs in the Tigris bucket, object ACLs will be copied from S3 with the shadow bucket migration workflow. However, I have tried the following scenarios: Public Tigris bucket, with object ACLs enabled, and private Tigris Bucket, with object ACLs enabled. In the former case I would expect that all objects that had everyone: READ ACL enabled in S3 would also have this ACL set in Tigris, but this doesn’t seem to be the case. I am inspecting the ACLs with the Cyberduck S3 client because there doesn’t seem to be a way to inspect object ACLs in the web dashboard.

A screenshot from Cyberduck is shown below comparing the ACLs for the shadow origin S3 bucket (left) and the Tigris bucket (right):

This is with a Private Tigris bucket. When I change bucket permissions to Public and re-migrate it, all objects have everyone: READ regardless of shadow bucket origin object ACL which is also not desired.

• Your Fly org name: eliot-lash
• The name of the bucket you’re working with: eliot
• UTC timestamps for when the issue occurred: Approximately Mon Dec 02 03:30:00 2024 UTC

[Yevgeniy]

Hi, Eliot,

When migrating, explicit per object ACL is only set when it’s different from bucket setting, otherwise object inheriting Tigris bucket setting.

Currently, your bucket is configured as “private”, so objects will be migrated as private, unless public ACL is set in S3.

For example mp3 file from your screenshot (FilmSchoolSurvivalHandbook.mp3) has public read ACL in Tigris and is accessible using curl without any authentication, while the rest of the files are private.

We will further check if it’s Cyberduck not properly shows the permissions.

The other way to check object ACLs is to use AWS CLI:

export AWS_ACCESS_KEY_ID=your_access_key_id
export AWS_SECRET_ACCESS_KEY=your_secret_access_key
export AWS_ENDPOINT_URL=https://fly.storage.tigris.dev

aws s3api get-object-acl --bucket eliot --key FilmSchoolSurvivalHandbook.mp3

[Eliot]

Thanks for the reply. You are correct, the object ACL is being set correctly and is being misreported by Cyberduck. The client didn’t have an official config for Tigris so I tried making one of my own. It’s possible I made a mistake or there is some other difference in Tigris’ implementation of the S3 API causing the client to misbehave.

aws s3api get-object-acl --bucket eliot --key FilmSchoolSurvivalHandbook.mp3
{
    "Owner": {
        "DisplayName": "fadookie",
        "ID": "7119ef0da0e90142b87fc103d291f61935b2703c9b6e3ac74b00954c1a33e28a"
    },
    "Grants": [
        {
            "Grantee": {
                "DisplayName": "fadookie",
                "ID": "7119ef0da0e90142b87fc103d291f61935b2703c9b6e3ac74b00954c1a33e28a",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        },
        {
            "Grantee": {
                "Type": "Group",
                "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
            },
            "Permission": "READ"
        }
    ]
}

I was able to confirm this asset is accessible via the web endpoint at https://fly.storage.tigris.dev/eliot/FilmSchoolSurvivalHandbook.mp3

I was trying to upload my Cyberduck profile to my bucket for reference purposes but ran into another issue. Trying to set the object ACL via Cyberduck raised an error popup, so I used the AWS CLI to set a public ACL on this key and confirm it was set:

aws s3api put-object-acl --bucket eliot --key releases/software/Tigris.cyberduckprofile --acl public-read
aws s3api get-object-acl --bucket eliot --key releases/software/Tigris.cyberduckprofile
{
    "Owner": {
        "DisplayName": "fadookie",
        "ID": "7119ef0da0e90142b87fc103d291f61935b2703c9b6e3ac74b00954c1a33e28a"
    },
    "Grants": [
        {
            "Grantee": {
                "DisplayName": "fadookie",
                "ID": "7119ef0da0e90142b87fc103d291f61935b2703c9b6e3ac74b00954c1a33e28a",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        },
        {
            "Grantee": {
                "Type": "Group",
                "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
            },
            "Permission": "READ"
        }
    ]
}

However when I hit https://fly.storage.tigris.dev/eliot/releases/software/Tigris.cyberduckprofile to test downloading this asset, I get a 403:

curl -i https://fly.storage.tigris.dev/eliot/releases/software/Tigris.cyberduckprofile
HTTP/2 403
content-type: application/xml
server: Tigris OS
server-timing: total;dur=109,cache;desc=miss;dur=1, server;desc=meta;dur=12, block;desc=local;dur=7
strict-transport-security: max-age=63072000; includeSubDomains; preload
x-amz-request-id: 1733289633857533037
content-length: 305
date: Wed, 04 Dec 2024 05:20:33 GMT

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/eliot/releases/software/Tigris.cyberduckprofile</Resource><RequestId>1733289633857533037</RequestId><Key>releases/software/Tigris.cyberduckprofile</Key><BucketName>eliot</BucketName></Error>

I created a signed URL for now so you can access it:
https://eliot.fly.storage.tigris.dev/releases/software/Tigris.cyberduckprofile?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=tid_pXxffDsFiUpZCmMxDTNAdsSolBfQTbZZlYkYfzuTFwsuYm_PiU%2F20241204%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20241204T050746Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=6480a7d6c990130e09510aef953552da5c2d964ab1ca7a6f4505309575a6f9c4

[jmj]

Hi @Eliot

We have Cyberduck profile available for Tigris.

And using Cyberduck I can see the object ACL correctly being displayed

For example:

private object

public-read object

I am using version 9.0.2 (42108)

Regarding the object key releases/software/Tigris.cyberduckprofile I see it is the private object.

{
    "Owner": {
        "ID": "fly_9r265zrmdz3le0x4"
    },
    "Grants": [
        {
            "Grantee": {
                "ID": "fly_9r265zrmdz3le0x4",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        },
        {
            "Grantee": {
                "Type": "Group",
                "URI": "https://groups.tigris.dev/org/admins"
            },
            "Permission": "FULL_CONTROL"
        }
    ]
}

I am happy to help look further into it on why you saw different behavior, do I have permission to write a small object to test this on your bucket ?

[Eliot]

Thanks, yeah feel free to write to the bucket if you need to, just let me know the key(s) of any files you add so I can clean them up later. A few things:

1. I deleted my old Cyberduck profile and replaced it with the official one. It seems to work about the same.
2. I was accidentally using my old AWS S3 access key with AWS CLI using the tigris endpoint URL via the AWS_ENDPOINT_URL envvar. Somewhat surprisingly, this appeared to work, but it was setting the ACL on the object in the shadow S3 bucket rather than the Tigris bucket. Further testing indicates that I have to set the endpoint url via --endpoint-url flag or it is being ignored for some reason.
3. I tried creating a new non-admin Tigris access key with editor permissions on all buckets. (Incidentally this doesn’t seem to automatically grant permissions to buckets created after doing the permissions grant, which seems like not what it should do, but we can table that for now.)
4. Using this new key, I can view the object ACL in the tigris bucket but appear to be unable to edit it:

$ AWS_ACCESS_KEY_ID=<redacted> AWS_SECRET_ACCESS_KEY=<redacted> AWS_ENDPOINT_URL=https://fly.storage.tigris.dev aws s3api --endpoint-url https://fly.storage.tigris.dev put-object-acl --bucket eliot --key releases/software/Tigris.cyberduckprofile --acl public-read

$ AWS_ACCESS_KEY_ID=<redacted> AWS_SECRET_ACCESS_KEY=<redacted> AWS_ENDPOINT_URL=https://fly.storage.tigris.dev aws s3api --endpoint-url https://fly.storage.tigris.dev get-object-acl --bucket eliot --key releases/software/Tigris.cyberduckprofile
{
    "Owner": {
        "ID": "fly_9r265zrmdz3le0x4"
    },
    "Grants": [
        {
            "Grantee": {
                "ID": "fly_9r265zrmdz3le0x4",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        },
        {
            "Grantee": {
                "Type": "Group",
                "URI": "https://groups.tigris.dev/org/admins"
            },
            "Permission": "FULL_CONTROL"
        }
    ]
}

5. Cyberduck appears to be unable to edit object ACLs as well. I get this error popping up when trying to set the “Everyone: READ” permission on the Tigris.cyberduckprofile file:
   
   

[jmj]

Hi @Eliot,

I wrote two keys in your bucket (eliot) (tigristest.public and tigristest.private) with public-read and private (default) object ACL.

I verified they are set correctly using AWS CLI

jmj@Jigars-MacBook-Pro tigris-infra % aws s3api get-object-acl --bucket eliot --key=tigristest.private
{
    "Owner": {
        "ID": "fly_9r265zrmdz3le0x4"
    },
    "Grants": [
        {
            "Grantee": {
                "ID": "fly_9r265zrmdz3le0x4",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        },
        {
            "Grantee": {
                "Type": "Group",
                "URI": "https://groups.tigris.dev/org/admins"
            },
            "Permission": "FULL_CONTROL"
        }
    ]
}
 % aws s3api get-object-acl --bucket eliot --key=tigristest.public
{
    "Owner": {
        "ID": "fly_9r265zrmdz3le0x4"
    },
    "Grants": [
        {
            "Grantee": {
                "ID": "fly_9r265zrmdz3le0x4",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        },
        {
            "Grantee": {
                "Type": "Group",
                "URI": "https://groups.tigris.dev/org/admins"
            },
            "Permission": "FULL_CONTROL"
        },
        {
            "Grantee": {
                "Type": "Group",
                "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
            },
            "Permission": "READ"
        }
    ]
}

Here is the Cyberduck info view for both the keys

Regarding the updates of ACL via Cyberduck, Cyberduck attempts to write the entire AccessControlPolicy document for object. Tigris doesn’t support that yet. we only have canned object ACL (public-read & private (default) support.