Tigris global object storage, now in private beta, released a new migration tool called shadow buckets.
Shadow buckets enable transparent copying and writing of objects as they are requested or uploaded. This is helpful for a few scenarios, like:
- Avoiding egress fees from large data migrations
- Avoiding downtime by serving requests from Tigris while running a data migration
- Testing Tigris global cache performance
- Testing Tigris functionality with the option to switch back to your current provider
Learn more about this in our documentation, or check out usage with
flyctl storage create --help.
This is pretty neat.
Also FYI, you linked the wrong private beta. Should be Global caching object storage on Fly.io in private beta
Just to add to what @joshua-fly wrote, here is what the migration process looks like with the shadow bucket feature.
When creating a new bucket, you can specify the bucket from where the data should be migrated. We call this the shadow bucket.
This is how the process works:
- When you upload a new object to the Tigris bucket, it is first uploaded to the shadow bucket and then copied to the Tigris bucket. This ensures that the data is always available in the shadow bucket and is eventually consistent in the Tigris bucket.
- When you request an object, we first check if the object exists in the Tigris bucket. If it does, we return it. If it doesn’t, we check the shadow bucket and return the object from there. Then, we copy the object to the Tigris bucket to make it available for future requests.
- The objects in the Tigris bucket are automatically stored in the region closest to the user.
- When you delete an object, we delete it from the Tigris bucket and the shadow bucket.
Has the issue with deleted buckets not being removed from flycyl been resolved?
Hi @atridad yes the issue is resolved. If you recently tried and running into the same issue, can you send the bucket names to the support email, I’ll look into it.
If any parameter, such as a shadow region, is wrong, an internal error will occur, and it would be nice to have a more friendly error message for that.
Thanks for the feedback. I’ll take care of it.
And other things:
- Allow users to see the usage space of the entire bucket
- Document the way to do a bulk migration, not just an incremental migration
etc. may also be a good improvement.
We are working on adding stats on the dashboard. Also, this information should be available in
flyctl storage list command, I’ll work on it. Where else do you think this information should be available?
I have asked team to document some bulk migration strategies.
For now you can use the AWS CLI to calculate the total storage used in a bucket:
- Configure the AWS CLI following AWS CLI | Tigris Object Storage Documentation
aws s3 ls --summarize --human-readable --recursive s3://my-bucket