Expected throughput between Tigris and Fly Machine

1

I have a workflow that produces up to hundreds of gigabytes of data on a Fly machine running in iad and pushes it into Tigris. In trying to tune things, I’ve found that the upload is my current bottleneck. I’ve tried a number of combinations of default.s3.max_ aws CLI configurations but the best I can get is ~100MiB/s upload (and download).

Is this the limit I should expect to see out of a single machine-Tigris connection?
I think I used to see much faster downloads in the past.

For what it’s worth, running the speedtest CLI on that machine yields 777.15 MB/s download and 110.26 MB/s upload over the external internet but I would have expected that since Tigris is hosted on Fly, the throughput would be higher.

2 Likes
2

I confirmed that disk performance isn’t at fault.
I’m getting reads and writes from/to the persistent attached volume around 900MB/s according to hdparm and dd

3

Oh, and I also restricted the bucket region to iad just to make sure things stay “local”.

4

I believe the maximum bandwidth you can get between a Fly machine and another Fly machine is 1Gbps. Because of this, you will see the maximum bandwidth restricted between a single Fly machine and Tigris hosted on Fly.

5

Would you mind running the test against this endpoint https://t3.storage.dev in iad?

6

Running AWS_ENDPOINT_URL_S3=https://t3.storage.dev aws s3 cp kz_noise_10g s3://<bucket>/kz_noise_10g_1 with default s3 configurations.
Seeing 70MiB/s upload.

7

That’s interesting that the internal network would have lower bandwidth than the external downlink, no?

1 Like
8

And BTW, the limit being 1Gbps (around the 100MiB/s I’ve been seeing) is a totally acceptable answer for me. I just want to know if there is something I can do to make things faster or if I should move on and start scaling horizontally.

9

You can scale horizontally and add more machines. Once the workload is distributed across these multiple machines, you will see higher aggregate throughput.

1 Like
10

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.