fly-prefer-region header not working

Questions / Help
1

I’m working on a prototype to evaluate Fly.io and so far unable to get requests with header ‘Fly-Prefer-Region: fra’ to go to that region.

My status:

❯ f status
App
  Name     = pdp-spike-fly          
  Owner    = prisma                 
  Version  = 9                      
  Status   = running                
  Hostname = pdp-spike-fly.fly.dev  

Deployment Status
  ID          = 86d50b07-c75b-0916-54d3-7fa40a5184a3         
  Version     = v9                                           
  Status      = successful                                   
  Description = Deployment completed successfully            
  Instances   = 2 desired, 2 placed, 2 healthy, 0 unhealthy  

Instances
ID              PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS   RESTARTS        CREATED   
8fc81bf3        app     9       yyz     run     running                 0               1m48s ago
44133f70        app     9       fra     run     running                 0               2m17s ago

My request:

curl https://pdp-spike-fly.fly.dev/demo/region --header 'Fly-Prefer-Region: fra'

The logs:

2022-08-01T18:57:00Z app[44133f70] fra [info]{
2022-08-01T18:57:00Z app[44133f70] fra [info]  'user-agent': 'curl/7.79.1',
2022-08-01T18:57:00Z app[44133f70] fra [info]  accept: '*/*',
2022-08-01T18:57:00Z app[44133f70] fra [info]  'x-request-start': 't=1659380220040406',
2022-08-01T18:57:00Z app[44133f70] fra [info]  host: 'pdp-spike-fly.fly.dev',
2022-08-01T18:57:00Z app[44133f70] fra [info]  'fly-client-ip': '70.80.207.113',
2022-08-01T18:57:00Z app[44133f70] fra [info]  'x-forwarded-for': '70.80.207.113, 37.16.17.20',
2022-08-01T18:57:00Z app[44133f70] fra [info]  'fly-forwarded-proto': 'https',
2022-08-01T18:57:00Z app[44133f70] fra [info]  'x-forwarded-proto': 'https',
2022-08-01T18:57:00Z app[44133f70] fra [info]  'fly-forwarded-ssl': 'on',
2022-08-01T18:57:00Z app[44133f70] fra [info]  'x-forwarded-ssl': 'on',
2022-08-01T18:57:00Z app[44133f70] fra [info]  'fly-forwarded-port': '443',
2022-08-01T18:57:00Z app[44133f70] fra [info]  'x-forwarded-port': '443',
2022-08-01T18:57:00Z app[44133f70] fra [info]  'fly-region': 'yyz',
2022-08-01T18:57:00Z app[44133f70] fra [info]  'fly-request-id': '01G9DC9G4854MVPAGMEFTRX5CM-yyz',
2022-08-01T18:57:00Z app[44133f70] fra [info]  via: '2 fly.io'
2022-08-01T18:57:00Z app[44133f70] fra [info]}

And when trying with yyz:

❯ curl https://pdp-spike-fly.fly.dev/demo/region --header 'Fly-Prefer-Region: yyz'

2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]{
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  'user-agent': 'curl/7.79.1',
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  accept: '*/*',
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  'x-request-start': 't=1659380242569037',
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  host: 'pdp-spike-fly.fly.dev',
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  'fly-client-ip': '70.80.207.113',
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  'x-forwarded-for': '70.80.207.113, 37.16.17.20',
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  'fly-forwarded-proto': 'https',
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  'x-forwarded-proto': 'https',
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  'fly-forwarded-ssl': 'on',
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  'x-forwarded-ssl': 'on',
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  'fly-forwarded-port': '443',
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  'x-forwarded-port': '443',
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  'fly-region': 'yyz',
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  'fly-request-id': '01G9DCA649P93SBV8D188QEA53-yyz',
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]  via: '2 fly.io'
2022-08-01T18:57:22Z app[8fc81bf3] yyz [info]}
2

Ah, I see that the log line prefix is changing, but the proxy header fly-region is not (it is always yyz. Is this a bug with fly or…?

3

The region in the log line is the instance’s region and the region in the header is the edge region you’re routing to. We can’t change the region you’re routing to, we can only determine what to do afterwards :slight_smile:

In other words: the fly-prefer-region header will send a request to an instance in the supplied region if it can, but you won’t connect directly to that region as we don’t control that.

1 Like
4

Ok interesting. Follow up question then. How different are these two things:

  1. Request routed to edge region X, continues with instance in region X
  2. Request routed to edge region Y, via fly-prefer-region goes to instance in region X

I guess what I’m asking for is clarification on the network hops.

5

Also just to confirm, generally speaking (maybe there are exceptions…) an application can know a request is using fly-prefer-region if it sees a discrepancy between the fly-region header and the FLY_REGION environment variable. Right?

6

They look like this:

client -> region x edge -> region x worker -> instance

client -> region y edge -> region x worker -> instance

The only difference is the hop in our internal network has extra latency unless the client is reaching region x by chance.

The fly-region header represents the edge region where the client was routed to (via BGP). Unless you have an instance in every single one of our regions and all your clients have perfect routing with us, then you can’t know if fly-prefer-region was used.

We also have some regions that are “edge-only” (but not for long). So you’d see jnb in the fly-region header and it couldn’t possible reach an instance in the same region since we don’t have “workers” there.

1 Like
7

Thanks for the info!