Specify instance-id in fly-replay header

Questions / Help
1

Hi,

All mentions of the fly-replay header I’ve seen in the docs are with region=xxx, so I was wondering, given that fly-force-instance-id exists, if it’s possible to say Fly-Replay: region=xxx,instance-id=somevmid or something like it?

I found these threads. They’re close to what I’m trying to do but not quite.

Thanks in advance.

1 Like
2

Hey - right now this isn’t possible. As we roll out a newer version of our platform, this sort of thing will become easier and we can revisit it.

What does your app want to do?

3

Got it, thank you @jsierles ! I’ll be watching.

As for the details of my setup, I was trying to have 2 app instances, both running the same sqlite-backed web service. One uses a live read-replica made with Litestream. When the instance with the replica receives a request with read operations an error is thrown and fly-replay could kick in so it gets sent to the instance that can write. This recipe works if the instances are in different regions, however, I was hoping to remove the region limitation by targeting a specific instance in the fly-replay response header.

Thanks again.

4

Nice use case! We’re working on improving the SQLite experience on Fly. Meanwhile, would it be detrimental to your app to run the other instance in a nearby region?

5

Sure. Other instance in a nearby region works, that’s what I’ll do. When the newer version makes this possible it will be real nice to not have this constraint, that’s cool!

FWIW, your product is the reason I’ve been procrastinating on my day job the last two days. That’s a compliment for the team :laughing:

Thank you!

1 Like
6

Is running the primary on one port and secondaries (read replicas) on another an option? If so, traffic sent to that port should always end up with primary?

# litestream.primary.yml
addr: ":7770"
dbs:
  - path: /to/dB


# litestream.replica.yml
addr: ":7771"
dbs:
  - path: /to/db
    upstream:
      url: http://$PRIMARY_HOSTNAME:7770

ref: github/fly-apps/litestream-base.

7

@ignoramous I’m afraid not. I could be wrong but I believe that Litestream’s HTTP endpoint is for metrics and for the replicas to catch up on the last WAL segments to be applied. In my use case we’d need the Fly.io instance that had the sqlite file open in write-mode to process the requests that need to write, and it can’t be over the network.

Thank you!

1 Like
8

Good news. This isn’t documented well (and by “not documented” I mean I had to go look at the source code), but fly-replay: instance=00bb33ff should replay to a specific instance.

4 Likes
9

How to fetch instance ids? Is FLY_ALLOC_ID it? Are instance-ids exposed via internal DNS on fdaa::3 (something similar to debug.fly.dev: UDP timeouts in Europe? - #9 by thomas)?

10

Attempting to answer my own question here: The first 4 bytes (d8899331) from FLY_ALLOC_ID is allegedly instance-id too.

➜  flyctl status --all -a <appname>                                                                           
App
  Name     = <appname>          
  Owner    = <owner>           
  Version  = <1>               
  Status   = running           
  Hostname = <appname>.fly.dev  

Instances
ID      	PROCESS	VERSION	REGION	DESIRED	STATUS 
d8899331	app    	<1>    	<r>   	run    	running

-

➜  flyctl ssh console -a <appname>
Connecting to top1.nearest.of.<appname>.internal... complete
# echo $FLY_ALLOC_ID
d8899331-7771-8881-9991-111111111112
note-to-self:

Instead of fly-replay: instance=d8899331, the client can fly-force-instance-id http-header (works only if Fly http handler is able to see and act on the incoming http-header): Is it possibly for a client to connect to a particular instance - #4 by jerome

Otherwise, forwarding requests over Fly’s 6pn (“internal services” listening on fly-local-6pn or :: or _local_ip.internal Private networking not working - #4 by kurt) to the primary instance at <fly-alloc-id>.vm.appname.internal should work just as well: Send request to a specific VM - #2 by greg

4 Likes