RTT: round trip time - yep I was equating that to latency. The VM moving around definitely seems related.
What does your CPU utilization metric look like when you’re seeing those logs?
Also, could you share the output of fly scale show? If you’re running something like micros you might want to bump up to shared-1x since there’s a bit more compute power there.
While I was writing this up I was looking more into the issue with placing your worker VM. I discovered some logic that affects region placement for multi-process apps with volumes specifically. Basically, if you have volumes we allow our volume driver to schedule VMs near your volumes. The other region logic seems to get skipped and we only rely on the volumes.
The problem with this setup you have is that you have a process with a volume and one without volume - so we don’t pin it to a region near a volume.
So there are a couple options for this.
The first option; you could create another volume and set both processes to mount volumes so it winds up pinned to a region. This is probably sub-optimal for a couple reasons; you have potentially one volume not being used, your web volume might sometimes flip to the other volume - and depending on your logic that might be a problem.
The second option - this is probably simplest - would be to create a worker app; just yourapp-worker and deploy it as a separate app that just doesn’t setup web services. This way you can use regions the normal way. If you need volumes you have the flexibility to add them when you need them on the worker etc. I do this on my own projects actually.
A third wilder option: our new version of Apps. I think you can skirt some of your issues by moving to our newer Apps platform which moves away from a lot of these weird Nomad scheduling issues that you’re experiencing right now and to our own platform. There’s a lot of active work going into these, and migration is pretty straightforward.
One complexity with migrating your app is that we haven’t fully automated the migration with volumes so if you’re interested in this option we can talk about how to deal with your volume to switch over. There’s some more information here: Sneak peek: docs for Apps V2 (help us improve them!) and in the links within if you’re curious about that.