Flycast returns PR03/PU03 ("unreachable worker host") between Fly Machines, even when target machine is started and healthy

Questions / Help
1

We’re seeing repeatable private routing failures from one Fly Machine app to another Fly Machine app via Flycast.

  • Target app is running on Fly Machines (same org, region nrt)

  • Target app is created by machines api and its network status is following

    image
    image823×489 42.3 KB

  • Target machine starts successfully and serves health checks (healthcheck with 200)

  • source machine gets 502 after ~35-36s

  • Target machine logs at same time show:

    • [PR03] could not find a good candidate within 40 attempts at load balancing

    • last error: [PU03] unreachable worker host. the host may be unhealthy. this is a Fly issue.

I’m fairly lost at this point
Has anyone seen similar behavior, and what diagnostics/checklist would you recommend to isolate whether this is service config, process-group/routing setup, or platform-side networking/proxy behavior?

2

:waving_hand: This should be fixed now, there was a problem between the two specific hosts you were (un)fortunate enough to land on. Sorry about that!

3

Thanks for the update.. yes I see it working.

Is this a type of issue that won’t reach fly status? Because I’ve been having this problem for hours and checked fly status several times.

4

Our statuspage only shows platform-wide issues (or issues affecting at least multiple nodes), this was an issue specific to one single pair of hosts. I do agree it should have still generated a warning in your app’s dashboard (we have something specifically for this kind of single-host failures), but that wasn’t triggered this time, which is a problem I am going to fix. Apologies for the inconvenience.

1 Like
5

Understood. I really appreciate the quick response thanks.