I’ve been looking at Fly.io as a potentially good platform for an add-on product we are developing.
I’d like make sure it’s a good match - as we’ve moved further in our prototyping, the number of components and complexity have increased.
The system will handle a large number of events (oidc log-in/log out) and potentially keep long-lived connections to 10K+ end-points per customer (it’s going to be multi-tenant) for push notifications/event delivery. From the architectural stand-point, it’ll be a number of GO based micro-services (each one being fairly simple), communicating internally via NATS and exposed externally via GRPC/Rest/Webhooks. A reverse proxy (either HA-Proxy or NGINX, possibly Traefik) will need to run (per region?) to handle connections/routing.
Is this still something that is a good match for Fly.io or is it too complex? We do run our main services in a self-managed K8S environment, it is however not regionally/geographically distributed.
That should work really well! You might not even need the reverse proxy, our proxy will route people to the nearest instance of an app.
Assuming those 10k+ connections are geographically distributed, this is almost the perfect use case for Fly.io.
We know of a few people who’ve been playing with embedding NATS into go binaries. It seems to work well, and might be a nice way to decrease the number of moving parts.
Just to clarify the idea behind the reverse proxy - it’s more to route to a correct micro-service and use internal port that is not exposed externally (e.g. 2 separate grpc back-ends exposed via 1 external port/url), handle blue/green deployment, those type of things.
BTW, how would it work with the internal DNS? Will we need to specify regions as part of the the downstream url in the reverse proxy or will the internal DNS handle that automatically, based on the source address of the reverse proxy (assuming we deploy one per region).
Are you asking how the internal DNS works cross region?
When you query <appname>.internal it usually returns the oldest record first, it doesn’t do any sorting based on latency (yet). If you’re trying to connect to an instance in the same region, you will want to use <region>.<appname>.internal for now.