Internal network DNS propagation and new deployments

Questions / Help
1

I have setup Cloudflare in front of our application like described in Cloudflare 525 error randomly occurs - #19 by pier, but additionally added a nginx proxy in front of my node application to serve the Cloudflare Origin Certifiate (instead of letting node handle it).

I am just conneting via the private network, for example my-node-app.internal:8080.

This works well, but whenever I deploy an update to my-node-app the app becomes unavailable for what feels like eternity.

I guess the problem is that the internal ipv6 address the my-node-app.internal points to becomes unavailable as the new container is deployed, and it takes some time until the DNS is updated with the new container ip.

Any idea how I can avoid this?

For now I just restarted the nginx app via flyctl restart but even without nginx inbetween there seems to be at least a half a minute of downtime between deployments. Is that because I currently only have a single instance deployed?

2

Can you share your nginx config? nginx resolves IPs at boot time and never again in most configs. You will need to set that as an nginx variable to get it to catch IP changes: linux - How to force nginx to resolve DNS (of a dynamic hostname) everytime when doing proxy_pass? - Server Fault

3

Sure, this is what I am using:

upstream app {

    server my-node-app.internal:8080;

}

server {

    listen 443 ssl;
    listen [::]:443 ssl;

    ssl_certificate /etc/ssl/SSL_CERT.pem;
    ssl_certificate_key /etc/ssl/SSL_KEY.key;

    ssl_client_certificate /etc/nginx/certs/cloudflare.crt;
    ssl_verify_client on;

    # Catch all @see https://nginx.org/en/docs/http/server_names.html#miscellaneous_names
    server_name _;

    location / {
        proxy_pass http://app;
        proxy_set_header Host $host;
    }

}

So trying something like this should work?

- upstream app {
-
-    server my-node-app.internal:8080;
-
-}

server {

    listen 443 ssl;
    listen [::]:443 ssl;

    ssl_certificate /etc/ssl/SSL_CERT.pem;
    ssl_certificate_key /etc/ssl/SSL_KEY.key;

    ssl_client_certificate /etc/nginx/certs/cloudflare.crt;
    ssl_verify_client on;

    # Catch all @see https://nginx.org/en/docs/http/server_names.html#miscellaneous_names
    server_name _;

    location / {
-        proxy_pass http://app;
+        set $backend "http://my-node-app.internal:8080"
+        proxy_pass $backend;
        proxy_set_header Host $host;
    }

}

It mentions a resolver must be configured as well, e.G. resolver 127.0.0.1 [::1]:5353 valid=30s; - what would I set this to on fly.io?

And any idea why I see the app briefly unavailable during deployment even if I am not using cloudflare, a nginx proxy and the internal network but just a standard fly app?

Thanks!

4

Yes I think that will work!

The deploy downtime you see through <app>.fly.dev is probably due to our slow service propagation. If you’re running behind an nginx, it probably won’t affect you.

The workaround until we get this fixed is to fly scale count 3. You can see more details here: how do fly.io deploys work? - #2 by kurt

1 Like
5

Just for completion sake the above nginx config changes is missing the correct resolver value. This one should work:

- upstream app {
-
-    server my-node-app.internal:8080;
-
-}

server {

    listen 443 ssl;
    listen [::]:443 ssl;

    ssl_certificate /etc/ssl/SSL_CERT.pem;
    ssl_certificate_key /etc/ssl/SSL_KEY.key;

    ssl_client_certificate /etc/nginx/certs/cloudflare.crt;
    ssl_verify_client on;

    # Catch all @see https://nginx.org/en/docs/http/server_names.html#miscellaneous_names
    server_name _;

+   resolver [fdaa::3] valid=5s;

    location / {
-       proxy_pass http://app;
+       set $backend "http://my-node-app.internal:8080"
+       proxy_pass $backend;
        proxy_set_header Host $host;
    }

}
2 Likes