So I’ve noticed that when I redeploy a prebuilt image, just basically redeploying and nothing changed in the code, the fly app doesn’t show any signals that it restarted (from the log), and usually, I notice some odd problems, for instance, the request took much longer to come and be served by the application. It does create a new release though.
To solve this problem, I have to perform a complete redeployment, basically building a new docker image and deploying that image.
More info: my app is a rails application, bookforward-prod-us if that helps.
Observing a bit more, I think it’s some network/routing issue, similar to the one I reported a while back Network issue from Fly?. The symptom is the request is queued somewhere and couldn’t reach the application.
Hmm, another finding, which contradicts what I said above. I did a Rails.cache.clear and everything is back to normal. Don’t really know the connection between the network issue and cache
Hi, sorry to hear you’re running into this. There’s a couple things we might want to do off the bat to learn more about what’s going on.
some general info about your app-- the output of fly status -a appname will help us narrow things down a bit
looking at the debug logs for the strange deployment, eg, LOG_LEVEL=debug fly deploy; this will help us figure out what flyctl is instructing our api to do to your app.
running a curl against debug.fly.dev and a curl against your app with the flyio: doit header set. This will allow you to compare and contrast the traffic that’s being sent to your app’s instances vs. the debug endpoint.
I hope this helps! Feel free to reach back out with any questions you might have during your investigations.
awesome, thank you! I’m glad you replied to confirm because I just realized I got the custom header wrong for curl: it’s actually flyio-debug: doit and you can see an example here:
App
Name = bookforward-prod-us
Owner = personal
Version = 95
Status = running
Hostname = bookforward-prod-us.fly.dev
Deployment Status
ID = b09947fb-4214-782a-6d6f-63d3ba88569d
Version = v95
Status = successful
Description = Deployment completed successfully
Instances = 1 desired, 1 placed, 1 healthy, 0 unhealthy
Instances
ID PROCESS VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED
ce1dc6ee app 95 iad run running 1 total, 1 passing 0 5m35s ago
Yes, I agree-- all of this output appears as if everything is running smoothly. Could you send over the logs covering your most recent redeploy (where you’d normally see a reboot?).
With that app metadata, we’ll be able to investigate more thoroughly on our end.
Any additional data you have on the performance problems you’re seeing (say, specific timeframes for slow quieries, for example) could also come in handy.
Thank you for confirming! I think I might understand at least part of the issue.
Taking another look, I think the behavior that you’re describing is expected. If you redeploy with the exact same image and config, nothing should happen. If you’re redeploying with the same image, but making changes to the services, the services should update, but the instance shouldn’t reboot.
I’m likewise glad to hear you’re not currently having the intermittent network issue-- feel free to reach out here if you run into it again!
Isn’t it a typical expectation though? I deploy my app, no matter what version, it should stop the old process and start a new one. If a deployment doesn’t help me with the above goal, I can only restart the app, which will result in downtime, is that right?
Oh one more thing, I have another sidekiq app that runs on the same image as the web server app, and it does start a new instance upon deployment, even with the same image, every single time. So the behavior is not consistent. This is the log when redeploying the same docker image:
2022-07-21T00:45:28Z app[2753d8bd] iad [info]2022-07-21T00:45:28.941Z pid=516 tid=3fqc class=ActionMailer::MailDeliveryJob jid=acac331c7b4617d981f12a01 elapsed=0.048 INFO: done
2022-07-21T00:52:06Z runner[17d77156] iad [info]Starting instance
2022-07-21T00:52:10Z runner[17d77156] iad [info]Configuring virtual machine
2022-07-21T00:52:10Z runner[17d77156] iad [info]Pulling container image
2022-07-21T00:52:16Z runner[17d77156] iad [info]Unpacking image
2022-07-21T00:52:34Z runner[17d77156] iad [info]Configuring firecracker
2022-07-21T00:52:34Z runner[17d77156] iad [info]Starting virtual machine
2022-07-21T00:52:34Z app[17d77156] iad [info]Setting up swapspace version 1, size = 512 MiB (536866816 bytes)
2022-07-21T00:52:34Z app[17d77156] iad [info]no label, UUID=7a39c6d0-f3af-4376-9507-d8216b202571
2022-07-21T00:52:34Z app[17d77156] iad [info]Preparing to run: `bundle exec rails db:migrate` as root
2022-07-21T00:52:34Z app[17d77156] iad [info]2022/07/21 00:52:34 listening on [fdaa:0:402d:a7b:9d35:17d7:7156:2]:22 (DNS: [fdaa::3]:53)
2022-07-21T00:52:41Z app[17d77156] iad [info]You can remove `require 'dalli/cas/client'` as this code has been rolled into the standard 'dalli/client'.
2022-07-21T00:52:42Z app[17d77156] iad [info]I, [2022-07-21T00:52:42.470806 #518] INFO -- : Dalli/SASL authenticating as memcached-app151343598
2022-07-21T00:52:42Z app[17d77156] iad [info]I, [2022-07-21T00:52:42.473493 #518] INFO -- : Dalli/SASL: Authenticated
2022-07-21T00:52:43Z app[17d77156] iad [info]Main child exited normally with code: 0
2022-07-21T00:52:43Z app[17d77156] iad [info]Starting clean up.
fly status:
➜ bookforward git:(master) fly status -c fly.sidekiq.prod.us.toml
Update available 0.0.338 -> v0.0.356.
Run "fly version update" to upgrade.
App
Name = bookforward-sidekiq
Owner = personal
Version = 80
Status = running
Hostname = bookforward-sidekiq.fly.dev
Deployment Status
ID = 12b314ae-46a3-9cdb-ebc7-669d5e070f30
Version = v80
Status = successful
Description = Deployment completed successfully
Instances = 1 desired, 1 placed, 1 healthy, 0 unhealthy
Instances
ID PROCESS VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED
2753d8bd app 80 iad run running 0 12h37m ago
