Elixir/Redix Timeouts on Connection

kitplummer · November 29, 2022, 3:23am

Trying to run fly deploy on my Elixir app and when I do:

[info]{"Kernel pid terminated",application_controller,"{application_start_failure,my_app,{{shutdown,{failed_to_start_child,'Elixir.Redix',#{'__exception__' => true,'__struct__' => 'Elixir.Redix.ConnectionError',reason => timeout}}},{'Elixir.MyApp.Application',start,[normal,[]]}}}"}

This deploy has successfully worked before. I’ve changed a few things on my side, but it looks like I’m passing the right stuff to the Redis connection. Deleted the app and reran fly launch - and now get this error all the time. Any chance this isn’t on my side?

I can fly redis connect to it no problem. But is it possible that I can’t connect the second+ time because of the existing app’s connection to Redis?

jsierles · November 29, 2022, 11:32am

Have you tried logging in to the VM and connecting from there, either through iex or using redis-cli on that VM?

kitplummer · November 29, 2022, 1:56pm

Thanks for checkin in. Is the new VM still running after the failed deploy - not sure how to connect to it? If I run fly ssh console that logs me into the VM with the currently running app right?

Failure #1

Instance
ID              PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS           RESTARTS        CREATED  
3b48994e        app     5       iad     run     failed  1 total, 1 critical     2               1m2s ago

Recent Events
TIMESTAMP               TYPE            MESSAGE                                                         
2022-11-29T03:21:31Z    Received        Task received by client                                        
2022-11-29T03:21:31Z    Task Setup      Building Task Directory                                        
2022-11-29T03:21:46Z    Started         Task started by client                                         
2022-11-29T03:21:58Z    Terminated      Exit Code: 1                                                   
2022-11-29T03:21:58Z    Restarting      Task restarting in 1.15693331s                                 
2022-11-29T03:22:07Z    Started         Task started by client                                         
2022-11-29T03:22:19Z    Terminated      Exit Code: 1                                                   
2022-11-29T03:22:19Z    Restarting      Task restarting in 1.178005874s                                
2022-11-29T03:22:27Z    Started         Task started by client                                         
2022-11-29T03:22:39Z    Terminated      Exit Code: 1                                                   
2022-11-29T03:22:39Z    Not Restarting  Exceeded allowed attempts 2 in interval 5m0s and mode is "fail"
2022-11-29T03:22:39Z    Alloc Unhealthy Unhealthy because of failed task                               

2022-11-29T03:22:15Z   [info]03:22:15.487 [info]  Tzdata has updated the release from 2021e to 2022f
2022-11-29T03:22:15Z   [info]03:22:15.487 [debug] Tzdata deleting ETS table for version 2021e
2022-11-29T03:22:15Z   [info]03:22:15.490 [debug] Tzdata deleting ETS table file for version 2021e
2022-11-29T03:22:16Z   [info]03:22:16.106 [info]  Application my_app exited: MyApp.Application.start(:normal, []) returned an error: shutdown: failed to start child: Redix
2022-11-29T03:22:16Z   [info]    ** (EXIT) %Redix.ConnectionError{reason: :timeout}
2022-11-29T03:22:17Z   [info]{"Kernel pid terminated",application_controller,"{application_start_failure,my_app,{{shutdown,{failed_to_start_child,'Elixir.Redix',#{'__exception__' => true,'__struct__' => 'Elixir.Redix.ConnectionError',reason => timeout}}},{'Elixir.MyApp.Application',start,[normal,[]]}}}"}
2022-11-29T03:22:17Z   [info]Kernel pid terminated (application_controller) ({application_start_failure,my_app,{{shutdown,{failed_to_start_child,'Elixir.Redix',#{'__exception__' => true,'__struct__' => 'Elixir.Redix.Co
2022-11-29T03:22:18Z   [info]Starting clean up.
2022-11-29T03:22:24Z   [info]Starting instance
2022-11-29T03:22:26Z   [info]Configuring virtual machine
2022-11-29T03:22:26Z   [info]Unpacking image
2022-11-29T03:22:27Z   [info]Configuring firecracker
2022-11-29T03:22:27Z   [info]Starting virtual machine
2022-11-29T03:22:27Z   [info]Starting init (commit: 81d5330)...
2022-11-29T03:22:27Z   [info]Preparing to run: `/cnb/process/web` as heroku
2022-11-29T03:22:27Z   [info]Virtual machine started successfully
2022-11-29T03:22:27Z   [info]2022/11/29 03:22:27 listening on [fdaa:0:33c6:a7b:93:3b48:994e:2]:22 (DNS: [fdaa::3]:53)
2022-11-29T03:22:30Z   [info]03:22:30.603 [info]  REDIS_URL: redis://default:XXX@fly-myapp-redis.upstash.io
2022-11-29T03:22:33Z   [info]03:22:33.558 [debug] Tzdata polling for update.
2022-11-29T03:22:34Z   [info]03:22:34.026 [debug] Tzdata downloading new data from https://data.iana.org/time-zones/tzdata-latest.tar.gz
2022-11-29T03:22:34Z   [info]03:22:34.092 [debug] Tzdata data downloaded. Release version 2022f.
2022-11-29T03:22:35Z   [info]03:22:35.158 [info]  Tzdata has updated the release from 2021e to 2022f
2022-11-29T03:22:35Z   [info]03:22:35.158 [debug] Tzdata deleting ETS table for version 2021e
2022-11-29T03:22:35Z   [info]03:22:35.162 [debug] Tzdata deleting ETS table file for version 2021e
2022-11-29T03:22:35Z   [info]03:22:35.679 [info]  Application lowendinsight_get exited: LowendinsightGet.Application.start(:normal, []) returned an error: shutdown: failed to start child: Redix
2022-11-29T03:22:35Z   [info]    ** (EXIT) %Redix.ConnectionError{reason: :timeout}
2022-11-29T03:22:37Z   [info]{"Kernel pid terminated",application_controller,"{application_start_failure,my_app,{{shutdown,{failed_to_start_child,'Elixir.Redix',#{'__exception__' => true,'__struct__' => 'Elixir.Redix.ConnectionError',reason => timeout}}},{'Elixir.MyApp.Application',start,[normal,[]]}}}"}
2022-11-29T03:22:37Z   [info]Kernel pid terminated (application_controller) ({application_start_failure,my_app,{{shutdown,{failed_to_start_child,'Elixir.Redix',#{'__exception__' => true,'__struct__' => 'Elixir.Redix.Co
2022-11-29T03:22:37Z   [info]Crash dump is being written to: erl_crash.dump...done
2022-11-29T03:22:37Z   [info]Starting clean up.
--> v5 failed - Failed due to unhealthy allocations - rolling back to job version 4 and deploying as v6

jsierles · November 29, 2022, 2:40pm

Can you try deploying to a different region to see if it helps?

kitplummer · November 29, 2022, 3:02pm

Trying to run fly deploy -r dfw but it looks like it is still trying to deploy to iad:

1 desired, 1 placed, 0 healthy, 1 unhealthy [restarts: 2] [health checks: 1 total, 1 critical]
Failed Instances

Failure #1

Instance
ID              PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS           RESTARTS        CREATED  
520ed997        app     15      iad     run     running 1 total, 1 critical     2               1m8s ago

Recent Events
TIMESTAMP               TYPE            MESSAGE                         
2022-11-29T14:51:36Z    Received        Task received by client        
2022-11-29T14:51:36Z    Task Setup      Building Task Directory        
2022-11-29T14:51:50Z    Started         Task started by client         
2022-11-29T14:52:02Z    Terminated      Exit Code: 1                   
2022-11-29T14:52:02Z    Restarting      Task restarting in 1.029717838s
2022-11-29T14:52:10Z    Started         Task started by client         
2022-11-29T14:52:22Z    Terminated      Exit Code: 1                   
2022-11-29T14:52:22Z    Restarting      Task restarting in 1.087583623s
2022-11-29T14:52:33Z    Started         Task started by client

And failing with that timeout issue.

Tried other region too - fly deploy -r lax. Probably doing something dumb here…but how to force deploy to other region?

kitplummer · November 29, 2022, 3:18pm

Ah, I had to add the region first. Then I can deploy to it. And that worked. Seems I needed two regions in order to roll out the canary.

Topic		Replies	Views
Single VM deploy causes temporary unavailability Questions / Help	7	549	February 23, 2022
ssh console and checking app says it's down but it's not Questions / Help elixir	25	2275	August 2, 2022
Could not start IEx CLI due to reason: :noconnection Questions / Help elixir	2	596	July 28, 2023
Feedback: First use of Fly.io help-me-help-you , elixir	6	488	February 18, 2023
Elixir app not starting up Questions / Help	13	2126	June 3, 2021

Elixir/Redix Timeouts on Connection

Related Topics