# DB Connection issues **URL:** \ **Category:** General\ **Tags:** elixir, postgres\ **Created:** [June 28, 2022, 6:52pm UTC](https://community.fly.io/t/db-connection-issues/5801 "2022-06-28T18:52:50Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![eliel](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/eliel/32/2005_2.png) [@eliel](https://community.fly.io/u/eliel)\ **Post date:** [June 28, 2022, 6:52pm UTC](https://community.fly.io/t/db-connection-issues/5801/1 "2022-06-28T18:52:50Z")
Having issues with connecting to my database all of a sudden. Region in lax. Everything was fine then all of a sudden nothing can connect to it. I’ve had similar issues with my other apps in production. Do others experience this issue? Can we rely on fly for hosting a database? It seems like there are always connection issues that happen way too frequently for anything stable. On the PG instance getting this: Error opening connection to database (postgresql://flypgadmin:PASSWORD\_REMOVED@[address]:5432/postgres?sslmode=disable): dial tcp [address]:5432: connect: connection refused source=“postgres\_exporter.go:1658” ---
**Author:** ![Mark](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mark/32/15656_2.png) [@Mark](https://community.fly.io/u/Mark)\ **Post date:** [July 1, 2022, 12:24am UTC](https://community.fly.io/t/db-connection-issues/5801/2 "2022-07-01T00:24:00Z")
Hi @eliel! Have you checked on your IPv6 settings for Ecto? They can be set in the Dockerfile using ENV. > [@Could not contact remote node reason: :nodedown. Aborting...](https://community.fly.io/t/could-not-contact-remote-node-reason-nodedown-aborting/5221/3): > > I’m experiencing this issue with my application. ➜ app git:(fly) fly ssh console Connecting to top1.nearest.of.savvycal-staging.internal... complete # app/bin/mighty remote Erlang/OTP 24 [erts-12.1.5] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1] [jit] Could not contact remote node savvycal-staging@fdaa:0:61d4:a7b:9d36:7c1f:f184:2, reason: :nodedown. Aborting... In the logs: 2022-06-13T16:21:57Z app[7c1ff184] iad [info]Reaped child process with pid: 728, exit code: 0 2022-06-13T… ---
**Author:** ![eliel](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/eliel/32/2005_2.png) [@eliel](https://community.fly.io/u/eliel)\ **Post date:** [July 1, 2022, 12:33am UTC](https://community.fly.io/t/db-connection-issues/5801/3 "2022-07-01T00:33:27Z")
@Mark I believe our project is setup for ipv6. We have been running this server for a few months now. It only happened all of a sudden after I made some UI changes to my phoenix app. The postgres VM was alive, it just wouldn’t allow any connections. I have a postres client that I use to regularly check the database (Postico) and I couldn’t connect there either. I had to scale down the postgres vm, update the postgres image and scale it back up. Very alarming, the error I was getting was this. ```auto (postgresql://flypgadmin:PASSWORD_REMOVED@[address]:5432/postgres?sslmode=disable): dial tcp [address]:5432: connect: connection refused source=“postgres_exporter.go:1658” ``` This was captured from the logs in the Postgres VM btw ---
**Author:** ![Mark](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mark/32/15656_2.png) [@Mark](https://community.fly.io/u/Mark)\ **Post date:** [July 1, 2022, 5:02pm UTC](https://community.fly.io/t/db-connection-issues/5801/4 "2022-07-01T17:02:55Z")
Ah, thanks for the context. Yes, it’s likely not suddenly a config issue. ---
**Author:** ![Mark](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mark/32/15656_2.png) [@Mark](https://community.fly.io/u/Mark)\ **Post date:** [July 1, 2022, 5:46pm UTC](https://community.fly.io/t/db-connection-issues/5801/5 "2022-07-01T17:46:22Z")
@eliel I looked into it more. Sorry that I don’t have an answer, but just wanted to relay that it is being actively researched by the platform team. ---
**Author:** ![eliel](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/eliel/32/2005_2.png) [@eliel](https://community.fly.io/u/eliel)\ **Post date:** [October 9, 2022, 9:20pm UTC](https://community.fly.io/t/db-connection-issues/5801/6 "2022-10-09T21:20:11Z")
@Mark Any updates on this? We are getting this issue again. Our production app was down for about 5mins because of our Postgres app again. ---
**Author:** ![Mark](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mark/32/15656_2.png) [@Mark](https://community.fly.io/u/Mark)\ **Post date:** [October 10, 2022, 7:39pm UTC](https://community.fly.io/t/db-connection-issues/5801/7 "2022-10-10T19:39:51Z")
Hi @eliel, Wanted to make sure you were aware of how to get the logs for your postgres DB app: ```auto fly logs -a my-postgres-app-db ``` To find the name of the postgres app, you can use: ```auto fly apps list ``` ---
**Author:** ![Mark](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mark/32/15656_2.png) [@Mark](https://community.fly.io/u/Mark)\ **Post date:** [October 10, 2022, 7:54pm UTC](https://community.fly.io/t/db-connection-issues/5801/8 "2022-10-10T19:54:17Z")
You can also run this command for info: ```auto fly checks list -a my-postgres-app-db ``` ---
**Author:** ![eliel](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/eliel/32/2005_2.png) [@eliel](https://community.fly.io/u/eliel)\ **Post date:** [October 10, 2022, 8:10pm UTC](https://community.fly.io/t/db-connection-issues/5801/9 "2022-10-10T20:10:57Z")
@Mark This is the output, doesnt seem to show anything recent: ```auto Health Checks for classly-prod-pg NAME | STATUS | ALLOCATION | REGION | TYPE | LAST UPDATED | OUTPUT -------*---------*------------*--------*------*----------------------*-------------------------------------------------------------------------- pg | passing | fae7f0d0 | lax | HTTP | 57s ago | HTTP GET http://172.19.1.90:5500/flycheck/pg: 200 OK Output: "[✓] | | | | | | | | | | | | transactions: read/write (246.62µs)\n[✓] | | | | | | | | | | | | connections: 72 used, 3 reserved, 300 max (5.75ms)"[✓] | | | | | | | | | | | | vm | passing | fae7f0d0 | lax | HTTP | 4m46s ago | HTTP GET http://172.19.1.90:5500/flycheck/vm: 200 OK Output: "[✓] | | | | | | | | | | | | checkDisk: 676.66 MB (69.3%) free space on /data/ (67.96µs)\n[✓] | | | | | | | | | | | | checkLoad: load averages: 0.05 0.03 0.00 (129.29µs)\n[✓] | | | | | | | | | | | | memory: system spent 0s of the last 60s waiting on memory (47.5µs)\n[✓] | | | | | | | | | | | | cpu: system spent 1.12s of the last 60s waiting on cpu (53.52µs)\n[✓] | | | | | | | | | | | | io: system spent 0s of the last 60s waiting on io (49.77µs)"[✓] | | | | | | | | | | | | role | passing | fae7f0d0 | lax | HTTP | 2022-10-09T20:08:48Z | leader[✓] | | | | | | | | | | | | ``` ---
**Author:** ![Mark](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mark/32/15656_2.png) [@Mark](https://community.fly.io/u/Mark)\ **Post date:** [October 10, 2022, 8:18pm UTC](https://community.fly.io/t/db-connection-issues/5801/10 "2022-10-10T20:18:34Z")
I checked with some people on the Platform team and Support. > I remember seeing issues with Elixir/Phoenix apps a while back. I’m not 100% this is the issue, but there were issues with connections not being closed properly during deploys. The end result was either OOM or running out of connections. > > I’ve only seen it happen with Phoenix apps. (OOM = Out Of Memory) I don’t know if that’s related. But it makes me wonder if older versions of `ecto` or `ecto_sql` or `postgrex` might be a problem? ---
**Author:** ![eliel](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/eliel/32/2005_2.png) [@eliel](https://community.fly.io/u/eliel)\ **Post date:** [October 10, 2022, 8:20pm UTC](https://community.fly.io/t/db-connection-issues/5801/11 "2022-10-10T20:20:43Z")
@Mark Weirdly looking at the graphana dashboard, this is a gap in the logs for the pg app ![Screen Shot 2022-10-10 at 1.11.51 PM](https://global.discourse-cdn.com/fly1/original/3X/4/4/44f5ad66dc0776e3b8b070159adb2278a4ec0d7e.jpeg) ---
**Author:** ![Mark](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mark/32/15656_2.png) [@Mark](https://community.fly.io/u/Mark)\ **Post date:** [October 10, 2022, 8:34pm UTC](https://community.fly.io/t/db-connection-issues/5801/12 "2022-10-10T20:34:05Z")
@eliel Weird. It blanks out then the CPU pegs, memory falls and network blips, then silence. ---
**Author:** ![eliel](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/eliel/32/2005_2.png) [@eliel](https://community.fly.io/u/eliel)\ **Post date:** [October 10, 2022, 8:54pm UTC](https://community.fly.io/t/db-connection-issues/5801/13 "2022-10-10T20:54:19Z")
Yeah, very weird. Our phoenix app was fine during that period. We also haven’t deployed in a few weeks so it shouldn’t be a deploy issue. Those blimps the graphana logs correspond to the times we noticed the app was down due to postgres connection issues. We also had very very low traffic so it should be memory / connections issue ---
**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\ **Post date:** [October 10, 2022, 8:59pm UTC](https://community.fly.io/t/db-connection-issues/5801/14 "2022-10-10T20:59:42Z")
That looks like pg crashed from the graphs. Note how one color goes away and a new color of line replaces it. `fly status --all -a ` will let you know if one of the VMs failed. Then you can run `fly vm status -a ` to see why. ---
**Author:** ![eliel](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/eliel/32/2005_2.png) [@eliel](https://community.fly.io/u/eliel)\ **Post date:** [October 10, 2022, 9:04pm UTC](https://community.fly.io/t/db-connection-issues/5801/15 "2022-10-10T21:04:07Z")
@Mark @kurt This is what I get when running the status: ```auto Instances ID PROCESS VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED fae7f0d0 app 5 lax run running (leader) 3 total, 3 passing 0 2022-10-09T20:06:17Z e9ce1fb2 app 5 lax stop failed 0 2022-10-09T19:51:14Z 542badff app 5 lax stop failed 0 2022-10-09T19:49:16Z ``` Then running `vm status` on instance `e9ce1fb2`: ```auto Events TIMESTAMP TYPE MESSAGE 2022-10-09T19:51:32Z Received Task received by client 2022-10-09T19:51:32Z Task Setup Building Task Directory 2022-10-09T20:05:33Z Driver Failure rpc error: code = Unknown desc = unable to create microvm: could not find device for volume with name pg_data 2022-10-09T20:05:33Z Not Restarting Error was unrecoverable 2022-10-09T20:05:35Z Killing Sent interrupt. Waiting 5m0s before force killing Checks ID SERVICE STATE OUTPUT Recent Logs ``` Then on instance `542badff`: ```auto Events TIMESTAMP TYPE MESSAGE 2022-10-09T19:48:56Z Received Task received by client 2022-10-09T19:48:56Z Task Setup Building Task Directory 2022-10-09T19:50:25Z Driver Failure rpc error: code = Unknown desc = unable to make tap and generate ip addresses: kill zombies: found colliding a88263-2f9a9d3d: interface is up, cannot reap 2022-10-09T19:50:25Z Not Restarting Error was unrecoverable 2022-10-09T19:50:53Z Killing Sent interrupt. Waiting 5m0s before force killing 2022-10-09T19:50:53Z Killing Sent interrupt. Waiting 5m0s before force killing 2022-10-09T20:05:09Z Received Task received by client 2022-10-09T20:07:21Z Killing Sent interrupt. Waiting 5m0s before force killing Checks ID SERVICE STATE OUTPUT Recent Logs ``` ---
**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\ **Post date:** [October 10, 2022, 9:05pm UTC](https://community.fly.io/t/db-connection-issues/5801/16 "2022-10-10T21:05:46Z")
That looks like it went into a crash loop and then took a bit for us to recover. Those driver failures are us cleaning up the env after a previous crash. ---
**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\ **Post date:** [October 10, 2022, 9:07pm UTC](https://community.fly.io/t/db-connection-issues/5801/17 "2022-10-10T21:07:23Z")
If there are older failed vms, just check status on each of them. You’ll eventually find one with the original error. Also, scaling up to two nodes will help your app handle this state. These crashes are almost always out-of-memory errors, so adding more memory is a good bet. ---
**Author:** ![eliel](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/eliel/32/2005_2.png) [@eliel](https://community.fly.io/u/eliel)\ **Post date:** [October 10, 2022, 9:15pm UTC](https://community.fly.io/t/db-connection-issues/5801/18 "2022-10-10T21:15:07Z")
Memory usage seemed fine? Using ~330 / 512mb. Very little to no traffic. When I listed the status only 2 old instances came up. Is there a way to narrow this down? Seems a little random, although this has happened in the past (a month or two ago) ---
**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\ **Post date:** [October 10, 2022, 9:36pm UTC](https://community.fly.io/t/db-connection-issues/5801/19 "2022-10-10T21:36:42Z")
Ok I found the original alloc. It exited with no information, then took a couple of minutes to come back. I think your best bet is to add a second node and see if it happens again. Two nodes will give you HA, so if there are issues with one your app won’t have connectivity problems.