# Error failed to launch VM: nats: no responders available for request

**URL:** <https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973>\
**Category:** General\
**Created:** [November 26, 2022, 1:24am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973 "2022-11-26T01:24:19Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![nicksergeant](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/nicksergeant/32/7155_2.png) [@nicksergeant](https://community.fly.io/u/nicksergeant)\
**Post date:** [November 26, 2022, 1:24am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/1 "2022-11-26T01:24:19Z")

</div>

Not entirely sure what’s up, there are no issues listed on [https://status.flyio.net/](https://status.flyio.net/)

```auto
❯ fly postgres create --name flxwebsites-db-staging
? Select Organization: FLX Websites (flx-websites)
? Select region: Secaucus, NJ (US) (ewr)
For pricing information visit: https://fly.io/docs/about/pricing/#postgresql-cl
? Select configuration: Production - Highly available, 2x shared CPUs, 4GB RAM, 40GB disk
Creating postgres cluster in organization flx-websites
Creating app...
Setting secrets on app flxwebsites-db-staging...
Provisioning 1 of 2 machines with image flyio/postgres:14.4
Waiting for machine to start...
Machine e148e056ae0892 is created
Provisioning 2 of 2 machines with image flyio/postgres:14.4
Error failed to launch VM: nats: no responders available for request

```

---

<div class="post-metadata">

**Author:** ![JP\_Phillips](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/jp_phillips/32/18337_2.png) [@JP\_Phillips](https://community.fly.io/u/JP_Phillips)\
**Post date:** [November 26, 2022, 1:29am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/2 "2022-11-26T01:29:17Z")

</div>

Hey @nicksergeant! We saw your Tweet as well and have been looking at things, will let you know when things are fixed.

---

<div class="post-metadata">

**Author:** ![JP\_Phillips](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/jp_phillips/32/18337_2.png) [@JP\_Phillips](https://community.fly.io/u/JP_Phillips)\
**Post date:** [November 26, 2022, 1:34am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/3 "2022-11-26T01:34:18Z")

</div>

Ok, would you mind giving it another try?

---

<div class="post-metadata">

**Author:** ![nicksergeant](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/nicksergeant/32/7155_2.png) [@nicksergeant](https://community.fly.io/u/nicksergeant)\
**Post date:** [November 26, 2022, 2:20am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/4 "2022-11-26T02:20:31Z")

</div>

It looks different than it did earlier, but hanging on health checks:

```auto
==> Monitoring health checks
  Waiting for 7328725ec43859 to become healthy (started, 3/3)
  Waiting for 21781e60b06896 to become healthy (started, 1/3)

```

Logs in the dashboard look suspect, also:

```auto
2022-11-26T02:19:34.060 app[21781e60b06896] ewr [info] exporter | ERRO[0264] Error opening connection to database (postgresql://flypgadmin:PASSWORD_REMOVED@[fdaa:0:db85:a7b:94:d127:926e:2]:5433/postgres?sslmode=disable): dial tcp [fdaa:0:db85:a7b:94:d127:926e:2]:5433: connect: connection refused source="postgres_exporter.go:1658"

2022-11-26T02:19:48.063 app[21781e60b06896] ewr [info] exporter | INFO[0278] Established new database connection to "fdaa:0:db85:a7b:94:d127:926e:2:5433". source="postgres_exporter.go:970"

2022-11-26T02:19:49.064 app[21781e60b06896] ewr [info] exporter | ERRO[0279] Error opening connection to database (postgresql://flypgadmin:PASSWORD_REMOVED@[fdaa:0:db85:a7b:94:d127:926e:2]:5433/postgres?sslmode=disable): dial tcp [fdaa:0:db85:a7b:94:d127:926e:2]:5433: connect: connection refused source="postgres_exporter.go:1658"

2022-11-26T02:19:57.151 app[21781e60b06896] ewr [info] keeper | pg_basebackup: error: connection to server at "fdaa:0:db85:a7b:95:6753:be9e:2", port 5433 failed: Connection timed out

2022-11-26T02:19:57.151 app[21781e60b06896] ewr [info] keeper | Is the server running on that host and accepting TCP/IP connections?

2022-11-26T02:19:57.152 app[21781e60b06896] ewr [info] keeper | 2022-11-26T02:19:57.151Z ERROR cmd/keeper.go:1364 failed to resync from followed instance {"error": "sync error: exit status 1"}

2022-11-26T02:20:02.287 app[21781e60b06896] ewr [info] keeper | 2022-11-26T02:20:02.286Z ERROR cmd/keeper.go:1109 db failed to initialize or resync

```

---

<div class="post-metadata">

**Author:** ![nicksergeant](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/nicksergeant/32/7155_2.png) [@nicksergeant](https://community.fly.io/u/nicksergeant)\
**Post date:** [November 26, 2022, 2:21am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/5 "2022-11-26T02:21:48Z")

</div>

Actually looks like it just went through.

---

<div class="post-metadata">

**Author:** ![JP\_Phillips](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/jp_phillips/32/18337_2.png) [@JP\_Phillips](https://community.fly.io/u/JP_Phillips)\
**Post date:** [November 26, 2022, 2:22am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/6 "2022-11-26T02:22:49Z")

</div>

Nice! It can take some time for all of the health checks to be passing.

---

<div class="post-metadata">

**Author:** ![nicksergeant](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/nicksergeant/32/7155_2.png) [@nicksergeant](https://community.fly.io/u/nicksergeant)\
**Post date:** [November 26, 2022, 2:23am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/7 "2022-11-26T02:23:11Z")

</div>

Err, yeah, I think something’s still borked:

```auto
Proxying local port 15432 to remote [flxwebsites-db-staging.internal]:5432
psql: error: connection to server at "localhost" (127.0.0.1), port 15432 failed: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
make: *** [connect-db-local] Error 2

```

---

<div class="post-metadata">

**Author:** ![nicksergeant](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/nicksergeant/32/7155_2.png) [@nicksergeant](https://community.fly.io/u/nicksergeant)\
**Post date:** [November 26, 2022, 2:27am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/8 "2022-11-26T02:27:45Z")

</div>

The connection error is also consistent with an issue I was seeing earlier when attempting to create a pg instance on the Development configuration. I was getting a pooling connection failure from a service that was attempting to connect (via another Fly app).

`fly postgres connect -a flxwebsites-db-staging` works fine but if I `flyctl proxy 15432:5432 -a flxwebsites-db-staging` and then `psql postgres://postgres:<password>@localhost:15432`, I get:

```auto
psql: error: connection to server at "localhost" (127.0.0.1), port 15432 failed: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

```

---

<div class="post-metadata">

**Author:** ![dangra](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/dangra/32/4413_2.png) [@dangra](https://community.fly.io/u/dangra)\
**Post date:** [November 26, 2022, 2:36am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/9 "2022-11-26T02:36:25Z")

</div>

Hi Nick, what postgresql version are you running locally? the output of `psql --version`

---

<div class="post-metadata">

**Author:** ![nicksergeant](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/nicksergeant/32/7155_2.png) [@nicksergeant](https://community.fly.io/u/nicksergeant)\
**Post date:** [November 26, 2022, 2:38am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/10 "2022-11-26T02:38:45Z")

</div>

14.5

---

<div class="post-metadata">

**Author:** ![dangra](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/dangra/32/4413_2.png) [@dangra](https://community.fly.io/u/dangra)\
**Post date:** [November 26, 2022, 2:48am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/11 "2022-11-26T02:48:00Z")

</div>

One of the nodes is still unhealthy

```auto
ID STATE ROLE REGION HEALTH CHECKS IMAGE CREATED UPDATED  
21781e60b06896 started error ewr 3 total, 1 passing, 2 critical flyio/postgres:14.4 (v0.0.32) 2022-11-26T02:14:56Z 2022-11-26T02:15:09Z
7328725ec43859 started leader ewr 3 total, 3 passing flyio/postgres:14.4 (v0.0.32) 2022-11-26T02:14:37Z 2022-11-26T02:14:52Z

```

try selecting which instance to proxy for:

```auto
$ fly proxy 15432:5432 -a flxwebsites-db-staging -s
? Select instance: ewr.flxwebsites-db-staging.internal (fdaa:0:db85:a7b:95:6753:be9e:2)
Proxying local port 15432 to remote [fdaa:0:db85:a7b:95:6753:be9e:2]:5432

```

and then psql to it

```auto
psql: error: connection to server at "localhost" (127.0.0.1), port 15432 failed: FATAL: password authentication failed for user "postgres"

```

(it failed for me because I don’t know the password)

if that works, delete instance 21781e60b06896 and clone 7328725ec43859 to add a new replica

---

<div class="post-metadata">

**Author:** ![dangra](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/dangra/32/4413_2.png) [@dangra](https://community.fly.io/u/dangra)\
**Post date:** [November 26, 2022, 3:04am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/12 "2022-11-26T03:04:50Z")

</div>

I went ahead and cloned the leader and stopped the offending instance

```auto
~$ fly m clone -a flxwebsites-db-staging 7328725ec43859
Cloning machine 7328725ec43859 into region ewr
Provisioning a new machine with image flyio/postgres:14.4...
  Machine 9185ee3c429583 has been created...
  Waiting for machine 9185ee3c429583 to start...
  Waiting for 9185ee3c429583 to become healthy (started, 3/3)
Machine has been successfully cloned!

~$ fly m stop -a flxwebsites-db-staging 21781e60b06896
Sending kill signal to machine 21781e60b06896...
21781e60b06896 has been successfully stopped

```

I didn’t remove it just in case, but feel free to do so with `fly machine remove` anytime.

---

<div class="post-metadata">

**Author:** ![nicksergeant](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/nicksergeant/32/7155_2.png) [@nicksergeant](https://community.fly.io/u/nicksergeant)\
**Post date:** [November 26, 2022, 3:10am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/13 "2022-11-26T03:10:16Z")

</div>

Thanks! What’s the command you used to do the health check on the individual instances?

---

<div class="post-metadata">

**Author:** ![dangra](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/dangra/32/4413_2.png) [@dangra](https://community.fly.io/u/dangra)\
**Post date:** [November 26, 2022, 3:14am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/14 "2022-11-26T03:14:38Z")

</div>

`fly checks list`

---

<div class="post-metadata">

**Author:** ![nicksergeant](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/nicksergeant/32/7155_2.png) [@nicksergeant](https://community.fly.io/u/nicksergeant)\
**Post date:** [November 26, 2022, 3:15am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/15 "2022-11-26T03:15:02Z")

</div>

👍👍 thanks!

---

<div class="post-metadata">

**Author:** ![ignoramous](https://avatars.discourse-cdn.com/v4/letter/i/a9adbd/32.png) [@ignoramous](https://community.fly.io/u/ignoramous)\
**Post date:** [November 26, 2022, 8:43am UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/16 "2022-11-26T08:43:25Z")

</div>

> [@dangra](#):
>
> I didn’t remove it just in case, but feel free to do so with `fly machine remove` anytime.

A (regular, non-db) machine I have got that has entered this state refuses to be `remove`d: [Machines error blocking deploys: No responders available for request](https://community.fly.io/t/machines-error-blocking-deploys-no-responders-available-for-request/8954)

What gives? I want it gone too as it’s blocking our use of `flyctl` for `deploy`s.

> [@JP\_Phillips](#):
>
> Hey @nicksergeant! We saw your Tweet as well and have been looking at things, will let you know when things are fixed.

I guess everyone should tweet than create posts here 😉 seems super effective: [https://archive.is/TFGIo](https://archive.is/TFGIo)

---

<div class="post-metadata">

**Author:** ![nicksergeant](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/nicksergeant/32/7155_2.png) [@nicksergeant](https://community.fly.io/u/nicksergeant)\
**Post date:** [November 26, 2022, 9:57pm UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/17 "2022-11-26T21:57:07Z")

</div>

@dangra is there a reason that deploying the configuration `Production - Highly available, 2x shared CPUs, 4GB RAM, 40GB disk` in EWR continues to only result in one healthy instance?

I tried again this morning to create this cluster (I’m working on creating repeatable scripts for my infra, so I want to make sure this is consistent), and I still only have one instance healthy:

```auto
❯ fly checks list -a flxwebsites-db
Health Checks for flxwebsites-db
  NAME | STATUS | MACHINE | LAST UPDATED | OUTPUT
-------*----------*----------------*--------------*--------------------------------------------------------------------------
  pg | passing | 0e286961b50867 | 4m36s ago | [✓] transactions: read/write (341.43µs)
       | | | | [✓] connections: 9 used, 3 reserved, 300 max (4.39ms)
-------*----------*----------------*--------------*--------------------------------------------------------------------------
  role | passing | 0e286961b50867 | 4m37s ago | leader
-------*----------*----------------*--------------*--------------------------------------------------------------------------
  vm | passing | 0e286961b50867 | 5m3s ago | [✓] checkDisk: 37.06 GB (94.7%) free space on /data/ (86.66µs)
       | | | | [✓] checkLoad: load averages: 0.00 0.00 0.00 (181.59µs)
       | | | | [✓] memory: system spent 0s of the last 60s waiting on memory (96.84µs)
       | | | | [✓] cpu: system spent 0s of the last 60s waiting on cpu (82.71µs)
       | | | | [✓] io: system spent 0s of the last 60s waiting on io (73.62µs)
-------*----------*----------------*--------------*--------------------------------------------------------------------------
  pg | critical | 21781e67a03896 | 4m43s ago | 500 Internal Server Error
       | | | | failed to connect to proxy: context deadline exceeded
-------*----------*----------------*--------------*--------------------------------------------------------------------------
  role | critical | 21781e67a03896 | 4m32s ago | 500 Internal Server Error
       | | | | failed to connect to local node: context deadline exceeded
-------*----------*----------------*--------------*--------------------------------------------------------------------------
  vm | passing | 21781e67a03896 | 3m57s ago | [✓] checkDisk: 37.06 GB (94.7%) free space on /data/ (67.84µs)
       | | | | [✓] checkLoad: load averages: 0.00 0.00 0.00 (68.78µs)
       | | | | [✓] memory: system spent 0s of the last 60s waiting on memory (79.78µs)
       | | | | [✓] cpu: system spent 144ms of the last 60s waiting on cpu (31.2µs)
       | | | | [✓] io: system spent 384ms of the last 60s waiting on io (22.64µs)
-------*----------*----------------*--------------*--------------------------------------------------------------------------

```

---

<div class="post-metadata">

**Author:** ![nicksergeant](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/nicksergeant/32/7155_2.png) [@nicksergeant](https://community.fly.io/u/nicksergeant)\
**Post date:** [November 27, 2022, 9:04pm UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/18 "2022-11-27T21:04:13Z")

</div>

I’m pretty sure the problem here is the `ewr` region. I switched to `iad` and everything works perfectly fine. [Launching a new Postgres 1.7 app with DB results in "Error release command failed, deployment aborted" - #8 by nicksergeant](https://community.fly.io/t/launching-a-new-postgres-1-7-app-with-db-results-in-error-release-command-failed-deployment-aborted/8999/8)

---

<div class="post-metadata">

**Author:** ![dangra](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/dangra/32/4413_2.png) [@dangra](https://community.fly.io/u/dangra)\
**Post date:** [November 28, 2022, 8:09pm UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/19 "2022-11-28T20:09:46Z")

</div>

True, a misconfigured network interface on one host of ewr region was causing issues to apps whose machines were assigned to run there. It is fixed now. thanks.

> [@Launching a new Postgres 1.7 app with DB results in "Error release command failed, deployment aborted"](https://community.fly.io/t/launching-a-new-postgres-1-7-app-with-db-results-in-error-release-command-failed-deployment-aborted/8999/9):
>
> Hi Nick. Glad to hear you found iad region to run your app and thanks for not giving up. Following your report we have nailed down the problem on ewr region to a misconfigured host and fixed it. thanks again

---

<div class="post-metadata">

**Author:** ![nicksergeant](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/nicksergeant/32/7155_2.png) [@nicksergeant](https://community.fly.io/u/nicksergeant)\
**Post date:** [November 28, 2022, 9:09pm UTC](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973/20 "2022-11-28T21:09:36Z")

</div>

@dangra 🙏 thank you!

[Next page](https://community.fly.io/t/error-failed-to-launch-vm-nats-no-responders-available-for-request/8973.md?page=2)