Postgres replica can't be added (failed to resolve member over dns)

hodoli · February 3, 2025, 1:05pm

when i run fly scale count 3 to add 1 more replica to the Postgres cluster,
i get this error.

2025-02-03T12:59:44.329 runner[5683770c600228] nrt [info] Machine started in 1.463s

2025-02-03T12:59:44.450 app[5683770c600228] nrt [info] 2025/02/03 12:59:44 INFO SSH listening listen_address=[fdaa:3:b480:a7b:d86c:416e:39f8:2]:22 dns_server=[fdaa::3]:53

2025-02-03T12:59:44.558 app[5683770c600228] nrt [info] Provisioning standby

2025-02-03T12:59:44.591 app[5683770c600228] nrt [info] panic: failed to resolve member over dns: unable to resolve cloneable member

2025-02-03T12:59:44.591 app[5683770c600228] nrt [info] goroutine 1 [running]:

2025-02-03T12:59:44.591 app[5683770c600228] nrt [info] main.panicHandler({0xa0c0a0, 0xc000297690})

2025-02-03T12:59:44.591 app[5683770c600228] nrt [info] /go/src/github.com/fly-apps/fly-postgres/cmd/start/main.go:190 +0x4a

2025-02-03T12:59:44.591 app[5683770c600228] nrt [info] main.main()

2025-02-03T12:59:44.591 app[5683770c600228] nrt [info] /go/src/github.com/fly-apps/fly-postgres/cmd/start/main.go:67 +0xe65

2025-02-03T12:59:45.204 app[5683770c600228] nrt [info] INFO Main child exited normally with code: 2

2025-02-03T12:59:45.229 app[5683770c600228] nrt [info] INFO Starting clean up.

2025-02-03T12:59:45.290 app[5683770c600228] nrt [info] INFO Umounting /dev/vdc from /data

I can see the error is coming from this line

github.com/fly-apps/postgres-flex

internal/flypg/repmgr.go

master


      
          			return nil, fmt.Errorf("failed to close connection: %s", err)
          		}
          
          		if member.Role == PrimaryRoleName && member.Active {
          			target = member
          			break
          		}
          	}
          
          	if target == nil {
          		return nil, fmt.Errorf("unable to resolve cloneable member")
          	}
          
          	return target, nil
          }
          
          func (r *RepMgr) InRegionPeerIPs(ctx context.Context) ([]net.IPAddr, error) {
          	targets := fmt.Sprintf("%s.%s", r.PrimaryRegion, r.AppName)
          	return privnet.AllPeers(ctx, targets)
          }

but I do have the primary instance in the cluster. (1 primary and 1 replica currently)

hodoli · February 3, 2025, 2:00pm

The problem was resolved after the incident below finished.
Still have no idea why DFW region’s maintenance affected a NRT region instance though.

Network maintenance in DFW

Completed - The scheduled maintenance has been completed.
Feb 3, 13:52 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 3, 12:52 UTC

Scheduled - An upstream provider is performing urgent network maintenance in DFW where you may experience some short periods of connectivity issues.

system · February 10, 2025, 2:01pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Error connecting to Postgres Questions / Help postgres	1	260	September 19, 2022
Unable to connect to my postgres instance Questions / Help postgres	6	585	February 22, 2023
Postgres instance count increase fails Questions / Help postgres , issue	0	397	April 30, 2022
Postgres cluster broken since last Fly migration Questions / Help postgres	1	121	July 2, 2024
App -> Postgres DNS Issue Questions / Help postgres	3	147	March 13, 2024

Postgres replica can't be added (failed to resolve member over dns)

Related topics