Same here
When you removed the wire_guard_state and tried it again it still didn’t work, right? We’d expect it to put new keys back in, but it’s still not working?
Right
It put new keys and it’s still not working
1 Like
Could this be due to a tools update (flyctl)?
If it was a flyctl problem I’d expect it to break all the orgs you’re connecting to, not just this one. We’re in the process of debugging this, though, will share more info when we have something.
I have the problem for any instances trusseltrust and concordia They both are in region fra
fra I meant in ~/.fly/config.yml but they are in lhr
Thanks, we’re debugging this. In the meanwhile let me try to find a workaround.
@nickolay.loshkarev Could you do a fly status on the trusseltrust and concordia apps? Would help to see how long they’ve been running.
At this point the CLI seems to be connecting to the wrong IPs — could you also run
fly ips private
to get the current list of your VM IPs, and then
fly ssh console -s
— when you choose an app/region combination here it’ll show which IP it’s trying to connect to. If there’s a mismatch there we’ll know that that’s the issue.
That’s really helpful, thanks. The DNS call to get the IP(s) to connect do seems to be failing, we can look specifically at that.
Wow. That looks network-y (that’s the error you get — it should be a better error! — when flyctl can’t talk to our DNS at all).
A question: does this work sporadically for you, or never?
If it works for you sometimes, does it depend in any way on where you’re working from (home, office, etc)?
@nickolay.loshkarev you can try changing ~/.fly/config.yml with a new peer to get going on this, while we fix the problem:
- run
fly wg create concordia lhrto create a new peer inlhr, you can choose to get the output onstdout. This will print something like
[Peer]
PublicKey = eCP0xXXXXXXXXXXXXpFUTxhjvubgDlLfVZyFk=
Endpoint = lhr1.gateway.6pn.dev:51820
- run
fly wg list, you should see something like
+-----------------------------------------------------+--------+----------------------------+
| NAME | REGION | PEER IP |
+-----------------------------------------------------+--------+----------------------------+
| interactive-Sudhirs-Mac-mini-sudhir-j-gmail-com-785 | maa | fdaa:0:33b5:a7b:1bfe:0:a:2 |
| interactive-Sudhirs-Mac-mini-sudhir-j-gmail-com-996 | lhr | fdaa:0:33b5:a7b:dc6:0:a:2 |
+-----------------------------------------------------+--------+----------------------------+
- kill the agent if it’s running
❯ ps aux | grep "fly agent" 22:48:56
sj 50704 0.0 0.2 409255600 35248 s006 S 10:32PM 0:01.73 fly agent daemon-start
sj 50866 0.0 0.0 408103312 1344 s006 S+ 10:51PM 0:00.00 grep fly agent
❯ kill 50704
You can then update the section for wire_guard_state.concordia.peer in config.yml with peerip from the list above, and pubkey, and endpoint (remove the :51820).
You should then be able to run the console for condordia, and you can do the same for the other org as well.
Will update again once we get the issue fixed, but this should route you via lhr instead of fra.
It doesn’t work at all.
I work from Russia.
I tried connect to VPN in London, removed wire_guard_state record and ran flyctl ssh console -a concordia-production-web -r lhr
But for @ matt2 this doesn’t work either, he’s in UK
We don’t have a clear fix for the problem yet — I’ve replicated it and fixed it for myself by removing all my peers using fly wireguard remove, then trying to re-connect (it should create a new peer for you).
You could do fly wireguard list to see which peers are in fra and then remove those selectively.
This is a critical issue. Please, send me any script or anything for ssh connection
If necessary I can share my screen to do this together