Create app & add WireGuard peer via API

Hello, I’m trying to create an app and add a WireGuard peer (all via the API).

The problem is that the WireGuard peer doesn’t end up in the same network as the app.

  1. Running dig +short _apps.internal fdaa:0:33::3 on the fly instance, I can see the app "tpproxy-lb0jv6".
  2. Running dig +short _apps.internal fdaa:0:33::3 on the WireGuard peer returns nothing.

Debugging

  1. Using flyctl to create both the app and the WireGuard peer works fine.
  2. Using the API to create the app and flyctl to create the WireGuard peer works fine.
  3. Using the API to create both the app and the WireGuard peer doesn’t work.

I assume that my WireGuard config is in the problem.

Here are the step to recreate the issue:

Set some environment variables:

FLY_REGION="lhr"
WG_PEER_NAME="jason-tp"
FLY_API_TOKEN="YOUR-FLY-API-TOKEN"
FLY_ORG_ID="YOUR-FLY-ORG-ID"
APP_NAME="tpproxy-qezw1f"

Create the fly.toml file:

app = "tpproxy-qezw1f"
kill_signal = "SIGINT"
kill_timeout = 5

# Remove this section to build from the local Dockerfile
[build]
  image = "sspreitzer/shellinabox:latest"

[experimental]
  private_network=true

[[services]]
  internal_port = 4200
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    port = "443"

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    port = "4200"
    restart_limit = 6
    timeout = "2s"

Create a new app:

curl 'https://api.fly.io/graphql' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer ${FLY_API_TOKEN}" \
  --data '{
  "query": "mutation($input: CreateAppInput!){ createApp(input: $input) { app { id name organization { id slug } network runtime regions { name code } } } }",
  "variables": {
    "input": {
      "name": "'"${APP_NAME}"'",
      "runtime": "FIRECRACKER",
      "organizationId": "'"${FLY_ORG_ID}"'",
      "preferredRegion": "'"${FLY_REGION}"'"
    }
  }
}'; echo

Generate your private & public keys for WireGuard:

wg genkey > wg-test.priv && \
  wg pubkey < wg-test.priv > wg-test.pub && \
  cat wg-test.pub

Set your WireGuard public key (that you just created):

WG_PEER_PUBLIC_KEY="<put contents of wg-test.pub here>"

Add a WireGuard peer:

curl 'https://api.fly.io/graphql' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer ${FLY_API_TOKEN}" \
  --data '{
  "query": "mutation($input: AddWireGuardPeerInput!){ addWireGuardPeer(input: $input){ clientMutationId endpointip peerip pubkey } }",
  "variables": {
    "input": {
      "organizationId": "'"${FLY_ORG_ID}"'",
      "region": "'"${FLY_REGION}"'",
      "name": "'"${WG_PEER_NAME}"'",
      "pubkey": "'"${WG_PEER_PUBLIC_KEY}"'"
    }
  }
}'; echo

Create the WireGuard config file as wg0.conf:

[Interface]
PrivateKey = <put contents of wg-test.priv here>
Address = <put peerip from addWireguardPeer response here>/120
DNS = fdaa:0:33::3 # DNS From https://fly.io/docs/reference/privatenetwork/

[Peer]
PublicKey = <put pubkey from addWireGuardPeer response here>
AllowedIPs = fdaa:0:28a6::/48 # I think this is always hardcoded
Endpoint = <put endpointip from addWireGuardPeer response here>:51820
PersistentKeepalive = 15

Create the wg0 WireGuard interface:

sudo cp wg0.conf /etc/wireguard && \
  wg-quick up wg0

Deploy the app:

fly deploy \
  --env SIAB_USER=jason \
  --env SIAB_PASSWORD=mysecretpassword \
  --env SIAB_SUDO=true

Please let me know if you see anything wrong with my WireGuard config that would result in the peer not being able to see the fly instance. Thanks :slight_smile:

1 Like

This is such great information. Thanks for generating it.

So right off the bat, a problem I see here: you’re using the wrong DNS server. I wonder if we’re giving out the wrong DNS server in the peer configuration API. You’re trying to use fdaa:0:33::3, but your network ID isn’t 0:33; it’s 0:28a6.

Your DNS server on your instance is fdaa::3 — instance DNS servers aren’t network-qualified (an instance can only be in one network, so we try to keep it simple; also, we did instance 6PN before we did WireGuard).

Your DNS server on your WireGuard connection is fdaa:0:28a6::3.

I think a reason this appeared to work on the instance is that your dig syntax is wrong; you need to prepend a @ to the DNS server address, so: dig +short _apps.internal @fdaa:0:28a6::3. I tried with your syntax on my own instance, and any address I put in without a @ appeared to worked fine, but was actually using fdaa::3.

We’re looking into the API thing now. Meanwhile, for your API code: you can just take your WireGuard address, take the first 3 parts (the fdaa and the 0:28a6) and add ::3 to get the DNS server for your connection.

2 Likes

Thanks @thomas, that did the trick! I fixed the DNS line in the WireGuard config and now I am able to list all apps and all WireGuard peers on the network. So creating both the app and WireGuard peer via the API works as expected.

From the fly instance

dig +short txt _apps.internal @fdaa::3
# "tpproxy-nv6p2s"

From the WireGuard peer

dig +short txt _apps.internal @fdaa:0:28a6::3
# "tpproxy-nv6p2s"

I also see that the docs about Private Networking has been updated. Awesome!

Going a bit deeper on this, I want to do the exact same thing (i.e. create the app & WireGuard peer via the API) while also specifying a custom network ID.

I did this by adding the additional network parameter when creating the app + WireGuard peer via the API.

# Set additional environment variables
WG_PEER_NAME="jason-tp12"
WG_PEER_NETWORK="jason-tp12-network"

# Create app with custom network
curl 'https://api.fly.io/graphql' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer ${FLY_API_TOKEN}" \
  --data '{
  "query": "mutation($input: CreateAppInput!){ createApp(input: $input) { app { id name organization { id slug } network runtime regions { name code } } } }",
  "variables": {
    "input": {
      "name": "'"${APP_NAME}"'",
      "runtime": "FIRECRACKER",
      "organizationId": "'"${FLY_ORG_ID}"'",
      "preferredRegion": "'"${FLY_REGION}"'",
      "network": "'"${WG_PEER_NETWORK}"'"
    }
  }
}'; echo

# Add WireGuard peer to the specific network 
curl 'https://api.fly.io/graphql' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer ${FLY_API_TOKEN}" \
  --data '{
  "query": "mutation($input: AddWireGuardPeerInput!){ addWireGuardPeer(input: $input){ clientMutationId endpointip peerip pubkey } }",
  "variables": {
    "input": {
      "organizationId": "'"${FLY_ORG_ID}"'",
      "region": "'"${FLY_REGION}"'",
      "name": "'"${WG_PEER_NAME}"'",
      "pubkey": "'"${WG_PEER_PUBLIC_KEY}"'",
      "network": "'"${WG_PEER_NETWORK}"'"
    }
  }
}'; echo

The problem is that when I interrogate the DNS records (like before), some of them seem to be malformed.

From the fly instance

dig +short txt _apps.internal @fdaa::3                                                                                                               
# "tpproxy-q72ggq"
dig +short txt _peer.internal @fdaa::3                                                                                                               
# ;; Warning: Message parser reports malformed message packet.
dig -t txt _peer.internal @fdaa::3                                                                                                                   
# ;; Warning: Message parser reports malformed message packet.                                                                                                          
#                                                                                                                                                                       
# ; <<>> DiG 9.16.1-Ubuntu <<>> -t txt _peer.internal @fdaa::3                                                                                                          
# ;; global options: +cmd                                                                                                                                               
# ;; Got answer:                                                                                                                                                        
# ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15120                                                                                                             
# ;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0                                                                                                     
# ;; WARNING: recursion requested but not available                                                                                                                     
#                                                                                                                                                                       
# ;; QUESTION SECTION:                                                                                                                                                  
# ;_peer.internal.                        IN      TXT                                                                                                                   
#                                                                                                                                                                       
# ;; Query time: 0 msec                                                                                                                                                 
# ;; SERVER: fdaa::3#53(fdaa::3)                                                                                                                                        
# ;; WHEN: Fri Jun 04 21:56:34 CEST 2021                                                                                                                                
# ;; MSG SIZE  rcvd: 44

From the WireGuard peer:

dig +short txt _apps.internal @fdaa:0:28a6::3
# ;; Warning: Message parser reports malformed message packet.
dig -t txt _apps.internal @fdaa:0:28a6::3
# ;; Warning: Message parser reports malformed message packet.
# 
# ; <<>> DiG 9.11.5-P4-5.1+deb10u5-Raspbian <<>> -t txt _apps.internal @fdaa:0:28a6::3
# ;; global options: +cmd
# ;; Got answer:
# ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18303
# ;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
# ;; WARNING: recursion requested but not available
# 
# ;; QUESTION SECTION:
# ;_apps.internal.			IN	TXT
# 
# ;; Query time: 144 msec
# ;; SERVER: fdaa:0:28a6::3#53(fdaa:0:28a6::3)
# ;; WHEN: Fri Jun 04 21:03:03 BST 2021
# ;; MSG SIZE  rcvd: 44
dig +short txt _peer.internal @fdaa:0:28a6::3
# "jason-tp12"

Again, I am unable to establish a connection between the fly instance and WireGuard peer when using a custom network ID. Any help would be greatly appreciated, thanks :slight_smile:

1 Like

I’m looking at this now; it looks like it might be a bug on our side generating responses for _peer when there are no matching records.

It’s also possible there’s a bug here with custom network IDs and DNS; I’ll catch up with Kurt on what he did here (see how I just threw Kurt under the bus?).

What I can say is that for this network ID, the database that backs our DNS server doesn’t have any peer entries.

1 Like

:laughing:

Thanks @thomas! Looking forward to hear about what you find.

@jdeanwallace we’re having trouble replicating this, would you mind configuring another app + gateway pair and try it out? And when it doesn’t work right, will you just leave it in place so we can see what might be missing?

1 Like

Hi @kurt, sure I’ve deployed another app (tpproxy-wtf9ja) with a custom network (jason-tp13-network) and added a WireGuard peer (jason-tp13) to that network. I’ll leave it up for you to debug. Thanks! :slight_smile:

For completeness, here are the steps to reproduce the issue:

Set some environment variables:

FLY_REGION="lhr"
FLY_ORG_ID="YOUR-FLY-ORG-ID"
FLY_API_TOKEN="YOUR-FLY-API-TOKEN"
APP_NAME="tpproxy-wtf9ja"
WG_PEER_NAME="jason-tp13"
WG_PEER_NETWORK="jason-tp13-network"

Create the app in a custom network:

curl 'https://api.fly.io/graphql' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer ${FLY_API_TOKEN}" \
  --data '{
  "query": "mutation($input: CreateAppInput!){ createApp(input: $input) { app { id name organization { id slug } network runtime regions { name code } } } }",
  "variables": {
    "input": {
      "name": "'"${APP_NAME}"'",
      "runtime": "FIRECRACKER",
      "organizationId": "'"${FLY_ORG_ID}"'",
      "preferredRegion": "'"${FLY_REGION}"'",
      "network": "'"${WG_PEER_NETWORK}"'"
    }
  }
}'; echo

Generate your private & public keys for WireGuard:

wg genkey > wg-test.priv && \
  wg pubkey < wg-test.priv > wg-test.pub && \
  cat wg-test.pub

Set your WireGuard public key (that you just created):

WG_PEER_PUBLIC_KEY="<put contents of wg-test.pub here>"

Add a WireGuard peer to the custom network:

curl 'https://api.fly.io/graphql' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer ${FLY_API_TOKEN}" \
  --data '{
  "query": "mutation($input: AddWireGuardPeerInput!){ addWireGuardPeer(input: $input){ clientMutationId endpointip peerip pubkey } }",
  "variables": {
    "input": {
      "organizationId": "'"${FLY_ORG_ID}"'",
      "region": "'"${FLY_REGION}"'",
      "name": "'"${WG_PEER_NAME}"'",
      "pubkey": "'"${WG_PEER_PUBLIC_KEY}"'",
      "network": "'"${WG_PEER_NETWORK}"'"
    }
  }
}'; echo

Create the WireGuard config file as wg0.conf:

[Interface]
PrivateKey = <put contents of wg-test.priv here>
Address = <put peerip from addWireguardPeer response here>/120
# The DNS is our fly organization network ID + ::3
# Our organization network ID is the first 3 sections of the WireGuard peer IP.
# Source: https://fly.io/docs/reference/privatenetwork/#discovering-apps-through-dns-on-an-instance
DNS = fdaa:0:28a6::3

[Peer]
PublicKey = <put pubkey from addWireGuardPeer response here>
AllowedIPs = fdaa:0:28a6::/48 # Allow all IPs within our org network prefix
Endpoint = <put endpoint IP from addWireGuard response peer here>:51820
PersistentKeepalive = 15

Create the wg0 WireGuard interface:

sudo cp wg0.conf /etc/wireguard && \
  wg-quick up wg0

Create the fly.toml file:

app = "tpproxy-wtf9ja"
kill_signal = "SIGINT"
kill_timeout = 5

# Remove this section to build from the local Dockerfile
[build]
  image = "sspreitzer/shellinabox:latest"

[experimental]
  private_network=true

[[services]]
  internal_port = 4200
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    port = "443"

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    port = "4200"
    restart_limit = 6
    timeout = "2s"

Deploy the app:

fly deploy \
  --env SIAB_USER=jason \
  --env SIAB_PASSWORD=mysecretpassword \
  --env SIAB_SUDO=true

Once on the fly instance, install some tools to debug the network:

sudo -i
apt update && \
  apt install -y net-tools iputils-ping dnsutils netcat socat
dig +short txt _peer.internal @fdaa::3                                                                                                               
# ;; Warning: Message parser reports malformed message packet.
1 Like

@kurt - just want to check whether this is still on your radar

It is! I just went on vacation. I’m digging on this today.

@jdeanwallace I think we found the issue, the peers you created weren’t getting put in the right network (you’ll notice the private ipv6 prefix differed from apps).

It should be fixed now. You can also retrieve the network name from the organization.wireGuardPeers query to double check. You’ll need to create new peers, but I think you’ll be good to go.

Hey @kurt, thanks for taking a look. There seems to be some progress, but it’s still not working as expected.

  1. Querying the DNS from the fly instance now works as expected.
  2. Querying the DNS from the WireGuard peer still doesn’t work as expected.
  3. I still can’t establish a connection between the fly instance and the WireGuard peer.

I recreated the app + WireGuard peer with the following details:

APP_NAME="tpproxy-pmias8"
WG_PEER_NAME="jason-tp14"
WG_PEER_NETWORK="jason-tp14-network"

Yes, I can see the correct network in the response:

{
   "id": "Gj25qXQmnp13RuVZqD0oJ05PDQfbNgDn",
   "name": "jason-tp14",
   "pubkey": "au3o0qFzEhOnixIdL6y7+BS7uYBmhDEAxCyHhJ99LG8=",
   "region": "lhr",
   "peerip": "fdaa:0:28a6:a7b:dc6:0:a:202",
   "network": "jason-tp14-network"
}

From the fly instance (DNS now works as expected):

dig +short txt _apps.internal @fdaa::3
# "tpproxy-pmias8"
dig +short txt _peer.internal @fdaa::3
# "jason-tp14"
# ^^^^^^^^^^ This didn't work before, but works now :)

From the WireGuard peer (Still some unexpected errors):

dig +short txt _apps.internal @fdaa:0:28a6::3
# ;; Warning: Message parser reports malformed message packet.
dig -t txt _apps.internal @fdaa:0:28a6::3
# ;; Warning: Message parser reports malformed message packet.
# 
# ; <<>> DiG 9.11.5-P4-5.1+deb10u5-Raspbian <<>> -t txt _apps.internal @fdaa:0:28a6::3
# ;; global options: +cmd
# ;; Got answer:
# ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48077
# ;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
# ;; WARNING: recursion requested but not available
# 
# ;; QUESTION SECTION:
# ;_apps.internal.			IN	TXT
# 
# ;; Query time: 143 msec
# ;; SERVER: fdaa:0:28a6::3#53(fdaa:0:28a6::3)
# ;; WHEN: Wed Jun 16 21:52:11 BST 2021
# ;; MSG SIZE  rcvd: 44
dig +short txt _peer.internal @fdaa:0:28a6::3
# "interactive-Jasons-MacBook-Pro-XXX,interactive-Jasons-MacBook-Pro-XXX"
# ^^^^ This only outputs peers in the default network. So the "jason-tp14" peer is not listed here :/

I’ll keep the new instance running again, if you want to debug further. Let me know if you need any further details. Thanks :slightly_smiling_face:

Ok I think we found the issue, hold tight.

Also GraphQL makes this a huge pain to test. We really need to get those args in the CLI. :smiley:

So your wireguard peer was landing on the wrong IPv6 block. This should be fixed (again). Will you give it one more try?

You can check this by running fly ips private on your running app, the first 3 IP octets should be the same as your wireguard peer config.

It works! :tada:

fly ips private
# ID       REGION IP
# 15a7d8db lhr    fdaa:0:2e12:a7b:a98:15a7:d8db:2

From the the fly instance:

dig +short txt _apps.internal @fdaa::3
# "tpproxy-ug35e4"
dig +short txt _peer.internal @fdaa::3
# "jason-tp15"

From the WireGuard peer:

dig +short txt _apps.internal @fdaa:0:2e12::3
# "tpproxy-ug35e4"
dig +short txt _peer.internal @fdaa:0:2e12::3
# "jason-tp15"

Thanks @kurt & @thomas for the help & patience.

1 Like

No thank you for trying our experimental unfinished features. This is going to be a really nice thing to release now. :wink:

2 Likes