We need to write small guides for using flyctl to do things almost every app needs to do. In particular:
I feel like there’s a ton of power in flyctl that no one knows about. A simple docs page with quick hits would do wonders when people ask how to do things. Will you all spend like 15 min writing a blurb like “how to with flyctl” in this topic?
For a lot of people, the idea of a container is that it’s sort of hermetically sealed; you aren’t mean to peek inside and muck with a running instance, and if you want to make a change, you replace the running container with another newly built container.
But not all apps are built that way, and even for cattle-fleet apps running on ephemeral containers, sometimes the fastest way to debug a problem is to pop a shell.
You can easily SSH into any of your instances with flyctl. Here’s how.
flyctl
Now’s a good time to run flyctl version update. What you want is a flyctl that has the flyctl ssh console command.
I know, we’re asking if it’s plugged in. But: is it plugged in? Try flyctl apps list. If you can see your app, you should be able to log in.
You can deploy a bunch of different apps to Fly from the same host, so we need to know which one you want to talk to. flyctl will read the fly.toml file for your app from the current directory, so that’s one way to pick an app to log into.
This is going to be anticlimactic:
Run flyctl ssh console -s. You’ll see a prompt like this:
> $ flyctl ssh console -s
Looking up regions in DNS... complete!
? Select instance: [Use arrows to move, type to filter]
> lhr (fdaa:0:18:a7b:a98:1e25:4045:2)
lhr (fdaa:0:18:a7b:a98:b208:7c5c:2)
ord (fdaa:0:18:a7b:60:bc71:6132:2)
gru (fdaa:0:18:a7b:60:9b4:1fdb:2)
Pick the instance you want to log in to, and then:
Looking up regions in DNS... complete!
? Select instance: lhr (fdaa:0:18:a7b:a98:8697:8018:2)
Connecting to [fdaa:0:18:a7b:a98:1e25:4045:2]... complete
# hostname
1e254045
#
You’ve got a shell. Use it in good health.
If you don’t want to bother selecting a specific node, you can run flyctl ssh console (without the -s) and we’ll pick one for you.
On the other hand: if you know exactly what region you want to log in to, you can run (say) flyctl ssh console lhr.myapp.internal (all your apps have “app-name.internal” DNS names).
A bunch of stuff is happening under the hood here, and you can read more about it if you want. But you don’t have to, if all you need is a shell so you can get your work done.
@kurt: roughly that?
Yes!
Most apps running on Fly will expose some kind of service, usually (but not always) HTTP, to the public Internet. That’s kind of what we’re all about. But did you know you can also expose private services to your developers and ops teams? It’s easy if you try.
We’re going to walk through connecting your organizations — all the applications you’re running — with WireGuard, and how to expose a service only WireGuard can talk to. This is a great way to set up admin interfaces and ops services, giving you a cryptographically secured network path.
Let’s assume for starters that you’re using a MacBook. Later, we’ll talk about how to get this working on Linux (it’s actually easier).
flyctl
Now’s a good time to run flyctl version update. What you want is a flyctl that has the flyctl wireguard command.
I know, we’re asking if it’s plugged in. But: is it plugged in? Try flyctl apps list. If you can see your app, you should be able to set up WireGuard.
It’s in the app store. Jason Donenfeld, the author of WireGuard and the hardest working person in show business, manages the project himself. It’s great.
When it’s installed, the goofy WireGuard snake will appear in your menu bar. That’s how you know it’s working. We’re moving on.
This sounds intimidating, but we’re going to do all the work for you.
Do flyctl wireguard create. You’ll see something like:
> $ flyctl wireguard create
? Select organization: [Use arrows to move, type to filter]
> Thomas Ptacek (personal)
I’m picking my personal organization, but you should pick the organization where all your apps are.
Now flyctl has a question for us:
? Select organization: Thomas Ptacek (personal)
? Region in which to add WireGuard peer: ord
Fly runs WireGuard endpoints in a bunch of different places around the world, so you can pick where you connect to our network. If you don’t care, ord (Chicago), iad (Dallas) and fra (Frankfort) are good bets.
One last question:
> $ flyctl wireguard create
? Select organization: Thomas Ptacek (personal)
? Region in which to add WireGuard peer: ord
? New DNS name for WireGuard peer: my-new-network
Give your WireGuard connection a name. We’ll link it to the DNS so your apps can find you (more on that later). This isn’t a life-or-death decision; you can quickly delete and recreate a WireGuard connection if you want to change this, or create a bunch of different ones. Go nuts!
Now, something happens:
> $ flyctl wireguard create
? Select organization: Thomas Ptacek (personal)
? Region in which to add WireGuard peer: ord
? New DNS name for WireGuard peer: my-new-net
Creating WireGuard peer "my-new-net" in region "ord" for organization personal
!!!! WARNING: Output includes private key. Private keys cannot be recovered !!!!
!!!! after creating the peer; if you lose the key, you'll need to remove !!!!
!!!! and re-add the peering connection. !!!!
? Filename to store WireGuard configuration in, or 'stdout':
This is much less scary than it looks; the thing to know here is that if you lose this configuration, we can’t regenerate it for you (we generate your keys clientside). Which isn’t the end of the world! You can delete the connection and make a new one.
Click the weird looking snake in your menu bar. Click “Import Tunnels From File”. Import your tunnel, from file.
The new tunnel should appear in a list in the WireGuard menu drop-down. Click it to connect. It should connect quickly!
You now have connectivity to your organization.
If you look at your new WireGuard configuration, you’ll see it includes a DNS server address. You can use that DNS server to look up names for your applications running on Fly; they’ll all have hostnames following the pattern {my-app}.internal. You can target specific regions with… you guessed it… {region}.{my-app}.internal.
All of these hosts will have IPv6 addresses — welcome to the world of tomorrow! — that follow a pattern starting in fdaa::*. As long as your WireGuard connection is up and running, you should be able to talk to any of these addresses.
Each instance of your app running on Fly has a special alias address on that IPv6 fdaa::* network we just talked about. We record it in /etc/hosts as fly-local-6pn.
So the trick to exposing a service to WireGuard, and keeping it off the Internet, is to bind/listen on that fly-local-6pn service. So for instance, if you have an admin console running on port 8443, you’d want to bind it to fly-local-6pn:8443. How you do this will depend on your development environment.
Modern Linux has WireGuard built it. Modern Linux is great! If you’re running earlier WireGuard, you can install it; for instance, on Ubuntu, you can apt install -y wireguard.
Once you’ve got WireGuard installed, just repeat the steps above, and save your WireGuard configuration in a file (say, newconfig.conf). Then it should just be wg-quick up newconfig.conf to connect.
We told you it was easier!
We keep running into DNS issue when people try and use <appname>.internal. I don’t understand how that magic works, but is it possible that something like an Elixir app’s internal resolution will fail on that? I’ve only been able to get migrations working with postgres://[<ipv6>]:5432/ from my laptop.
Yeah I’m also fuzzy on how macOS resolver configuration works. I can dig into it later.
Probably not a huge deal. The right way to run migrations is with this release_command we haven’t shipped yet …
Rough list of things to use when an app is misbehaving…
Use flyctl status to view current status of your app, the most recent deployment, and running VMs.
An app status of pending means the app hasn’t been deployed yet. dead means the app failed to start before the number of attempts was exhausted. running means the app is deployed.
The list of VMs includes the following fields:
⇡ glyph beside the number$ flyctl status
App
Name = your-app
Owner = fly
Version = 26
Status = running
Hostname = your-app.fly.dev
Deployment Status
ID = ba9f0a38-b82c-412a-179d-9adfad0e84a8
Version = v26
Status = successful
Description = Deployment completed successfully
Instances = 16 desired, 16 placed, 16 healthy, 0 unhealthy
Instances
ID VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED
715398d0 26 ord run running 3 total, 3 passing 0 9h46m ago
af2e3097 26 dfw run running 3 total, 3 passing 0 2021-03-25T08:20:43Z
c6345080 26 fra run running 3 total, 3 passing 3 2021-03-24T05:58:54Z
Only VMs that are currently running, starting, or recently terminated are listed. Use the --all flag to show all terminated VMs for the past 12 hours.
You can view logs from your app in real time with flyctl logs. By default, this includes logs from all VMs as well as platform events, such as health checks, deployments. Each line includes the VM id and region.
$ flyctl logs
2021-03-26T21:00:22.972Z 60a0c6e4 vin [info] 2021/03/26 20:59:47 51.81.106.41:57504 HEAD /
2021-03-26T21:00:22.974Z d4dd8ba5 vin [info] 2021/03/26 21:00:44 51.81.57.174:40858 HEAD /
2021-03-26T21:00:22.974Z 05a93924 ewr [info] 2021/03/26 21:00:03 136.144.58.155:41166 HEAD /
2021-03-26T21:00:22.975Z e096415a yyz [info] 2021/03/26 21:00:13 162.248.220.154:46548 HEAD /
2021-03-26T21:00:22.976Z 587d7cfc atl [info] 2021/03/26 20:59:51 86.109.8.85:44842 HEAD /
2021-03-26T21:00:22.977Z bfc3f74d vin [info] 2021/03/26 20:59:42 51.81.106.154:52822 HEAD /
2021-03-26T21:00:22.977Z 60a0c6e4 vin [info] 2021/03/26 20:59:47 51.81.106.41:57506 HEAD /
You can filter logs down to VMs in a single region with the --region flag or to a single VM with the --i flag.
You can view detailed status for a single VM using the flyctl vm status command. This works for both running and terminated VMs.
The Recent Events section shows a list of platform events, such as launch and restarts, and the time they took place.
The Checks section lists the configured health checks and their state. The check output is useful to determine why a check failed.
The Recent Logs section shows the most recent logs from the VM. You can see more by using the flyctl logs -i {vm-id} command.
$ flyctl vm status af2e3097
Instance
ID = af2e3097
Version = 26
Region = dfw
Desired = run
Status = running
Health Checks = 3 total, 3 passing
Restarts = 0
Created = 2021-03-15T18:51:46Z
Recent Events
TIMESTAMP TYPE MESSAGE
2021-03-15T18:51:41Z Received Task received by client
2021-03-15T18:51:45Z Task Setup Building Task Directory
2021-03-15T18:51:48Z Started Task started by client
Checks
ID SERVICE STATE OUTPUT
e33876d24774487cb427304deeef5bdd tcp-3000 passing HTTP GET http://172.19.0.98:3000/status: 200 OK Output: ahoyhoy
68ffe271b417d0b50c730de164e1951c tcp-8080 passing TCP connect 172.19.0.58:8080: Success
private-dns app passing [✓] lookupAppNames _apps.internal
Recent Logs
2021-03-26T20:46:53Z [info] 2021/03/26 20:45:30 139.178.86.253:33618 HEAD /
2021-03-26T20:47:01Z [info] 2021/03/26 20:45:38 139.178.86.253:35124 GET /
2021-03-26T20:47:11Z [info] 2021/03/26 20:45:48 139.178.86.253:37152 GET /
2021-03-26T20:47:21Z [info] 2021/03/26 20:45:58 139.178.86.253:38986 GET /
You can restart all running VMs in your app with flyctl restart. They will restart one at a time with a several second delay until completed.
$ flyctl restart
your-app is being restarted
You can also restart VMs individually.
$ flyctl vm restart 2ab81897
VM 2ab81897 is being restarted
You can stop a VM entirely with flyctl vm stop. After the VM stops, your app’s placement and scaling configuration might create a new VM to replace the stopped on.
$ flyctl vm stop 2ab81897
VM 2ab81897 is being stopped
Whoah, TIL flyctl vm restart <alloc>.