Guides for common flyctl patterns

We need to write small guides for using flyctl to do things almost every app needs to do. In particular:

  1. Connect to wireguard and run migrations
  2. Get a console on your app
  3. (Possibly) remote debugging

I feel like there’s a ton of power in flyctl that no one knows about. A simple docs page with quick hits would do wonders when people ask how to do things. Will you all spend like 15 min writing a blurb like “how to with flyctl” in this topic?

SSH into a Fly Instance

For a lot of people, the idea of a container is that it’s sort of hermetically sealed; you aren’t mean to peek inside and muck with a running instance, and if you want to make a change, you replace the running container with another newly built container.

But not all apps are built that way, and even for cattle-fleet apps running on ephemeral containers, sometimes the fastest way to debug a problem is to pop a shell.

You can easily SSH into any of your instances with flyctl. Here’s how.

Make Sure You’ve Got A Recent flyctl

Now’s a good time to run flyctl version update. What you want is a flyctl that has the flyctl ssh console command.

Make Sure You’re Logged In

I know, we’re asking if it’s plugged in. But: is it plugged in? Try flyctl apps list. If you can see your app, you should be able to log in.

(Optionally) Be In Your App Directory

You can deploy a bunch of different apps to Fly from the same host, so we need to know which one you want to talk to. flyctl will read the fly.toml file for your app from the current directory, so that’s one way to pick an app to log into.

Log In With SSH

This is going to be anticlimactic:

Run flyctl ssh console -s. You’ll see a prompt like this:

> $ flyctl ssh console -s                                                                                                       
Looking up regions in DNS... complete!
? Select instance:  [Use arrows to move, type to filter]
> lhr (fdaa:0:18:a7b:a98:1e25:4045:2)
  lhr (fdaa:0:18:a7b:a98:b208:7c5c:2)
  ord (fdaa:0:18:a7b:60:bc71:6132:2)
  gru (fdaa:0:18:a7b:60:9b4:1fdb:2)

Pick the instance you want to log in to, and then:

Looking up regions in DNS... complete!
? Select instance: lhr (fdaa:0:18:a7b:a98:8697:8018:2)
Connecting to [fdaa:0:18:a7b:a98:1e25:4045:2]... complete
# hostname
1e254045
# 

You’ve got a shell. Use it in good health.

If you don’t want to bother selecting a specific node, you can run flyctl ssh console (without the -s) and we’ll pick one for you.

On the other hand: if you know exactly what region you want to log in to, you can run (say) flyctl ssh console lhr.myapp.internal (all your apps have “app-name.internal” DNS names).

A bunch of stuff is happening under the hood here, and you can read more about it if you want. But you don’t have to, if all you need is a shell so you can get your work done.

1 Like

@kurt: roughly that?

Yes!

Connect To Private Services With WireGuard

Most apps running on Fly will expose some kind of service, usually (but not always) HTTP, to the public Internet. That’s kind of what we’re all about. But did you know you can also expose private services to your developers and ops teams? It’s easy if you try.

We’re going to walk through connecting your organizations — all the applications you’re running — with WireGuard, and how to expose a service only WireGuard can talk to. This is a great way to set up admin interfaces and ops services, giving you a cryptographically secured network path.

Let’s assume for starters that you’re using a MacBook. Later, we’ll talk about how to get this working on Linux (it’s actually easier).

Make Sure You’ve Got A Recent flyctl

Now’s a good time to run flyctl version update. What you want is a flyctl that has the flyctl wireguard command.

Make Sure You’re Logged In

I know, we’re asking if it’s plugged in. But: is it plugged in? Try flyctl apps list. If you can see your app, you should be able to set up WireGuard.

Install WireGuard

It’s in the app store. Jason Donenfeld, the author of WireGuard and the hardest working person in show business, manages the project himself. It’s great.

When it’s installed, the goofy WireGuard snake will appear in your menu bar. That’s how you know it’s working. We’re moving on.

Generate A WireGuard Configuration

This sounds intimidating, but we’re going to do all the work for you.

Do flyctl wireguard create. You’ll see something like:

> $ flyctl wireguard create                                                                                                               
? Select organization:  [Use arrows to move, type to filter]
> Thomas Ptacek (personal)

I’m picking my personal organization, but you should pick the organization where all your apps are.

Now flyctl has a question for us:

? Select organization: Thomas Ptacek (personal)
? Region in which to add WireGuard peer:  ord

Fly runs WireGuard endpoints in a bunch of different places around the world, so you can pick where you connect to our network. If you don’t care, ord (Chicago), iad (Dallas) and fra (Frankfort) are good bets.

One last question:

> $ flyctl wireguard create                                                                                                               
? Select organization: Thomas Ptacek (personal)
? Region in which to add WireGuard peer:  ord
? New DNS name for WireGuard peer:  my-new-network

Give your WireGuard connection a name. We’ll link it to the DNS so your apps can find you (more on that later). This isn’t a life-or-death decision; you can quickly delete and recreate a WireGuard connection if you want to change this, or create a bunch of different ones. Go nuts!

Now, something happens:

> $ flyctl wireguard create                                                                                                               
? Select organization: Thomas Ptacek (personal)
? Region in which to add WireGuard peer:  ord
? New DNS name for WireGuard peer:  my-new-net
Creating WireGuard peer "my-new-net" in region "ord" for organization personal

!!!! WARNING: Output includes private key. Private keys cannot be recovered !!!!
!!!! after creating the peer; if you lose the key, you'll need to remove    !!!!
!!!! and re-add the peering connection.                                     !!!!
? Filename to store WireGuard configuration in, or 'stdout':  

This is much less scary than it looks; the thing to know here is that if you lose this configuration, we can’t regenerate it for you (we generate your keys clientside). Which isn’t the end of the world! You can delete the connection and make a new one.

Add The Configuration To macOS WireGuard

Click the weird looking snake in your menu bar. Click “Import Tunnels From File”. Import your tunnel, from file.

Connect

The new tunnel should appear in a list in the WireGuard menu drop-down. Click it to connect. It should connect quickly!

You now have connectivity to your organization.

If you look at your new WireGuard configuration, you’ll see it includes a DNS server address. You can use that DNS server to look up names for your applications running on Fly; they’ll all have hostnames following the pattern {my-app}.internal. You can target specific regions with… you guessed it… {region}.{my-app}.internal.

All of these hosts will have IPv6 addresses — welcome to the world of tomorrow! — that follow a pattern starting in fdaa::*. As long as your WireGuard connection is up and running, you should be able to talk to any of these addresses.

Expose A Service On Your App To WireGuard

Each instance of your app running on Fly has a special alias address on that IPv6 fdaa::* network we just talked about. We record it in /etc/hosts as fly-local-6pn.

So the trick to exposing a service to WireGuard, and keeping it off the Internet, is to bind/listen on that fly-local-6pn service. So for instance, if you have an admin console running on port 8443, you’d want to bind it to fly-local-6pn:8443. How you do this will depend on your development environment.

Getting This To Work On Linux

Modern Linux has WireGuard built it. Modern Linux is great! If you’re running earlier WireGuard, you can install it; for instance, on Ubuntu, you can apt install -y wireguard.

Once you’ve got WireGuard installed, just repeat the steps above, and save your WireGuard configuration in a file (say, newconfig.conf). Then it should just be wg-quick up newconfig.conf to connect.

We told you it was easier!

2 Likes

We keep running into DNS issue when people try and use <appname>.internal. I don’t understand how that magic works, but is it possible that something like an Elixir app’s internal resolution will fail on that? I’ve only been able to get migrations working with postgres://[<ipv6>]:5432/ from my laptop.

Yeah I’m also fuzzy on how macOS resolver configuration works. I can dig into it later.

Probably not a huge deal. The right way to run migrations is with this release_command we haven’t shipped yet …

Rough list of things to use when an app is misbehaving…

Debugging a deployed app

View app status

Use flyctl status to view current status of your app, the most recent deployment, and running VMs.

An app status of pending means the app hasn’t been deployed yet. dead means the app failed to start before the number of attempts was exhausted. running means the app is deployed.

The list of VMs includes the following fields:

  • ID: the unique id of this VM
  • Version: the auto-incrementing version of this VM’s configuration. It increases If this version isn’t the most recent version you’ll see a glyph beside the number
  • Desired: the state this VM is supposed to be in
  • Status: the state this VM is actually in
  • Health Checks: a summary of health check states for this VM
  • Restarts: the number of times this VM has been restarted. This includes both restarts from failing health checks and manual restarts
$ flyctl status
App
  Name     = your-app          
  Owner    = fly            
  Version  = 26             
  Status   = running        
  Hostname = your-app.fly.dev  

Deployment Status
  ID          = ba9f0a38-b82c-412a-179d-9adfad0e84a8            
  Version     = v26                                            
  Status      = successful                                      
  Description = Deployment completed successfully               
  Instances   = 16 desired, 16 placed, 16 healthy, 0 unhealthy  

Instances
ID       VERSION REGION DESIRED STATUS  HEALTH CHECKS      RESTARTS CREATED              
715398d0 26      ord    run     running 3 total, 3 passing 0        9h46m ago            
af2e3097 26      dfw    run     running 3 total, 3 passing 0        2021-03-25T08:20:43Z 
c6345080 26      fra    run     running 3 total, 3 passing 3        2021-03-24T05:58:54Z 

Only VMs that are currently running, starting, or recently terminated are listed. Use the --all flag to show all terminated VMs for the past 12 hours.

View app logs

You can view logs from your app in real time with flyctl logs. By default, this includes logs from all VMs as well as platform events, such as health checks, deployments. Each line includes the VM id and region.

$ flyctl logs
2021-03-26T21:00:22.972Z 60a0c6e4 vin [info] 2021/03/26 20:59:47 51.81.106.41:57504 HEAD /
2021-03-26T21:00:22.974Z d4dd8ba5 vin [info] 2021/03/26 21:00:44 51.81.57.174:40858 HEAD /
2021-03-26T21:00:22.974Z 05a93924 ewr [info] 2021/03/26 21:00:03 136.144.58.155:41166 HEAD /
2021-03-26T21:00:22.975Z e096415a yyz [info] 2021/03/26 21:00:13 162.248.220.154:46548 HEAD /
2021-03-26T21:00:22.976Z 587d7cfc atl [info] 2021/03/26 20:59:51 86.109.8.85:44842 HEAD /
2021-03-26T21:00:22.977Z bfc3f74d vin [info] 2021/03/26 20:59:42 51.81.106.154:52822 HEAD /
2021-03-26T21:00:22.977Z 60a0c6e4 vin [info] 2021/03/26 20:59:47 51.81.106.41:57506 HEAD /

You can filter logs down to VMs in a single region with the --region flag or to a single VM with the --i flag.

View VM status

You can view detailed status for a single VM using the flyctl vm status command. This works for both running and terminated VMs.

The Recent Events section shows a list of platform events, such as launch and restarts, and the time they took place.

The Checks section lists the configured health checks and their state. The check output is useful to determine why a check failed.

The Recent Logs section shows the most recent logs from the VM. You can see more by using the flyctl logs -i {vm-id} command.

$ flyctl vm status af2e3097
Instance
  ID            = af2e3097              
  Version       = 26                    
  Region        = dfw                   
  Desired       = run                   
  Status        = running               
  Health Checks = 3 total, 3 passing    
  Restarts      = 0                     
  Created       = 2021-03-15T18:51:46Z  

Recent Events
TIMESTAMP            TYPE       MESSAGE                 
2021-03-15T18:51:41Z Received   Task received by client 
2021-03-15T18:51:45Z Task Setup Building Task Directory 
2021-03-15T18:51:48Z Started    Task started by client  

Checks
ID                               SERVICE  STATE   OUTPUT                                                                                                                                                                                                                                                                                                                                                           
e33876d24774487cb427304deeef5bdd tcp-3000 passing HTTP GET http://172.19.0.98:3000/status: 200 OK Output: ahoyhoy
68ffe271b417d0b50c730de164e1951c tcp-8080 passing TCP connect 172.19.0.58:8080: Success         
private-dns                      app      passing [✓] lookupAppNames _apps.internal

Recent Logs
  2021-03-26T20:46:53Z [info] 2021/03/26 20:45:30 139.178.86.253:33618 HEAD /
  2021-03-26T20:47:01Z [info] 2021/03/26 20:45:38 139.178.86.253:35124 GET /
  2021-03-26T20:47:11Z [info] 2021/03/26 20:45:48 139.178.86.253:37152 GET /
  2021-03-26T20:47:21Z [info] 2021/03/26 20:45:58 139.178.86.253:38986 GET /

Restart all VMs

You can restart all running VMs in your app with flyctl restart. They will restart one at a time with a several second delay until completed.

$ flyctl restart
your-app is being restarted

Restart a VM

You can also restart VMs individually.

$ flyctl vm restart 2ab81897
VM 2ab81897 is being restarted

Stop a VM

You can stop a VM entirely with flyctl vm stop. After the VM stops, your app’s placement and scaling configuration might create a new VM to replace the stopped on.

$ flyctl vm stop 2ab81897
VM 2ab81897 is being stopped
2 Likes

Whoah, TIL flyctl vm restart <alloc>.

1 Like