Issues with UDP on Machines/AppsV2

[Brandon15811]

Hi, I have a UDP app deployed on appsv1 that works great, but when deployed to appsv2/machines, the UDP responses seem inconsistent.

For testing, I’ve deployed echo servers with AppsV2, Machines, and also to an app that I still have on AppsV1

I’m using https://github.com/fly-apps/udp-echo- as the server for testing, and the command echo -n Response{1..100} | xargs -d ' ' --verbose -I {} bash -c 'echo -n {} | nc -w1 -u ip port; echo' to test echo responses

These are my results from tests that I’ve done from both home and from Google Cloud Shell, to rule out ISP issues.

Responses received/sent from echo server:
(Raw logs here: Fly UDP issues · GitHub)

AppsV1/Home: 100/100

AppsV2/Home: 55/100

AppsV2/Cloudshell: 22/100

Machine/Home: 0/100

Machine/Cloudshell: 33/100

Also, the echo server machines that I deployed to other regions doesn’t seem to be responding either.

AppsV2 was deployed using https://github.com/fly-apps/udp-echo-

Terraform config for the Machine deployment:

terraform {
  required_providers {
    fly = {
      source = "fly-apps/fly"
      version = "0.0.20"
    }
  }
}

provider "fly" {
  useinternaltunnel    = true
  internaltunnelorg    = "pmtest"
  internaltunnelregion = "iad"
}

resource "fly_app" "serverApp" {
  name = "udp-test"
  org  = "pmtest"
}

resource "fly_ip" "serverIp" {
  app        = fly_app.serverApp.name
  type       = "v4"
  depends_on = [fly_app.serverApp]
}

resource "fly_machine" "serverMachine" {
  for_each = {
    1 = "iad",
    2 = "lax",
    3 = "dfw" 
  }

  app    = fly_app.serverApp.name
  region = each.value
  name   = "server-${each.key}"
  image  = "brandon15811/udpechotest:latest" # Built from https://github.com/fly-apps/udp-echo-
  
  services = [{
    ports = [
      {
        port = 1000 * (each.key + 1)
      },
    ]
    protocol      = "udp"
    internal_port = 1000 * (each.key + 1)
  }]
  env = {
    ECHO_PORT = 1000 * (each.key + 1)
  }
  cpus       = 1
  cputype = "shared"
  memorymb   = 1024
  depends_on = [fly_app.serverApp]
}

output "machine_details" {
  value = {
    machines = {
        for machine in fly_machine.serverMachine : machine.name => {
          starting_port = machine.services[0].ports[0].port,
          id           = machine.id

          public_ip = fly_ip.serverIp.address
      }
    }
  }
}

Terraform output:

machine_details = {
  "machines" = {
    "server-1" = {
      "id" = "1781327c90ed08"
      "public_ip" = "137.66.38.64"
      "starting_port" = 2000
    }
    "server-2" = {
      "id" = "4d891d77be6678"
      "public_ip" = "137.66.38.64"
      "starting_port" = 3000
    }
    "server-3" = {
      "id" = "32874ed7ae3685"
      "public_ip" = "137.66.38.64"
      "starting_port" = 4000
    }
  }
}

[ignoramous]

(I’m not sure if this is even valid) By default the number of open UDP “connections” (I guess, in-flight “packets”) is limited to 25. Try setting it to 1000 or something?

[kwaw]

Hey, this should be working now! Can you try again?
New UDP services weren’t getting into our routing layer.

[Brandon15811]

Hi, thank you for the quick response. My AppsV2 app that I deployed is now replying properly, but I’m still having trouble with my machines that are deployed.

On google cloud shell, I’m only receiving responses from my iad machine, and from home, I’m only receiving responses from my lax machine.

I also tried changing the ports on the existing machines, and added 3 new machines.

New terraform output

machine_details = {
  "machines" = {
    "server-1" = {
      "id" = "1781327c90ed08"
      "public_ip" = "137.66.38.64"
      "region" = "iad"
      "starting_port" = 6000
    }
    "server-2" = {
      "id" = "4d891d77be6678"
      "public_ip" = "137.66.38.64"
      "region" = "lax"
      "starting_port" = 7000
    }
    "server-3" = {
      "id" = "32874ed7ae3685"
      "public_ip" = "137.66.38.64"
      "region" = "dfw"
      "starting_port" = 8000
    }
    "server-4" = {
      "id" = "9080ee5da37787"
      "public_ip" = "137.66.38.64"
      "region" = "atl"
      "starting_port" = 9000
    }
    "server-5" = {
      "id" = "91857753f37d83"
      "public_ip" = "137.66.38.64"
      "region" = "bos"
      "starting_port" = 10000
    }
    "server-6" = {
      "id" = "9080e717fd4987"
      "public_ip" = "137.66.38.64"
      "region" = "den"
      "starting_port" = 11000
    }
  }
}

[kwaw]

Hey i’m curious, what are your expectations?

We use anycast routing to direct traffic to machines.
If you’re reaching IAD from gcloud shell, then it means the IAD machine is the closest to your gcp region.
If you’re reaching lax from your home, then your lax machine is the closest to your home.
That’s how the platform works.

[Brandon15811]

Hey kwaw,

Thanks for the explanation, I was doing experiments to see if I could route by port, but the way you explained that it works makes more sense. The routing fix that was applied earlier should be good enough for my use case.

I do have one more question: if I set up multiple machines on the same app in the same region, and give them each a unique port, will it route to the specific machine based on the port, or will it load balance between them regardless of the port?

I did find a reference to UDP port routing here, but I wanted to make sure that was still the case:

Dedicated IP Billing

You can make that happen if you spawn machines manually. You can delegate each machine to listen to one specific port. Do remember that for UDP on Fly you have to listen on the same port you’re exposing for each machine. So maybe this is how’d do: Create a machines app Add a dedicated IPv4 Spawn each server on a machine listening to their own port. Your users would use you-domain.com:machine-port to connect Does that sound like a good enough approach for WebRTC?

[kwaw]

@Brandon15811 It’s going to route based on port.

[Brandon15811]

I’m getting inconsistent responses with the same region different ports configuration. Is there anything I can do to improve it?

Terraform config:

terraform {
  required_providers {
    fly = {
      source = "fly-apps/fly"
      version = "0.0.20"
    }
  }
}

provider "fly" {
  useinternaltunnel    = true
  internaltunnelorg    = "pmtest"
  internaltunnelregion = "iad"
}

resource "fly_app" "serverApp" {
  name = "udp-test3"
  org  = "pmtest"
}

resource "fly_ip" "serverIp" {
  app        = fly_app.serverApp.name
  type       = "v4"
  depends_on = [fly_app.serverApp]
}

resource "fly_machine" "serverMachine" {
  count = 3

  app    = fly_app.serverApp.name
  region = "dfw"
  name   = "server2-${count.index}"
  image  = "brandon15811/udpechotest:latest"
  
  services = [{
    ports = [
      {
        port = 1000 * (count.index + 1)
      },
    ]
    protocol      = "udp"
    internal_port = 1000 * (count.index + 1)
  }]
  env = {
    ECHO_PORT = 1000 * (count.index+ 1)
  }
  cpus       = 1
  cputype = "shared"
  memorymb   = 768
  depends_on = [fly_app.serverApp]
}

output "machine_details" {
  value = {
    machines = {
        for machine in fly_machine.serverMachine : machine.name => {
          starting_port = machine.services[0].ports[0].port,
          id           = machine.id
          region = machine.region
          public_ip = fly_ip.serverIp.address
      }
    }
  }
}

Output:

machine_details = {
  "machines" = {
    "server2-0" = {
      "id" = "9185700db02683"
      "public_ip" = "149.248.222.85"
      "region" = "dfw"
      "starting_port" = 1000
    }
    "server2-1" = {
      "id" = "e2865339fd2586"
      "public_ip" = "149.248.222.85"
      "region" = "dfw"
      "starting_port" = 2000
    }
    "server2-2" = {
      "id" = "32874406a74178"
      "public_ip" = "149.248.222.85"
      "region" = "dfw"
      "starting_port" = 3000
    }
  }
}

[kwaw]

What does “inconsistent responses” mean here? Is your udp service listening on all 3 ports? Are you listening on fly-global-services?

[Brandon15811]

By inconsistent responses, I mean that I get an echo response only sometimes. With an app with a single machine, I get responses 100% of the time. I’m using GitHub - fly-apps/udp-echo-: Sample TCP/UDP Echo Service as my test server, so it is listening to all 3 ports, and on fly-global-services

Logs:

App with multiple machines in same region, different ports (with same terraform config from my previous post):

$ echo -n Response{1..100} | xargs -d ' ' --verbose -I {} bash -c 'echo -n {} | nc -w1 -u 149.248.222.85 2000; echo'
bash -c 'echo -n Response1 | nc -w1 -u 149.248.222.85 2000; echo'
Response1
bash -c 'echo -n Response2 | nc -w1 -u 149.248.222.85 2000; echo'
Response2
bash -c 'echo -n Response3 | nc -w1 -u 149.248.222.85 2000; echo'

bash -c 'echo -n Response4 | nc -w1 -u 149.248.222.85 2000; echo'
Response4
bash -c 'echo -n Response5 | nc -w1 -u 149.248.222.85 2000; echo'

bash -c 'echo -n Response6 | nc -w1 -u 149.248.222.85 2000; echo'

bash -c 'echo -n Response7 | nc -w1 -u 149.248.222.85 2000; echo'

$ echo -n Response{1..100} | xargs -d ' ' --verbose -I {} bash -c 'echo -n {} | nc -w1 -u 149.248.222.85 3000; echo'
bash -c 'echo -n Response1 | nc -w1 -u 149.248.222.85 3000; echo'
Response1
bash -c 'echo -n Response2 | nc -w1 -u 149.248.222.85 3000; echo'

bash -c 'echo -n Response3 | nc -w1 -u 149.248.222.85 3000; echo'

bash -c 'echo -n Response4 | nc -w1 -u 149.248.222.85 3000; echo'

bash -c 'echo -n Response5 | nc -w1 -u 149.248.222.85 3000; echo'
Response5
bash -c 'echo -n Response6 | nc -w1 -u 149.248.222.85 3000; echo'

bash -c 'echo -n Response7 | nc -w1 -u 149.248.222.85 3000; echo'
Response7
$ echo -n Response{1..100} | xargs -d ' ' --verbose -I {} bash -c 'echo -n {} | nc -w1 -u 149.248.222.85 1000
; echo'
bash -c 'echo -n Response1 | nc -w1 -u 149.248.222.85 1000; echo'

bash -c 'echo -n Response2 | nc -w1 -u 149.248.222.85 1000; echo'

bash -c 'echo -n Response3 | nc -w1 -u 149.248.222.85 1000; echo'

bash -c 'echo -n Response4 | nc -w1 -u 149.248.222.85 1000; echo'
Response4
bash -c 'echo -n Response5 | nc -w1 -u 149.248.222.85 1000; echo'

bash -c 'echo -n Response6 | nc -w1 -u 149.248.222.85 1000; echo'
Response6
bash -c 'echo -n Response7 | nc -w1 -u 149.248.222.85 1000; echo'
Response7

App with single machine:

$ echo -n Response{1..100} | xargs -d ' ' --verbose -I {} bash -c 'echo -n {} | nc -w1 -u 149.248.213.177 1000; echo'
bash -c 'echo -n Response1 | nc -w1 -u 149.248.213.177 1000; echo'
Response1
bash -c 'echo -n Response2 | nc -w1 -u 149.248.213.177 1000; echo'
Response2
bash -c 'echo -n Response3 | nc -w1 -u 149.248.213.177 1000; echo'
Response3
bash -c 'echo -n Response4 | nc -w1 -u 149.248.213.177 1000; echo'
Response4
bash -c 'echo -n Response5 | nc -w1 -u 149.248.213.177 1000; echo'
Response5
bash -c 'echo -n Response6 | nc -w1 -u 149.248.213.177 1000; echo'
Response6
bash -c 'echo -n Response7 | nc -w1 -u 149.248.213.177 1000; echo'
Response7

[kwaw]

hey @Brandon15811 sorry but turns out you can do that. it would work sometimes but you clearly can’t rely on it. if you want to load balance based on port, you would have to build your own proxy/router unfortunately.

[Brandon15811]

I can handle doing a proxy, I just didn’t want to start re-implementing platform features if it was available.

Thank you for all the help!