Fly not sending traffic to my apps anymore?

I have 2 apps on fly, both are single machine and use caddy as a server.
So my fly.toml file doesn’t list anything http, just tcp going straight to caddy which handles http / https and getting certificates.

Today as I redeployed an app with some changes, I lost ability to get the site to load.
Since I was unable to figure it out, as a test, I redeployed the second app where there are no code changes and it has since also become fully unavailable.

When I deploy those with Caddy, it tries to setup certs and outputs error messages saying that lets encrypt could not reach the app.

Right now I have dumbed down the app as much as possible to just validate that it receives traffic.

The fly.toml only has:

app = 'testapp'
primary_region = 'sjc'
kill_signal = "SIGINT"
kill_timeout = 5

[[services]]
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    port = 80

  [[services.ports]]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

So the app listens to TCP connections on ports 80 or 443 and both of those should arrive on port 8080 within the machine.

The app has been reduced to a minimal Go app that listens for a TCP connection, writes “Welcome to the server” and shuts off:

package main

import (
	"fmt"
	"net"
	"os"
)

func main() {
	listener, err := net.Listen("tcp", "0.0.0.0:8080")
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	defer listener.Close()

	fmt.Println("Server listening on :8080")

	for {
		conn, err := listener.Accept()
		fmt.Println("new connection")
		if err != nil {
			fmt.Println(err)
			continue
		}
		fmt.Println("handling connection")
		go handleConnection(conn)
	}
}

func handleConnection(conn net.Conn) {
	defer conn.Close()
	fmt.Println("received connection:", conn.LocalAddr(), conn.RemoteAddr())
	fmt.Println("writing to connection")
	_, err := conn.Write([]byte("Welcome to the server\n"))
	fmt.Println("wrote to connection")
	if err != nil {
		fmt.Println("error writing to connection")
		fmt.Println(err)
	}
}

Once deployed, I see periodic connections that seem to be fly’s healthcheck:

2025-02-08T00:21:43.121 app[d8d9e35a22e678] sjc [info] received connection: 172.19.2.114:8080 172.19.2.113:48322

Common IP prefix makes me think this is internal fly traffic.

I can also run nc localhost 8080 from within the machine and see my connection:

2025-02-08T00:23:23.588 app[d8d9e35a22e678] sjc [info] received connection: 127.0.0.1:8080 127.0.0.1:39192

And I do get the Welcome to the server response.

However, nc <public ipv4 of app> 80 results in nothing. The app doesn’t receive a connection at all. Tried on 80, 443, 8080 all give no results.

Very confused at what has happened or changed in fly’s config that suddenly makes my apps unavailable.

1 Like

Hello :wave:

There was an internal change in our platform that caused problems with some apps that use the raw TCP handler (as opposed to terminating TLS / HTTP at our proxy / LB). This was rolled out gradually over the week, but it looks like it only hit your app at the last moment.

We have now temporarily rolled back the change, and your app should be back with another deploy. We will be working to fully address this issue (and any similar ones) before we enable the change again.

2 Likes

I think I may have also been hit by this. I’ve been iteratively building a bit of functionality that uses raw TCP as well, and just about lost my marbles tonight trying to find out why TCP connections seemed to get through and trip Fly Proxy’s autostart yesterday(?) but not today. I had been playing today with making the app private and putting a proxy in another app that would reach it over flycast, so at first I assumed the fly proxy autostart simply didn’t work with flycast traffic, but then I noticed I couldn’t connect with TCP or UDP from a dedicated IPv4 any longer either.

I make enough changes rapidly that I assumed I had borked something, between allocating/releasing IPs and throwing machines with different service configurations up and down. But good to know there may be something larger going on. I’ve since bulldozed that app to start over with one that had a different name, and now connections finally seem to work. There might be more affected customers.

Indeed… I think this puzzle from yesterday may already be a third, actually:

(↑ i.e., no handlers.)

Pretty much exactly how I felt yes!

Thank you all for fixing and updating us. Everything works again now! Keep on doing great work!

2 Likes

Before I get too deep down a hole again, is there possibly any change that was rolled out affecting raw UDP apps too? I still seem to have weird connectivity problems with UDP machines that previously worked fine. I can’t seem to figure out how to connect to them at all via flycast private networking, but even from outside again with a dedicated IPv4 it doesn’t seem to be working like I expect (edit: that was me, not using -k flag on netcat listener). I was able to get netcat to receive packets sent to .internal addresses.

Is it just me? Or are you able to reproduce machines listening on a UDP port not receiving traffic over flycast private/dedicated ipv4?

I had the exact same behavior, and it’s now fixed as well, so it was almost certainly the same issue.

1 Like

Yep, I tried this out of curiousity and can reproduce. UDP is relatively limited on the Fly.io platform—and never actually has worked over Flycast, as far as I know:

When Fly.io refers to their “proxy”, they mean only TCP—and not the eBPF magic that handles the UDP side, :honeybee:.

(I didn’t realize this at first, either…)

Got it – I read the docs, but not that thread, and I don’t think the docs mention that. Sad to find out. I won’t derail the topic further with that though.

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.