SSE events unexpected error

[joseamica]

2023-06-26T15:14:48Z proxy[080eed5f661048] dfw [error]could not make HTTP request to instance: connection error: timed out

**fly.toml **

# fly.toml app configuration file generated for av on 2023-06-13T14:37:21-06:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = "av"
primary_region = "dfw"
kill_signal = "SIGINT"
kill_timeout = "5s"

[experimental]
auto_rollback = true
enable_consul = true
allowed_public_ports = []

[deploy]
release_command = "bash ./scripts/migrate.sh"


[mounts]
source = "av"
destination = "/data"


[env]
FLY_REGION = "dfw"
METRICS_PORT = "8081"
PORT = "8080"
PRIMARY_REGION = "dfw"

[[services]]
# http_checks = []
script_checks = []
protocol = "tcp"
internal_port = 8080
processes = ["app"]

[[services.ports]]
port = 80
handlers = ["http"]
force_https = true

[[services.ports]]
port = 443
handlers = ["tls", "http"]

[services.concurrency]
type = "connections"
hard_limit = 25
soft_limit = 20

[[services.tcp_checks]]
interval = "15s"
timeout = "2s"
grace_period = "15s"
restart_limit = 0

[[services.http_checks]]
interval = "10s"
timeout = "2s"
grace_period = "5s"
method = "get"
path = "/healthcheck"
protocol = "http"
tls_skip_verify = false
headers = {}
# [services.http_checks.headers]

[metrics]
port = 8081
path = "/metrics"

I am using server send events which register when a user pay, joins and perform some activities. But suddendly this error appears and it wont refresh the data for other users meaning the SSE got some errors, this doesnt happen on my localhost server.

any thoughts?

[joseamica]

Someone knows how to fix this?

[greg]

Hi,

By looking at that error message alone, it looks like the proxy was unable to connect to your app within X seconds (whatever it allows). It might be worth looking at the metrics to see if any resource was overloaded CPU/RAM etc. See if anything might be causing it to be slow and not responding in time.

There may well be a need to use SSE but if not sometimes WebSockets can be an alternative approach. Without knowing more it’s hard to say if that would be better or worse.

[joseamica]

So basically, I need to increment my bandwidth usage?

[greg]

Er … Well I’d say if your app was slow to respond (which appears to be the case, based on that error) look at the metrics for the app. In the dashboard, you can click on it and its “metrics” tab. There you can see a load of graphs for load, memory, connections etc. If you see spikes corresponding with the times you get errors/issues in your app … that would probably indicate a connection.

If the metrics on that page all look fine (low load, minimal memory usage, minimal connections … etc), then it’s probably not that and you’d need to investigate elsewhere. Since there’s no point in paying to increase the CPU and/or RAM if you already have enough.

[greg]

Possibly … but it would depend which metric is the issue.

The events are unlikely to be using much data (aka bandwidth). Events are usually just bit of text. Not sending huge binary data like video files.

If you see a spike in e.g CPU (causing your app to be slower to respond at that same time), the (quick) solution would be to pay for a bigger CPU.

If you see a spike in memory usage, same: get a VM/machine with more RAM.

More resources costs more money, but if your app needs them, that’s what you’d do.

But it all depends on why your app is slow to respond. I’ve no idea why. That’s what you would need to investigate/debug I’m afraid. If your app is taking 11 seconds to respond, that would certainly point to an issue. As 11 seconds is a long time for a web request. And probably explain the time out. You’d need to figure out what the app is doing within those 11 seconds.

[joseamica]

2023-06-27T20:51:43Z app[e286533db74798] dfw [info]GET /events/table/cliuvgygi0006r0fpu776sr3n 200 - - 11090.770 ms

2023-06-27T20:51:43Z proxy[080eed5f661048] dfw [error]could not make HTTP request to instance: connection error: timed out

apparently before the error, without reason the app make a get request on /events/table/ and then the time out error.

this is my splat route on my code:

export const loader = ({request, params}: LoaderArgs) => {
  const path = `/${params['*']}`

  return eventStream(request.signal, send => {
    const handler = (message: string) => {
      if (message) {
        send({event: 'message', data: message})
      } else {
        send({data: Date.now().toString()})
      }
    }

    emitter.addListener(path, handler)
    return () => {
      emitter.removeListener(path, handler)
    }
  })
}

[greg]

No idea, I’m afraid.

But it sounds like you are now on track with debugging: figure out why that call is happening and what your app is actually doing in those 11s. That seems like a long time, at least to me. Most web requests are in milliseconds. But it may not be. All depends.