Hard time setting up grpc server

1

I’m having a hard time setting up my GRPC server, the logs on the fly machine looks like this:

Pulling container image registry.fly.io/XXX
Successfully prepared image registry.fly.io/XXX 133.207199ms)
Configuring firecracker
Pulling container image registry.fly.io/XXX
Successfully prepared image registry.fly.io/XXX 144.98423ms)
Configuring firecracker

I’ve main looks like this

async def serve():
    address = '[::]:8080'
    server = grpc.aio.server()
    calls_pb2_grpc.add_CallsServiceServicer_to_server(CallsService(), server)
    server.add_insecure_port(address)
    print(f'server starting on {address}')
    await server.start()
    await server.wait_for_termination()


import asyncio
if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(serve())

literally do not even see that print statement anywhere and when I try to use postman to make a request to the server I get a “Service unavailable” error on postman, no log on the fly machine.
This is what my fly toml looks like but I think its fine :

[http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[[vm]]
  memory = '1gb'
  cpu_kind = 'shared'
  cpus = 1


[[services]]
  internal_port = 8080 
  protocol = "tcp"

  [[services.ports]]
    handlers = ["tls"]
    port = 443
    tls_options = { "alpn" = ["h2"] }
2

Hi @AJ1—based on the logs you shared, your Machine may be failing to boot before it’s even able to start your application. Otherwise, I’d expect to see a line like

2024-04-22T19:11:51Z app[xxxxxxxxxxxxxx] ord [info] INFO Starting init (commit: 65db7f7)...

in your logs.

It’s possible that this is an issue on our side, so I’d be happy to look into it if you’re willing to provide the app name or a Machine ID. Feel free to send them to support@fly.io if you’re not comfortable sharing them here—just follow up with me here too so that I know to look for it.

3

Hi @MatthewIngwersen. Thanks - I tried emailling support@fly.io but got a:
This is an unmonitored support mailbox
automated response.

4

@AJ1 sorry about that—the inbox is not actively monitored, but I can still find your email there if I look for it.

Based on the logs you sent, it seems like the Machines are actually booting fine, all the way into your app. So I think the issue may actually be with the service configuration in your fly.toml: you have both an [http_service] and a [[services]] block that conflict with each other (both define a service on port 443). I’m assuming that you added the [[services]] block since gRPC requires a raw TCP service—can you try removing the [http_service] and redeploying?

1 Like
5

Ohh I see, I modified the fly.toml to this:

app = XXX
primary_region = XXX

[build]

# VM def

[[services]]
  internal_port = 8080 
  protocol = "tcp"

  [[services.ports]]
    handlers = ["tls"]
    port = 443
    tls_options = { "alpn" = ["h2"] }

redeployed but still can’t query the server.
I’m using postman and am getting a * Service unavailable* error

6

Hi @MatthewIngwersen, I still can’t make requests to the server - maybe there is an issue with using asyncio with fly io - I get the errors:

Traceback (most recent call last):
  File "//main.py", line 34, in <module>
    loop.run_until_complete(serve())
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 607, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1884, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/local/lib/python3.11/selectors.py", line 468, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
KeyboardInterrupt

whenever I deploy

7

Hi @AJ1—by default, we shut down your application by sending it an interrupt signal (SIGINT). Python converts that signal into a KeyboardInterrupt exception, which I think is why you’re seeing it when you deploy (it’s the Machine shutting down before restarting). The same will probably happen if your Machine is automatically stopped (auto_stop_machines = true in fly.toml).

I just double-checked on our end, and it looks like your app has a shared IPv4 address. Since gRPC requires you to set up a raw TCP service (which you’ve done), you’ll need a dedicated IPv4 address to be able to access your app over IPv4. The link will show you how to allocate one; just be aware that dedicated IPv4 addresses are $2/month.

Otherwise, the stack trace does indicate that your app is starting and reaching the serve function, so I’m not sure what else might be an issue. (Is it still true that you do not see the server starting on {address} line printed in the logs?)

8

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.