URGENT: gRPC Server unreachable via grpcurl

Questions / Help
, ,
1

I tried setting up gRPC server using the example: http://github.com/fly-apps/grpc-service
The app builds and deploys without any issue and is available as usual at <app_name>.fly.dev

When I try the command as said in docs:

grpcurl -proto hello.proto <app_name>.fly.dev:443 MainService/Hello

I get the error:

Failed to dial target host "<app_name>.fly.dev:443": remote error: tls: no application protocol

Not able to connect via the gRPC client. Can someone help?
We had to rollback our gRPC server release due to the same reason as we validated with multiple gRPC servers - all facing this same issue!

3

Resolved:
I had to set handlers: [] in tcp as our gRPC server is insecure.

4

Resolved:
I had to set handlers: in tcp as our gRPC server is insecure.

This is more of a workaround (disabling TLS) than a solution, right?

Because having handlers: ["tls"] should cause the Fly proxy to terminate TLS and then forward a plaintext connection to your service, so it should work with an insecure/plaintext gRPC server - see Deploy gRPC and gRPC-Web Services · Fly Docs.

And it seems to have worked until recently - is this a behaviour change or a regression?

5

This appears to be a regression on our part. We’ll look into it!

We reverted (for other reasons) to a previous release which made the example code work again.

1 Like
6

I tried deploying a gRPC app, encountered the same issue and found this link.

My app’s a little quirky but I’m confident I’m experiencing the same issue.

I redeployed without tls (handlers[]) and am able to gRPCurl the endpoint but, when I add handlers["tls"], it fails

handlers[]:

grpcurl \
-plaintext \
-import-path ${X} \
-proto ${X}/health.proto \
freddie.fly.dev:10000 \
grpc.health.v1.Health.Check
{
  "status": "SERVING"
}

handlers["tls"]:

grpcurl \
-import-path ${X} \
-proto ${X}/health.proto \
freddie.fly.dev:10000 \
grpc.health.v1.Health.Check

Failed to dial target host “freddie.fly.dev:10000”: remote error: tls: no application protocol

7

@DazWilkin Could you post the the full fly.toml too? We’d need to check if the services are set up correctly as well.

8

Sure!

fly.toml:

kill_signal = "SIGINT"
kill_timeout = 5

[build]
  image = "registry.fly.io/freddie:a3339c154a89286265be205878a777aa1aef9f95"

[experimental]
  exec = [
    "/server",
    "--failure_rate=0.5",
    "--changes_rate=15s",
    "--endpoint=:50051"
  ]
  private_network = true

[[services]]
  internal_port = 50051
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = ["tls"]
    port = "10000"

I tried it with both 443 and 10000 with the same result.

Using handlers = [], worked.

10

Same issue here.
We critically need this to launch our App.

Pretty urgent. Thank you.

11

Although this is not a priority for me (I’m using GCP), I wonder whether Fly publishes issues|features lists?

It would be useful to get an indication as to when secure gRPC will be supported as I’d like to include Fly in my solution as a deployment target.

12

This might work now, actually. I think the root problem is that we don’t send the h2 alpn when there’s no http handler.

Will you try updating your fly.toml services block to look like this?

  [[services.ports]]
    handlers = ["tls"]
    tls_options = {"alpn":["h2"]}
    port = "443"
13

Excellent! You bet!

I’ll try it tomorrow morning.

Thank you

14

No change :frowning_face:

It disliked {"alpn":["h2"]} but accepted (!?) {"alpn"=["h2"]} and this appears to configure the app with this setting:

flyctl config display --app=${APP}

Yields:

"ports": [
  {
    "handlers": [
      "tls"
    ],
    "port": 10000,
    "tls_options": {
      "alpn": [
        "h2"
      ],
      "default_self_signed": false,
      "versions": []
    }
  }
],