URGENT: gRPC Server unreachable via grpcurl

I tried setting up gRPC server using the example: http://github.com/fly-apps/grpc-service
The app builds and deploys without any issue and is available as usual at <app_name>.fly.dev

When I try the command as said in docs:

grpcurl -proto hello.proto <app_name>.fly.dev:443 MainService/Hello 

I get the error:

Failed to dial target host "<app_name>.fly.dev:443": remote error: tls: no application protocol

Not able to connect via the gRPC client. Can someone help?
We had to rollback our gRPC server release due to the same reason as we validated with multiple gRPC servers - all facing this same issue!

Resolved:
I had to set handlers: [] in tcp as our gRPC server is insecure.

Resolved:
I had to set handlers: [] in tcp as our gRPC server is insecure.

This is more of a workaround (disabling TLS) than a solution, right?

Because having handlers: ["tls"] should cause the Fly proxy to terminate TLS and then forward a plaintext connection to your service, so it should work with an insecure/plaintext gRPC server - see Deploy gRPC and gRPC-Web Services.

And it seems to have worked until recently - is this a behaviour change or a regression?

This appears to be a regression on our part. We’ll look into it!

We reverted (for other reasons) to a previous release which made the example code work again.

1 Like

I tried deploying a gRPC app, encountered the same issue and found this link.

My app’s a little quirky but I’m confident I’m experiencing the same issue.

I redeployed without tls (handlers[]) and am able to gRPCurl the endpoint but, when I add handlers["tls"], it fails

handlers[]:

grpcurl \
-plaintext \
-import-path ${X} \
-proto ${X}/health.proto \
freddie.fly.dev:10000 \
grpc.health.v1.Health.Check
{
  "status": "SERVING"
}

handlers["tls"]:

grpcurl \
-import-path ${X} \
-proto ${X}/health.proto \
freddie.fly.dev:10000 \
grpc.health.v1.Health.Check

Failed to dial target host “freddie.fly.dev:10000”: remote error: tls: no application protocol

@daz.wilkin Could you post the the full fly.toml too? We’d need to check if the services are set up correctly as well.

Sure!

fly.toml:

kill_signal = "SIGINT"
kill_timeout = 5

[build]
  image = "registry.fly.io/freddie:a3339c154a89286265be205878a777aa1aef9f95"

[experimental]
  exec = [
    "/server",
    "--failure_rate=0.5",
    "--changes_rate=15s",
    "--endpoint=:50051"
  ]
  private_network = true

[[services]]
  internal_port = 50051
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = ["tls"]
    port = "10000"

I tried it with both 443 and 10000 with the same result.

Using handlers = [], worked.

Same issue here.
We critically need this to launch our App.

Pretty urgent. Thank you.