# URGENT: gRPC Server unreachable via grpcurl

**URL:** <https://community.fly.io/t/urgent-grpc-server-unreachable-via-grpcurl/2694>\
**Category:** Questions / Help\
**Tags:** example-app, proxy\
**Created:** [October 2, 2021, 5:49pm UTC](https://community.fly.io/t/urgent-grpc-server-unreachable-via-grpcurl/2694 "2021-10-02T17:49:47Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![skimhq](https://avatars.discourse-cdn.com/v4/letter/s/8e8cbc/32.png) [@skimhq](https://community.fly.io/u/skimhq)\
**Post date:** [October 2, 2021, 5:49pm UTC](https://community.fly.io/t/urgent-grpc-server-unreachable-via-grpcurl/2694/1 "2021-10-02T17:49:47Z")

</div>

I tried setting up gRPC server using the example: [GitHub - fly-apps/grpc-service: Running gRPC services on Fly.io](http://github.com/fly-apps/grpc-service)  
The app builds and deploys without any issue and is available as usual at `<app_name>.fly.dev`

When I try the command as said in docs:

```auto
grpcurl -proto hello.proto <app_name>.fly.dev:443 MainService/Hello 

```

I get the error:

```auto
Failed to dial target host "<app_name>.fly.dev:443": remote error: tls: no application protocol

```

Not able to connect via the gRPC client. Can someone help?  
We had to rollback our gRPC server release due to the same reason as we validated with multiple gRPC servers - all facing this same issue!

---

<div class="post-metadata">

**Author:** ![skimhq](https://avatars.discourse-cdn.com/v4/letter/s/8e8cbc/32.png) [@skimhq](https://community.fly.io/u/skimhq)\
**Post date:** [October 2, 2021, 7:10pm UTC](https://community.fly.io/t/urgent-grpc-server-unreachable-via-grpcurl/2694/3 "2021-10-02T19:10:30Z")

</div>

Resolved:  
I had to set `handlers: []` in tcp as our gRPC server is insecure.

---

<div class="post-metadata">

**Author:** ![fredrik](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/fredrik/32/1895_2.png) [@fredrik](https://community.fly.io/u/fredrik)\
**Post date:** [October 3, 2021, 12:25pm UTC](https://community.fly.io/t/urgent-grpc-server-unreachable-via-grpcurl/2694/4 "2021-10-03T12:25:06Z")

</div>

> Resolved:  
> I had to set handlers: in tcp as our gRPC server is insecure.

This is more of a workaround (disabling TLS) than a solution, right?

Because having `handlers: ["tls"]` should cause the Fly proxy to terminate TLS and then forward a plaintext connection to your service, so it should work with an insecure/plaintext gRPC server - see [Deploy gRPC and gRPC-Web Services · Fly Docs](https://fly.io/docs/app-guides/grpc-and-grpc-web-services/#whats-happening-inside).

And it seems to have worked until recently - is this a behaviour change or a regression?

---

<div class="post-metadata">

**Author:** ![jsierles](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/jsierles/32/595_2.png) [@jsierles](https://community.fly.io/u/jsierles)\
**Post date:** [October 4, 2021, 3:38pm UTC](https://community.fly.io/t/urgent-grpc-server-unreachable-via-grpcurl/2694/5 "2021-10-04T15:38:44Z")

</div>

This appears to be a regression on our part. We’ll look into it!

We reverted (for other reasons) to a previous release which made the example code work again.

---

<div class="post-metadata">

**Author:** ![DazWilkin](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/dazwilkin/32/18597_2.png) [@DazWilkin](https://community.fly.io/u/DazWilkin)\
**Post date:** [November 23, 2021, 9:25pm UTC](https://community.fly.io/t/urgent-grpc-server-unreachable-via-grpcurl/2694/6 "2021-11-23T21:25:50Z")

</div>

I tried deploying a gRPC app, encountered the same issue and found this link.

My app’s a little quirky but I’m confident I’m experiencing the same issue.

I redeployed without tls (`handlers[]`) and am able to gRPCurl the endpoint but, when I add `handlers["tls"]`, it fails

`handlers[]`:

```bash
grpcurl \
-plaintext \
-import-path ${X} \
-proto ${X}/health.proto \
freddie.fly.dev:10000 \
grpc.health.v1.Health.Check
{
  "status": "SERVING"
}

```

`handlers["tls"]`:

```bash
grpcurl \
-import-path ${X} \
-proto ${X}/health.proto \
freddie.fly.dev:10000 \
grpc.health.v1.Health.Check

```

Failed to dial target host “[freddie.fly.dev:10000](http://freddie.fly.dev:10000)”: remote error: tls: no application protocol

---

<div class="post-metadata">

**Author:** ![sudhir.j](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/sudhir.j/32/39775_2.png) [@sudhir.j](https://community.fly.io/u/sudhir.j)\
**Post date:** [November 24, 2021, 6:46am UTC](https://community.fly.io/t/urgent-grpc-server-unreachable-via-grpcurl/2694/7 "2021-11-24T06:46:55Z")

</div>

@DazWilkin Could you post the the full `fly.toml` too? We’d need to check if the services are set up correctly as well.

---

<div class="post-metadata">

**Author:** ![DazWilkin](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/dazwilkin/32/18597_2.png) [@DazWilkin](https://community.fly.io/u/DazWilkin)\
**Post date:** [November 24, 2021, 4:10pm UTC](https://community.fly.io/t/urgent-grpc-server-unreachable-via-grpcurl/2694/8 "2021-11-24T16:10:42Z")

</div>

Sure!

`fly.toml`:

```auto
kill_signal = "SIGINT"
kill_timeout = 5

[build]
  image = "registry.fly.io/freddie:a3339c154a89286265be205878a777aa1aef9f95"

[experimental]
  exec = [
    "/server",
    "--failure_rate=0.5",
    "--changes_rate=15s",
    "--endpoint=:50051"
  ]
  private_network = true

[[services]]
  internal_port = 50051
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = ["tls"]
    port = "10000"

```

I tried it with both `443` and `10000` with the same result.

Using `handlers = []`, worked.

---

<div class="post-metadata">

**Author:** ![Ikechi\_Obiajunwa](https://avatars.discourse-cdn.com/v4/letter/i/8baadc/32.png) [@Ikechi\_Obiajunwa](https://community.fly.io/u/Ikechi_Obiajunwa)\
**Post date:** [November 29, 2021, 4:29am UTC](https://community.fly.io/t/urgent-grpc-server-unreachable-via-grpcurl/2694/10 "2021-11-29T04:29:24Z")

</div>

Same issue here.  
We critically need this to launch our App.

Pretty urgent. Thank you.

---

<div class="post-metadata">

**Author:** ![DazWilkin](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/dazwilkin/32/18597_2.png) [@DazWilkin](https://community.fly.io/u/DazWilkin)\
**Post date:** [December 21, 2021, 11:44pm UTC](https://community.fly.io/t/urgent-grpc-server-unreachable-via-grpcurl/2694/11 "2021-12-21T23:44:23Z")

</div>

Although this is not a priority for me (I’m using GCP), I wonder whether Fly publishes issues|features lists?

It would be useful to get an indication as to when secure gRPC will be supported as I’d like to include Fly in my solution as a deployment target.

---

<div class="post-metadata">

**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\
**Post date:** [December 22, 2021, 12:03am UTC](https://community.fly.io/t/urgent-grpc-server-unreachable-via-grpcurl/2694/12 "2021-12-22T00:03:51Z")

</div>

This might work now, actually. I think the root problem is that we don’t send the `h2` alpn when there’s no `http` handler.

Will you try updating your fly.toml services block to look like this?

```auto
  [[services.ports]]
    handlers = ["tls"]
    tls_options = {"alpn":["h2"]}
    port = "443"

```

---

<div class="post-metadata">

**Author:** ![DazWilkin](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/dazwilkin/32/18597_2.png) [@DazWilkin](https://community.fly.io/u/DazWilkin)\
**Post date:** [December 22, 2021, 12:10am UTC](https://community.fly.io/t/urgent-grpc-server-unreachable-via-grpcurl/2694/13 "2021-12-22T00:10:56Z")

</div>

Excellent! You bet!

I’ll try it tomorrow morning.

Thank you

---

<div class="post-metadata">

**Author:** ![DazWilkin](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/dazwilkin/32/18597_2.png) [@DazWilkin](https://community.fly.io/u/DazWilkin)\
**Post date:** [December 22, 2021, 5:36pm UTC](https://community.fly.io/t/urgent-grpc-server-unreachable-via-grpcurl/2694/14 "2021-12-22T17:36:07Z")

</div>

No change ☹

It disliked `{"alpn":["h2"]}` but accepted (!?) `{"alpn"=["h2"]}` and this appears to configure the app with this setting:

```bash
flyctl config display --app=${APP}

```

Yields:

```auto
"ports": [
  {
    "handlers": [
      "tls"
    ],
    "port": 10000,
    "tls_options": {
      "alpn": [
        "h2"
      ],
      "default_self_signed": false,
      "versions": []
    }
  }
],

```
