gRPC `502`s and `missing content-type field`

1

My apologies for being your dumbest user; I guess someone had to be :slight_smile:

I’ve a gRPC-based service that errors when I attempt to call its methods.

I wonder the service requires TLS?

TOML:

app = "${APP}"

kill_signal = "SIGINT"
kill_timeout = 5

[build]
  image = "${IMAGE}"

[experimental]
  exec = [
    "/server",
    "--failure_rate=0.5",
    "--changes_rate=15s",
    "--grpc_endpoint=:50051",
    "--prom_endpoint=:8080"
  ]

[[services]]
  internal_port = 50051
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = ["http"]
    port = "10001"

[[services]]
  internal_port = 8080
  protocol = "tcp"

  [[services.ports]]
    handlers = ["http"]
    port = "80"

But:

docker run \
--interactive --tty --rm \
--publish=8082:8082 \
${IMAGE} \
--period=5s \
--grpc_endpoint=${ENDPOINT}:${GRPC} \
--prom_endpoint=:8082
rpc error: code = Unavailable desc = Bad Gateway: HTTP status code 502;
transport: missing content-type field

And gRPCurl equally is unable to invoke the e.g. Check method:

docker run \
--interactive --tty --rm \
--volume=${PWD}:/protos \
fullstorydev/grpcurl \
-plaintext \
-proto protos/health.proto \
${ENDPOINT}:${GRPC} \
grpc.health.v1.Health/Check
ERROR:
  Code: Unavailable
  Message: Bad Gateway: HTTP status code 502;
           transport: missing content-type field

The server’s metrics endpoint works:

curl --request GET ${ENDPOINT}:80/metrics
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0

I can run the server locally successfully:

docker run \
--interactive --tty --rm \
--publish=${GRPC}:50051 \
--publish=8081:${PROM} \
${IMAGE} \
--failure_rate=0.5 \
--changes_rate=15s \
--grpc_endpoint=:50051 \
--prom_endpoint=:8080

And invoke its Check method:

docker run \
--interactive --tty --rm \
--net=host \
--volume=${PWD}:/protos \
fullstorydev/grpcurl \
-plaintext \
-proto protos/health.proto \
:${GRPC} \
grpc.health.v1.Health/Check
{
  "status": "SERVING"
}
2

gRPC can be a little moody, depending on the library. We have an example node app config that might work. Try changing your service entry to this:

[[services]]
  internal_port = 50051
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = ["tls"]
    port = "10001"

This disables our HTTP handling, but enables TLS.

Incidentally, we have custom metrics coming so we can show you how to scrape that Prometheus endpoint into our store sometime this week. :smiley:

3

Thank you for the reply.

The service is insecure (non-TLS) because I use a “sidecar” (on other platforms) to terminate TLS (and keep my code simple).

I wonder whether my issue is trying to use a non-TLS gRPC service on Fly!?

4

Ah, change it to this:

handlers = []

That should get you raw TCP.

1 Like
5

Aha! Will try that, thanks!