GRPC server on Machines App not reachable

I have a GRPC server setup with tls Fly handler, I setup using the guide Run a Machines App Using flyctl · Fly Docs
The relevant port parameter I provided was -p 44444:44444/tcp:tls where GRPC server is on port 44444

Everything comes up however the GRPC traffic never reaches the server in microVM. The server binds to 0.0.0.0:44444 in container.

To verify that server is up and running I ssh into machine and run grpcurl with localhost and it works.

The error on client side using the public DNS of App is following:

ChannelError(
        tonic::transport::Error(
            Transport,
            hyper::Error(
                Connect,
                Custom {
                    kind: UnexpectedEof,
                    error: "tls handshake eof",
                },
            ),
        ),
    ),

Is this a a Machine mis-configuration ? or something related to fly-proxy? Or something else ? Any help in right direction would be very helpful. Thank you.

This is working on Apps V1 with the tls_options, which is not somthing configurable for machines.
So likely something related to Machines

You should be able to add that same tls_options to the services block in the machine config. See if that helps?

Thank you Kurt.

Before posting this I noticed thattls_options is not documented on Create Machine API page

Moreoever I anyway went ahead and tried to included tls_options in the API payload - which API server accepted, however on making a GET call to fetch Machine I did not see it in response.

This is from my memory I tried last time, I havent had the opportunity to retry it yet after your suggestion. Hopefully soon.

Thank you.