empz
November 1, 2024, 9:59am
2
A few related resources:
opened 11:25PM - 25 Sep 20 UTC
closed 12:33PM - 16 Sep 24 UTC
bug
ext/fetch
I run deno in the docker container in the k8s pod for as a REST API service. Eve… ry N seconds this deno service sends a POST request with metrics data over to another service in k8s, [prometheus pushgateway](https://github.com/prometheus/pushgateway)
here is the code snippet of how the POST request is being made with fetch:
```ts
// function to push metrics to prometheus "pushgateway"
export async function pushMetrics(
id: string,
metrics: string,
) {
const pgURL = getPushgatewayHostName(pushgatewayJobName, id);
try {
const cleanedUpFileContent = unescape(encodeURIComponent(metrics));
const out = new TextEncoder().encode(cleanedUpFileContent);
const response = await fetch(
pgURL,
{
method: "POST",
body: out,
},
);
if (response.status != 202 && response.status != 200) {
log.error(`failed to POST to pushgateway URL ${pgURL}, status ${response.status}`);
}
} catch (e) {
log.error(`unexpected error: failed to POST to pushgateway URL ${pgURL}, ${e}`);
}
}
function getPushgatewayHostName(pushgatewayJobName: string, id: string): string {
const pushgatewayURI = `metrics/job/${pushgatewayJobName}/instance/${id}`;
if (ENV == undefined || ENV == "") {
return `http://pushgateway:9091/${pushgatewayURI}`;
}
return `http://${ENV}-${basePushgatewayK8sServiceHost}/${pushgatewayURI}`;
}
```
permissions for the network calls are wide open:
`CMD ["deno", "run", "--cached-only", "--unstable", **"--allow-net"**, "--allow-read=./workers", "--allow-env", "--v8-flags=--disallow-code-generation-from-strings", "main.ts"]`
request comes through OK if I run it outside of k8s (not using k8s dns service host name e.g. `pgURL=http://pushgateway.mynamespace.svc.cluster.local:9091`): bring up 2 docker containers attached to the same network, so the deno service POST to `http://pushgateway:9091`, pushgateway is the name of the second container deno service is POSTing to
if I run it in the k8s I get the following error:
`unexpected error: failed to POST to pushgateway URL http://myenv-pushgateway.mynamespace.svc.cluster.local:9091/metrics/job/myjob/instance/1234567890, Http: error sending request to url (http://myenv-pushgateway.mynamespace.svc.cluster.local:9091/metrics/job/myjob/instance/1234567890): error trying to connect: tcp connect error: Connection refused (os error 111)`
looks like a Rust error message:
`
error trying to connect: tcp connect error: Connection refused (os error 111)
`
I can exec into the deno container in k8s and curl pushgateway, so it's not related to k8s network policies
not sure if these 2 issues are related:
https://github.com/denoland/deno/issues/7660
https://github.com/denoland/deno/issues/6751
using:
* latest deno v1.4.2
* k8s cluster v1.18.6
@bartlomieju @kitsonk @hayd can you please take a look
That’s bonkers. VMs just use 8.8.8.8 to resolve addresses, I’m guessing your local docker was using a different nameserver? That’s the only difference I can think of. It’s especially weird that Deno.resolveDns worked.
Incidentally, if you control your Mongo servers might consider using WireGuard peers to connect to them from a Fly app. It’s a much better way to connect to DBs since peers are private and encrypted.
It looks like it might be related to the app not listening to the IP v6 address? The public server app works because that gets resolved to an IP v4 address, I think. But the .internal ones are resolved to ip v6, right?