Internal address connection error

Questions / Help
1

I trying to access a Kibana App inside my organization from other App but I received this error:

{“log.level”:“error”,“@timestamp”:“2023-12-15T21:23:03.810Z”,“log.logger”:“kibana”,“log.origin”:{“file.name”:“kibana/connecting_client.go”,“file.line”:79},“message”:“failed to obtain connection to Kibana: fail to get the Kibana version: HTTP GET request to http://obs-kibana.internal:5601/api/status fails: fail to execute the HTTP GET request: Get "http://obs-kibana.internal:5601/api/status": dial tcp [fdaa:2:7c07:a7b:1f61:1bc3:cdc1:2]:5601: connect: connection refused. Response: .”,“service.name”:“apm-server”,“ecs.version”:“1.6.0”}

fly.toml >>

app = “obs-kibana”
primary_region = “gru”

[build]

[http_service]
internal_port = 5601
force_https = true
auto_stop_machines = true
auto_start_machines = true
min_machines_running = 1
processes = [“app”]

[[vm]]
cpu_kind = “shared”
cpus = 2
memory_mb = 2048

[mounts]
source=“kibana_data”
destination=“/”
initial_size = “5gb”

kibana config file >>

server.host: “0.0.0.0”
server.shutdownTimeout: “5s”
server.publicBaseUrl: “https://obs-kibana.fly.dev”
elasticsearch.hosts: [ “http://obs-elasticsearch.internal:9200” ]
monitoring.ui.container.elasticsearch.enabled: true

All my other apps works fine, only Kibana has a problem with vpn.
I believe that its any config error, but I dont know what.

2

Hi… The external (.fly.dev) address does accept connections—although it gives a 503 response. Have you tried fetching from within an SSH session?

$ fly ssh console -a obs-kibana
# apt-get update
# apt-get install --no-install-recommends curl
# curl -i 'http://localhost:5601/api/status'; echo

This will help distinguish between network problems and server glitches…

3

I believe you need to set the host to “::”.
If you ssh into your app and run “netstat -tuln”, the process should be :::5601.

You may want to remove force_https since you’re not exposing the app to public
EDIT: nvm, I misread the [[http_service]], thought it was just [[services]]

1 Like
4

Hi,

External accept connections, but I need use internal because APM Server config file add default port to host if you set host config without port.

If I use .fly.dev:

APM Server config file >>

…
kibana:
# For APM Agent configuration in Kibana, enabled must be true.
enabled: true

# Scheme and port can be left out and will be set to the default (`http` and `5601`).
# In case you specify an additional path, the scheme is required: `localhost:5601/path`.
# IPv6 addresses should always be defined as: `[2001:db8::1]:5601`.
host: "https://obs-kibana.fly.dev"

…

Error:

{“log.level”:“error”,“@timestamp”:“2023-12-16T00:50:07.114Z”,“log.logger”:“kibana”,“log.origin”:{“file.name”:“kibana/connecting_client.go”,“file.line”:79},“message”:“failed to obtain connection to Kibana: fail to get the Kibana version: HTTP GET request to https://obs-kibana.fly.dev:5601/api/status fails: fail to execute the HTTP GET request: Get "https://obs-kibana.fly.dev:5601/api/status\”: read tcp 172.19.135.170:38678->66.241.124.7:5601: read: connection reset by peer. Response: .",“service.name”:“apm-server”,“ecs.version”:“1.6.0”}

5

Only Kibana app is not accessible

curl -vi http://obs-kibana.internal:5601/api/status

output:

curl: (7) Failed to connect to obs-kibana.internal port 5601 after 1 ms: Couldn’t connect to server

Trying to access Elasticsearch works fine:

curl -v http://obs-elasticsearch.internal:9200
output:

GET / HTTP/1.1
Host: obs-elasticsearch.internal:9200
User-Agent: curl/7.88.1
Accept: /

< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 549
<
{
“name” : “e2865666b67d08”,
“cluster_name” : “docker-cluster”,
“cluster_uuid” : “XPjmgcqnRN69cBejPk4VgA”,
“version” : {
“number” : “7.17.16”,
“build_flavor” : “default”,
“build_type” : “docker”,
“build_hash” : “2b23fa076334f8d4651aeebe458a955a2ae23218”,
“build_date” : “2023-12-08T10:06:54.672540567Z”,
“build_snapshot” : false,
“lucene_version” : “8.11.1”,
“minimum_wire_compatibility_version” : “6.8.0”,
“minimum_index_compatibility_version” : “6.0.0-beta1”
},
“tagline” : “You Know, for Search”
}

  • Connection #0 to host obs-elasticsearch.internal left intact
6

is your kibana on host 0.0.0.0:5601? Try binding to :::5601.
I also had problems communicating intranetwork, eg <myapp>.internal:<port> when the app’s host wasn’t bounded on [::]

2 Likes
7

server.host: "::" in kibana resolved. Thanks!

8

Just so other people are aware:
0.0.0.0 binds on ipv4 and :: binds on ipv4 +ipv6.
Fly communicates on v6 (correct me if I’m wrong devs)

9

.internal hostnames resolve to IPv6 addresses associated with Fly machines, therefore you need to bind to :: for your internal (non-proxied) services to be available via the private network.

You can bind to 0.0.0.0 if all you need is to expose services via our public proxy.

1 Like
10

Perhaps that detail can be added to this section: Private Networking · Fly Docs

11

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.