How to connect to a 3rd party image that listens for IPv4 addresses using Fly's private networking?

I’m a networking novice, so please bear with me. I’d like to move from Kubernetes, but in my current setup, I’m using a 3rd party package called Cube for my analytics backend. There’s a couple different containers that need to be deployed that must communicate with each other.

However it appears that the containers are defined to listen for IPv4 addresses which I guess isn’t compatible with the internal private networking provided by Fly? As an example, it’s required to set the address of container #1 as an environment variable in container #2, so in my Dockerfile, I’ve done something like this:

ENV META_STORE_ADDR=meta-store-app.internal:9999.

I dont know any rust lang, but if anyone here is… I dug in to the Cube source, and I believe this is where the requests are being made. And from my understanding, this definition is for IPv4 addresses.

Is there a proposed solution for handling IPv4 across containers? I saw this question here but it seems to be resolved because the user was able to modify source for redis to listen for IPv6 properly. I unfortunately don’t have that luxury.

Any thoughts or suggestions? Thanks!

I think that Rust will work fine with IPv6, TcpStream::connect(&meta_remote_addr) should handle .internal lookups with no issues.

Where in the cube setup is the problem happening? The router isn’t able to connect to the worker?

Hey @kurt - thanks for looking in to this! Yea I believe so. What appears to be happening is that the Cube Store Refresh Worker (top right in this diagram) doesn’t seem to be able to connect to the Cube Store Router.

When I deploy a cubestore refresh worker, the first few logs I see are:

2021-10-23T05:07:45.625 app[5c51eb05] lax [info] 2021-10-23 05:07:45,624 ERROR [cubestore::cluster] <pid:510> Failed to get warmup partitions: Internal: Connection refused (os error 111)
2021-10-23T05:07:49.500 app[5c51eb05] lax [info] 2021-10-23 05:07:49,499 ERROR [cubestore::cluster] <pid:510> Error in processing loop: Internal: Connection refused (os error 111)

When I check the first error, it comes from calling this method

I (think) the second error afterwards comes from calling this method here

I’m not sure if this is enough useful information to help resolve this, so please let me know what else I can provide.

Ah! That error makes me think that the cubestore processes aren’t listening on IPv6 properly. It seems like the clients are trying to connect, but the port isn’t responding.

It looks like you may be able to set an environment variable to configure that:


@kurt I think you are on the right track here!! So close…there’s 2 bind address ENV variables that I can set so I did the following on my router node:


And now in my logs I see:

INFO  [cubestore::cluster] <pid:509> Meta store port open on
INFO  [cubestore::mysql] <pid:509> MySQL port open on [::]:3306
INFO  [cubestore::http] <pid:509> Http Server is listening on [::]:3030

Unfortunately there is no variable exposed for updating what the metastore port is binding to. So do you think this is something I should be requesting from Cube to expose? Or am I missing something where I don’t need to be setting that metastore binding?

There’s an ENV option on the worker node to tell it which router to connect to, which is set like this:

ENV CUBESTORE_META_ADDR=encore-staging-cubestore-router.internal:9999

But if I now understand correctly this doesn’t work because:

Worker node tries to send requests to encore-staging-cubestore-router.internal:9999 (an IPv6) while the Router node is listening on (an IPv4)?

Do I understand correctly?

CUBESTORE_META_ADDR=[::]:9999 seems like it might work on the router. It’s hard to tell for sure, but that env var might mean two different things for worker/router.

Also, if you have CUBESTORE_META_PORT set, unset it. It seems to force IPv4 when that’s set.

Yep that’s right! means listen on all IPv4 addresses. [::] means listen on all IPv6 addresses and, most the time, all IPv4 addresses.

@kurt So I tried adding CUBESTORE_META_ADDR to the router right before your message, HA!

Unfortunately seems like they have a check in code here that prevents adding that CUBESTORE_META_ADDR to the router node :sweat_smile:

So by removing the CUBESTORE_META_PORT on my Router, the logs no longer show a Meta store port opening, just the MySQL and Http Server:

INFO  [cubestore::mysql] <pid:509> MySQL port open on [::]:3306
INFO  [cubestore::http] <pid:509> Http Server is listening on [::]:3030

And then my Worker node logs show this, which is new:

Error in processing loop: Internal: No route to host (os error 113)

I just want to say a huge thank you for digging in to this here, please at any point let me know if this is too out of scope for you. I think this is on the Cube team to allow binding the meta on an IPv6 at this point, yea?

I would have expected a variable here that allowed the metastore listener at 9999 to bind to an IPv6 address, but it seems to be missing cube.js/ at b7128d43d2aaffdd7273555779176b3efe4e2aa6 · cube-js/cube.js · GitHub — something along the lines of CUBESTORE_META_BIND_ADDR which currently doesn’t exist.

If you look at the internal metastore_bind_address it seems to always assume IPv4: cube.js/ at b7128d43d2aaffdd7273555779176b3efe4e2aa6 · cube-js/cube.js · GitHub

I think the way to make this work would be to change that code to use an ENV var like the http_bind_address does: cube.js/ at b7128d43d2aaffdd7273555779176b3efe4e2aa6 · cube-js/cube.js · GitHub

This same problem will likely happen when trying to connect to workers as well, the worker_bind_address also seems to carry an IPv4 assumption: cube.js/ at b7128d43d2aaffdd7273555779176b3efe4e2aa6 · cube-js/cube.js · GitHub

thanks so much @sudhir.j I totally agree with you about the IPv4 assumption. I think it’s best if I submit a ticket on GitHub requesting the availability of a CUBESTORE_META_BIND_ADDR and then hopefully Cube will be operational on Fly :pray:

I’ve raised one here already with the references, IPv6 / Bind Address Configuration Support for Metastore and Workers · Issue #3575 · cube-js/cube.js · GitHub

Would help to upvote or draw attention to it. I’m not too familiar with Rust, but I can try making a PR if this is holding off your deployment for too long.

@sudhir.j oh excellent!! I’ll go add additional support for that ticket in a couple minutes. Thanks for being on top of this. The cube team is generally pretty quick, but if you are willing, I’d love to check out to see if that ENV variable is the final culprit to a successful Fly deployment

@uncvrd I’ve set up a PR with the changes necessary to specify a custom bind address for the META and WORKER as well. If you’d like to try it out you can build an image off GitHub - sudhirj/cube.js: 📊 Cube.js — Open-Source Analytics API for Building Data Apps

Hopefully that should handle the networking part of the problem, then we can see if there’s anything else that comes up.

Huge thanks @sudhir.j I’ll give the PR a shot once I finish converting my environment from Kubernetes. I also made the Cube team aware of your PR :pray:

Thanks for the attention boost. They’ve already reviewed it and pointed out a mistake, I’ll fix it and get it through.

@uncvrd the change has landed to master, not sure when the next release is though. If you’re up to running Cube off the master branch we can try to get it working.


Thank you very much for the work! Given how quickly they typically push updates, I may wait a few days. I’ll report back once i’ve tried to test it though

1 Like

@uncvrd Released in 0.28.50 :smiley:

Excellent!! I’m thinking I can get to testing this on the weekend and will let you know. Apologies for the delay.

@sudhir.j good news, those updates work like a charm!! I’ve got the whole Cube environment now deployed and communicating w/ each other. Thanks for your help, again.

Last random question. Cube allows external apps to run queries via an exposed API on the API Instance container shown in the image near the top of this thread. I can hit it externally no problem using https://[]/cubejs-api.

However, I have an internal service also running in Fly, and I’d like to query the Cube REST API within the private network (container to container). I tried building a URL like this: http://cube-api.internal/cubejs-api, similar to how I do it with docker-compose, but it returns an error stating an invalid URL. Am I formatting the URL incorrectly?

1 Like

That looks about right to me. I’m guessing that the Cube router and workers are talking to each other via the .internal address as well, so that shouldn’t be an issue. Is the app you’re trying to call from part of the same organisation / have there been any overrides of the DNS resolvers?