I’m a networking novice, so please bear with me. I’d like to move from Kubernetes, but in my current setup, I’m using a 3rd party package called Cube for my analytics backend. There’s a couple different containers that need to be deployed that must communicate with each other.
However it appears that the containers are defined to listen for IPv4 addresses which I guess isn’t compatible with the internal private networking provided by Fly? As an example, it’s required to set the address of container #1 as an environment variable in container #2, so in my Dockerfile, I’ve done something like this:
ENV META_STORE_ADDR=meta-store-app.internal:9999.
I dont know any rust lang, but if anyone here is… I dug in to the Cube source, and I believe this is where the requests are being made. And from my understanding, this definition is for IPv4 addresses.
Is there a proposed solution for handling IPv4 across containers? I saw this question here but it seems to be resolved because the user was able to modify source for redis to listen for IPv6 properly. I unfortunately don’t have that luxury.
Hey @kurt - thanks for looking in to this! Yea I believe so. What appears to be happening is that the Cube Store Refresh Worker (top right in this diagram) doesn’t seem to be able to connect to the Cube Store Router.
Ah! That error makes me think that the cubestore processes aren’t listening on IPv6 properly. It seems like the clients are trying to connect, but the port isn’t responding.
It looks like you may be able to set an environment variable to configure that:
INFO [cubestore::cluster] <pid:509> Meta store port open on 0.0.0.0:9999
INFO [cubestore::mysql] <pid:509> MySQL port open on [::]:3306
INFO [cubestore::http] <pid:509> Http Server is listening on [::]:3030
Unfortunately there is no variable exposed for updating what the metastore port is binding to. So do you think this is something I should be requesting from Cube to expose? Or am I missing something where I don’t need to be setting that metastore binding?
There’s an ENV option on the worker node to tell it which router to connect to, which is set like this:
But if I now understand correctly this doesn’t work because:
Worker node tries to send requests to encore-staging-cubestore-router.internal:9999 (an IPv6) while the Router node is listening on 0.0.0.0:9999 (an IPv4)?
CUBESTORE_META_ADDR=[::]:9999 seems like it might work on the router. It’s hard to tell for sure, but that env var might mean two different things for worker/router.
Also, if you have CUBESTORE_META_PORT set, unset it. It seems to force IPv4 when that’s set.
Yep that’s right! 0.0.0.0 means listen on all IPv4 addresses. [::] means listen on all IPv6 addresses and, most the time, all IPv4 addresses.
@kurt So I tried adding CUBESTORE_META_ADDR to the router right before your message, HA!
Unfortunately seems like they have a check in code here that prevents adding that CUBESTORE_META_ADDR to the router node
So by removing the CUBESTORE_META_PORT on my Router, the logs no longer show a Meta store port opening, just the MySQL and Http Server:
INFO [cubestore::mysql] <pid:509> MySQL port open on [::]:3306
INFO [cubestore::http] <pid:509> Http Server is listening on [::]:3030
And then my Worker node logs show this, which is new:
Error in processing loop: Internal: No route to host (os error 113)
I just want to say a huge thank you for digging in to this here, please at any point let me know if this is too out of scope for you. I think this is on the Cube team to allow binding the meta on an IPv6 at this point, yea?
thanks so much @sudhir.j I totally agree with you about the IPv4 assumption. I think it’s best if I submit a ticket on GitHub requesting the availability of a CUBESTORE_META_BIND_ADDR and then hopefully Cube will be operational on Fly
Would help to upvote or draw attention to it. I’m not too familiar with Rust, but I can try making a PR if this is holding off your deployment for too long.
@sudhir.j oh excellent!! I’ll go add additional support for that ticket in a couple minutes. Thanks for being on top of this. The cube team is generally pretty quick, but if you are willing, I’d love to check out to see if that ENV variable is the final culprit to a successful Fly deployment
@uncvrd the change has landed to master, not sure when the next release is though. If you’re up to running Cube off the master branch we can try to get it working.
Thank you very much for the work! Given how quickly they typically push updates, I may wait a few days. I’ll report back once i’ve tried to test it though
@sudhir.j good news, those updates work like a charm!! I’ve got the whole Cube environment now deployed and communicating w/ each other. Thanks for your help, again.
Last random question. Cube allows external apps to run queries via an exposed API on the API Instance container shown in the image near the top of this thread. I can hit it externally no problem using https://[MY-FLY-URL.dev]/cubejs-api.
However, I have an internal service also running in Fly, and I’d like to query the Cube REST API within the private network (container to container). I tried building a URL like this: http://cube-api.internal/cubejs-api, similar to how I do it with docker-compose, but it returns an error stating an invalid URL. Am I formatting the URL incorrectly?
That looks about right to me. I’m guessing that the Cube router and workers are talking to each other via the .internal address as well, so that shouldn’t be an issue. Is the app you’re trying to call from part of the same organisation / have there been any overrides of the DNS resolvers?
