Not able to attach to Elixir node with Livebook on Fly

I’ve been able to locally connect to an Elixir node through Wireguard. This works perfectly.

However, I’d prefer to use a hosted version of Livebook. So I tried Livebook · Launch on Fly.io and launched an instance. Trying to connect the same node (the Elixir node on Fly) is not possible and I’m not getting any errors in the livebook log as well.

Did anyone else manage to achieve this? Maybe the fly livebook is deployed differently (without ERL_AFLAGS="-proto_dist inet6_tcp") env variable.

Hi there,

Can you please check if this is useful - Connecting Livebook to Your App in Production
The procedure described here does seem to use ERL_AFLAGS.

You could also go through this for the WireGuard part - Private Networking

Hope this helps!

Thanks for your reply @FrequentFlyer. That procedure is for local usage, but Fly offers a version that is easily deployable and mentioned on https://livebook.dev itself. However, it is not able to connect to nodes inside Fly.

I’ve used the guide to setup an Elixir project (set node name and cookie) and try to connect to one another, which works - but is not able to connect within Livebook.

Here’s a video: https://noegip.s3.eu-central-1.amazonaws.com/dump/flyerbook.mp4

Maybe this is more like a feature request for fly.io: Launch Livebook cc @brainlid

Ahh ok, I understand now.
Local Livebook to Fly app works, but you want to connect the easy-deploy, hosted Livebook to your Fly app(s).

Thanks for clarifying… will wait to hear from Mark then.

1 Like

Hello @FrequentFlyer!

I like where you’re going with this! At the time I wrote the “Connecting Livebook to Your App in Production” guide, we hadn’t created the Livebook Launcher… so a hosted Livebook wasn’t an option. :slight_smile:

That’s all just to say, “I don’t know”.

This may still be possible. I haven’t tried it yet. I can see some obvious reasons to want to do it.

When trying to get Livebook to connect to another remote project, it’s helpful to watch the logs on both sides. Livebook injects modules into the project it’s trying to connect to. So there may be helpful log messages in your project. Then also the running Livebook instance’s logs may show something useful.

Then there’s the question of if the two deployed instances are accessible to each other via WireGuard or not. May have to look into that too.

I like the idea but I won’t have time to play with it for several days.

1 Like

The reason for me to do this is to use Livebook to create docs + BI + storybookJS-like tool, based on our application - all on Fly and within the Fly network. But I’m still pretty new to it and I didn’t know it injects modules into the project it connects to.

Afaik it’s not a Wireguard issue, but rather a Livebook issue on Fly. (As I show in the video) The instances can access each other, so that’s not the problem.

Is there something I can help with? If that’s possible/you want to, maybe you can publish the project to Github for us to take a look?

Here’s what I get if I ssh into my Livebook host created with the launcher:

root@44bd5d91:/# epmd -names
epmd: up and running on port 4369 with data:
name frt5mijd-livebook at port 46405
name livebook at port 42211
root@44bd5d91:/# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:42211           0.0.0.0:*               LISTEN     
518/beam.smp        
tcp        0      0 0.0.0.0:46405           0.0.0.0:*               LISTEN      575/beam.smp        
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      518/beam.smp        
tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN      548/epmd            
tcp6       0      0 fdaa:0:344e:a7b:2052:22 :::*                    LISTEN      519/hallpass        
tcp6       0      0 :::4369                 :::*                    LISTEN      548/epmd   

Now, if I launch an iex session on this same Livebook “machine” with inet6_tcp flag and a cookie of my other Elixir app:

root@44bd5d91:/# iex --name test@fdaa:0:344e:a7b:2052:0:6db5:2 --cookie "my-other-app-cookie" --erl "-proto_dist inet6_tcp"
Erlang/OTP 24 [erts-12.1.5] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1] [jit]

Interactive Elixir (1.13.0) - press Ctrl+C to exit (type h() ENTER for help)

I can both ping it from my-other-app node:

iex(my-other-app@fdaa:0:344e:a7b:23c4:bc82:9889:2)1> Node.ping :"test@fdaa:0:344e:a7b:2052:0:6db5:2"
:pong

…and connect to my-other-app node from the iex session on the Livebook host:

iex(test@fdaa:0:344e:a7b:2052:0:6db5:2)1> Node.connect :"my-other-app@fdaa:0:344e:a7b:23c4:bc82:9889:2"
true

So it seems the issue is, as @noegip suggested, that livebooks are launched without the -proto_dist inet6_tcp flag.

This worked for me (got the idea from this post):

  • run fly config save -a <name of the Livebook app> in an empty directory
  • in fly.toml, add the following line to the [env] section:
  ERL_AFLAGS = "-proto_dist inet6_tcp"

(there was already a line elixir_erl_optio_ns = "-proto_dist inet6_tcp" in this section, but (a) there’s a typo? and (b) even w/o the typo, it seemed to have no effect?)

  • run fly deploy -i livebook/livebook

Once the app is redeployed, Livebook should be able to attach to other nodes.

2 Likes

Nice @kkonstant! Thank you for helping out here.

That must be the problem. Thank you for providing a fix, it works now.

Wow! Nice sleuthing! I’ll check on the settings issue for the deployed Livebooks and see if it can be addressed there.

@noegip This issue has been fixed!

The problem was the ENV values from the launcher were being unintentionally modified. This broke the Livebook deployment and the expected ENV settings.

If you deploy a new Livebook using the launcher, it will work as expected! No additional ENV workarounds needed.

I verified this by attaching my Fly hosted Livebook to my Fly hosted Phoenix app. Just needed the node name and the cookie value. Very cool! Will have to blog about that approach and usage soon!

Thanks for the report!

1 Like