Unable to reach Cassandra database via Flycast from another app in same org

,
1

Hi, all, thanks in advance for any help.

I am broadening my horizons, database-wise and using Cassandra for an app I’m working on. The database uses a pretty simple Dockerfile

FROM cassandra:5
COPY ./cassandra.yaml /etc/cassandra/cassandra.yaml

EXPOSE 9042

CMD ["cassandra", "-f"]

With the fly.toml matching the expected port

[http_service]
  internal_port = 9042
  force_https = true
  #auto_stop_machines = 'stop'
  auto_stop_machines = false
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

Sure enough when I fly ssh console into my app I can do an nslookup and see

root@2867555c443268:/# nslookup <cass-app-name>.flycast
Server:		fdaa::3
Address:	fdaa::3#53

Name:	<cass-app-name>.flycast
Address: fdaa:3d:df18:0:1::2

Going from there to the Cassandra console confirms the IP address

root@2867555c443268:/# cqlsh -u *** -p ***

Warning: Using a password on the command line interface can be insecure.
Recommendation: use the credentials file to securely provide the password.

Connected to <cluster> at 127.0.0.1:9042
[cqlsh 6.2.0 | Cassandra 5.0.6 | CQL spec 3.4.7 | Native protocol v5]
Use HELP for help.

This should confirm that the URL I give Flyway would be jdbc:cassandra://<cass-app-name.flycast:9042?localdatacenter=datacenter1, based on the URL when running in local Docker being jdbc:cassandra://<cass-container-name>:9042?localdatacenter=datacenter1, however what I’m seeing

Caused by: org.flywaydb.core.internal.exception.sqlExceptions.FlywaySqlUnableToConnectToDbException: Unable to obtain connection from database (jdbc:cassandra://<cass-app-name.flycast:9042?localdatacenter=datacenter1) for user 'cassandra': com.datastax.oss.driver.api.core.AllNodesFailedException: Could not reach any contact point, make sure you've provided valid addresses (showing first 1 nodes, use getAllErrors() for more): Node(endPoint=<cass-app-name>.flycast/<unresolved>:9042, hostId=null, hashCode=39591661): [com.datastax.oss.driver.api.core.connection.ConnectionInitException: [s0|control|id: 0x57753412, L:/[fdaa:3d:df18:a7b:616:252e:bd09:2]:52296 - R:<cass-app-name>.flycast/[fdaa:3d:df18:0:1:0:0:2]:9042] Protocol initialization request, step 1 (OPTIONS): unexpected failure (com.datastax.oss.driver.api.core.connection.ClosedConnectionException: Unexpected error on channel)]

As we can see the host IP address is being resolved correctly, so I feel confident ruling out that I typo’d a URL somewhere. I know Cassandra is running. As it looks like both ends are working as intended the problem must be that I have something configured incorrectly in the fly.io internal networking, possibly with the port not being exposed correctly? Would appreciate any input on what might have gone wrong.

2

I’m pretty sure Cassandra does not speak http. Do you need a raw tcp service instead?

Also, did you try cqlsh in the machine where you’re trying to connect to Cassandra? the error may be less obtuse than the one produced by that Java thing.

1 Like
3

Thanks, these were good suggestions. I was sure the raw TCP suggestion would work so I made the change, with this now being the full TOML

app = "<cass-app-name>"
primary_region = "ord"

[[services]]
auto_start_machines = true
auto_stop_machines = false
internal_port = 9_042
min_machines_running = 0
protocol = "tcp"

[[vm]]
cpus = 1
memory = "1gb"
memory_mb = 1_024

Per your suggestion I got cqlsh running on a machine for the other app We can see the result here

root@5683412ebd5e28:/# cqlsh-env/bin/cqlsh <cass-app-name>.flycast -u *** -p ***

Warning: Using a password on the command line interface can be insecure.
Recommendation: use the credentials file to securely provide the password.

Connection error: ('Unable to connect to any servers', {'fdaa:3d:df18:0:1::2:9042': ConnectionResetError(104, 'Connection reset by peer')})

While you’re definitely right about the error being less obtuse, I’m not sure I’d call it more helpful. In case it’s unclear about the IPV6 being different, it looks like it changed when I redeployed the cassandra app to use TCP. Pinging the cassandra flycast URL from the service machine works fine, though.

5

This is missing the mandatory [[services.ports]] subsection now. That was implicit with the simplified [http_service], but not with this more general form. I’d recommend a fly config validate --strict whenever you encounter puzzling behavior on the Fly.io platform. It reveals many errors that flyctl doesn’t actually mention otherwise.

It’s the same IPv6 address, fdaa:3d:df18:0:1::2; it’s just broken syntax in the error message. (It tacked the :9042 port specifier on the end, without using square brackets to disambiguate.)

1 Like
6

Okay, yeah, me forgetting the ports section was the problem and the root problem, as roadmr pointed out, appears to have been the need for TCP. and this

fly config validate --strict

is going to save me so much time going forward :smiling_face_with_three_hearts:

Big thanks to the both of you.

1 Like
7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.