getting different errors with Machines

1

I had a Rails app running on V1 and working just fine. I decided to try and move it to V2 to see how hard it is to work with machines (spoiler: not hard at all) but I’m getting an erratic behavior from the app from different directions

  1. deployments usually don’t work the first time:
    here is my Makefile for machine deployment
uid := $(shell openssl rand -base64 12)
machine:
	flyctl deploy --image-label $(uid) --build-only --push --auto-confirm -c infrastructure/web.toml
	flyctl image update --image registry.fly.io/sumiu-machine:$(uid) -y

rarely this works in the first try, I get the following error usually twice before it works:

Sending build context to Docker daemon  237.3kB
Error failed to fetch an image or build from source: error building: Error response from daemon: invalid reference format
  1. Cert is not being issued and complains about IPs that don’t match:
    I deleted the former V1 app and issue a new cert for the same domain for V2 but the cert is not issued. This is what I see when I list the cert:
The certificate for sumiu.link has not been issued yet.

Hostname                  = sumiu.link

DNS Provider              = cloudflare

Certificate Authority     = Let's Encrypt

Issued                    =

Added to App              = 18 hours ago

Source                    = fly

A Record (104.21.6.27) does not match app's IP (66.241.125.146)
AAAA Record (2606:4700:3033::6815:61b) does not match app's IP (2a09:8280:1::a:bea1)
Address resolution (104.21.6.27) does not match app's IP (66.241.125.146/2a09:8280:1::a:bea1)
Address resolution (172.67.154.153) does not match app's IP (66.241.125.146/2a09:8280:1::a:bea1)
Address resolution (2606:4700:3033::6815:61b) does not match app's IP (66.241.125.146/2a09:8280:1::a:bea1)
Address resolution (2606:4700:3030::ac43:9a99) does not match app's IP (66.241.125.146/2a09:8280:1::a:bea1)
You are creating a certificate for sumiu.link
We are using lets_encrypt for this certificate.

I don’t really remember but I believe the IPs listed here are from the V1.

When I disable CF proxy, the certificate is issued but the RSA is marked with the red

  1. Random SSL handshake failed Error: code 525:

more often than I would’ve expected, I get this error when I load the app. AFAIK there is not much I can do.

here is my CF configuration:

image
image2568×778 109 KB

  1. Disabling CF proxy causes all kind of http vs https errors

When I disable the CF proxy, the css is not loaded and returns this error in the console:

Mixed Content: The page at 'https://sumiu.link/' was loaded over HTTPS, but requested an insecure stylesheet 'http://d54mowbuhi4an.cloudfront.net/assets/application-a3ef288bedebdf6e94b2c3bbdab80379fcfbcb5e297f30c6f993aaafa421af3e.css'. This request has been blocked; the content must be served over HTTPS.

trying to interact with the app is even worse. I get this error:

2023-01-02T14:20:08.684 app[6e82577b4d6287] fra [info] [b3636863-20c2-4197-acad-3fdad91635e0] ActionController::InvalidAuthenticityToken (HTTP Origin header (https://sumiu.link) didn't match request.base_url (http://sumiu.link)):

I’ve already tried a number of things, like updated my production.rb like this:

config.force_ssl = true

config.ssl_options = { redirect: { exclude: lambda { |request|
  request.headers["X-Forwarded-Ssl"] != "on"
} } }

to no success

2

Update:

Fixed #1 by changing uid := $(shell openssl rand -base64 12) to uid := $(shell openssl rand -hex 12) as base64 will include + sometimes in the string, causing the build to fail.

kinda fixed #4 by configuring config/initializers/content_security_policy.rb but it only got me so far: CSS and JS loaded just fine but I still get InvalidAuthenticityToken errors when posting a request.

1 Like
3

I’m kinda using this as a blog but I will post my findings in case anyone else find the same problems:

Update 2: I couldn’t fix the InvalidAuthenticityToken without proxying CF if my life depended on it. I keep getting this HTTP Origin header (https://sumiu.link) didn't match request.base_url (http://sumiu.link bs no matter what. I found that I only need to add records (@ and www) for IPv6. Deleting the IPv4 records didn’t make any difference. Ended up reverting the config/initializers/content_security_policy.rb changes since with CF there are no problems

I still get SSL handshake failed when I open the app about 40% of the time which is really annoying but running dig txt _acme-challenge.sumiu.link +short does return a Fly address plus some random string which I think is right.

Tried different SSL/TSL configurations on CF (Flexible/Full/Strict) but only Full works.

Would appreciate any leads on these things as network infrastructure is a topic where I still have muuuuch to learn

4

Guess work:

See if adding force_https=true to your Fly (http) service handler help?

Btw, is your app using RoR?

5

as far as I understand, fly.toml doesn’t work for Machines but I did that by updating the port and leaving the handler part empty and using the origin cert provided by CF which didn’t help, still got a lot of 525 errors.

I gave up on V2 and deployed back to V1