Using s3 and Wasabi for Build

Looks like a CA issue.
Setting up the domain for production.
Configuring the DNS and we should be good to go.

2025-07-17T15:34:15.525 app[781560db495558] iad [info] INFO Starting init (commit: d0572327e)...

2025-07-17T15:34:15.749 app[781560db495558] iad [info] INFO Checking filesystem on /mnt/name

2025-07-17T15:34:15.752 app[781560db495558] iad [info] /dev/vdc: clean, 11/64512 files, 8785/258048 blocks

2025-07-17T15:34:15.753 app[781560db495558] iad [info] INFO Mounting /dev/vdc at /mnt/name w/ uid: 65534, gid: 65534 and chmod 0755

2025-07-17T15:34:15.756 app[781560db495558] iad [info] INFO Resized /mnt/name to 1056964608 bytes

2025-07-17T15:34:15.774 app[781560db495558] iad [info] INFO Preparing to run: `/bin/bash /app/bin/litestream.sh /bin/bash /app/bin/litestream.sh /app/bin/webhook_portal start` as nobody

2025-07-17T15:34:15.780 app[781560db495558] iad [info] INFO [fly api proxy] listening at /.fly/api

2025-07-17T15:34:15.831 runner[781560db495558] iad [info] Machine started in 1.491s

2025-07-17T15:34:15.833 proxy[781560db495558] iad [info] machine started in 1.497085185s

2025-07-17T15:34:16.086 app[781560db495558] iad [info] 2025/07/17 15:34:16 INFO SSH listening listen_address=[fdaa:20:de8d:a7b:2fe:747b:e44b:2]:22

2025-07-17T15:34:16.196 app[781560db495558] iad [info] S3 storage configured - using Litestream replication

2025-07-17T15:34:17.168 app[781560db495558] iad [info] time=2025-07-17T15:34:17.167Z level=ERROR msg="failed to run" error="cannot fetch generations: RequestError: send request failed\ncaused by: Get \"https://fly.storage.tigris.dev/weathered-smoke-2716?delimiter=%2F&prefix=litestream%2Fmnt%2Fname%2Fname.db%2Fgenerations%2F\": tls: failed to verify certificate: x509: certificate signed by unknown authority"

2025-07-17T15:34:17.781 app[781560db495558] iad [info] INFO Main child exited normally with code: 1

2025-07-17T15:34:17.799 app[781560db495558] iad [info] INFO Starting clean up.

2025-07-17T15:34:17.856 app[781560db495558] iad [info] INFO Umounting /dev/vdc from /mnt/name

2025-07-17T15:34:17.857 app[781560db495558] iad [info] WARN could not unmount /rootfs: EINVAL: Invalid argument

2025-07-17T15:34:17.858 app[781560db495558] iad [info] [ 3.142205] reboot: Restarting system

2025-07-17T15:34:18.773 runner[781560db495558] iad [info] machine has reached its max restart count of 10

2025-07-17T15:34:21.607 proxy[781560db495558] iad [info] waiting for machine to be reachable on 0.0.0.0:8080 (waited 5.773426586s so far)

2025-07-17T15:34:27.610 proxy[781560db495558] iad [info] waiting for machine to be reachable on 0.0.0.0:8080 (waited 11.77702179

Get "https://fly.storage.tigris.dev/weathered-smoke-2716?delimiter=%2F&prefix=litestream%2Fmnt%2Fname%2Fname.db%2Fgenerations%2F\": tls: failed to verify certificate: x509: certificate signed by unknown authority

looks like a Tigris issue.

I am not seeing certificate error when going to this URL. Is there a specific environment that reproduces it.

BTW @lillian the certificate for fly.storage.tigris.dev is managed by Fly. Could you help look into why a certificate error would be thrown here.

I think the app config is messed. It’s trying to run SSL on the app instead as a proxy at the deployed layer. I think this might be messing up the connection will keep you posted.

No luck. Move to tom that had better toml file that I know works. Not sure why it can not connect.
The initial build was done during the outage, could that cause an issue?
Is there a way to delete?
Do I need a tigris account for production?
Do I need to set any para or configs in production?
Everything just says s3 bucket vars which seem to work on other apps.

2025-07-17T17:38:45.805 app[781560db495558] iad [info] 2025/07/17 17:38:45 INFO SSH listening listen_address=[fdaa:20:de8d:a7b:2fe:747b:e44b:2]:22

2025-07-17T17:38:46.111 app[781560db495558] iad [info] time=2025-07-17T17:38:46.111Z level=ERROR msg=“failed to run” error=“cannot fetch generations: RequestError: send request failed\ncaused by: Get "https://fly.storage.tigris.dev/weathered-smoke-2716?delimiter=%2F&prefix=litestream%2Fmnt%2Fname%2Fname.db%2Fgenerations%2F\”: tls: failed to verify certificate: x509: certificate signed by unknown authority"

You don’t need any additional setup for Tigris, no separate account setup is needed. You just need to have the environment vars setup, which it seems like you have? Let me reach out to my friends at Fly separately.

we looked into it, it’s possible your image might not have ca-certificates added - can you try adding apt install -y ca-certificates to your Dockerfile?

@lillian @ovaistariq
I got the certs in or at least it seems to be working.

Questions:
Is it possible to use Wasabi as a replacement for Tigris?
Do you need to have an API and account with Tigris or is it included with Fly deployments?
Are the current platform errors going to effect my deploys and am I just wasting time until those items are fixed?

I have setup Wasabi with the proper keys for prod and dev with proper policies, this is working but I had to rollback as I could not get the lightspeed script to run no matter what I do or how I change the file perms it is always blocked.

New logs.

2025-07-17T20:17:07Z app[2867614a9e6758] iad [info] INFO [fly api proxy] listening at /.fly/api
2025-07-17T20:17:07Z app[2867614a9e6758] iad [info]Starting litestream.sh...
2025-07-17T20:17:07Z app[2867614a9e6758] iad [info]S3 storage fully configured - using Litestream replication
2025-07-17T20:17:07Z runner[2867614a9e6758] iad [info]Machine started in 1.203s
2025-07-17T20:17:07Z app[2867614a9e6758] iad [info]2025/07/17 20:17:07 INFO SSH listening listen_address=[fdaa:20:de8d:a7b:2fe:2f69:2081:2]:22
2025-07-17T20:17:08Z app[2867614a9e6758] iad [info]time=2025-07-17T20:17:08.137Z level=ERROR msg="failed to run" error="cannot fetch generations: AccessDenied: Access Denied\n\tstatus code: 403, request id: E8D32D1269A45B27:B, host id: IZCpqBpi3kwklJPnOfKVKoHvhRge6CBSzWOVK5xjZxcO2dizyfOe3lwA73OyxebUQFv7bEiJMD+2"
2025-07-17T20:17:08Z app[2867614a9e6758] iad [info] INFO Main child exited normally with code: 1
2025-07-17T20:17:08Z app[2867614a9e6758] iad [info] INFO Starting clean up.
2025-07-17T20:17:08Z app[2867614a9e6758] iad [info] INFO Umounting /dev/vdc from /mnt/name
2025-07-17T20:17:08Z app[2867614a9e6758] iad [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2025-07-17T20:17:08Z app[2867614a9e6758] iad [info][    1.975877] reboot: Restarting system
2025-07-17T20:17:09Z runner[2867614a9e6758] iad [info]machine has reached its max restart count of 10

We need to have the proper config for the policy that handles the s3 for the litestream.

Anyone able to get that live?

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.