"Authentication required to access image" when fly pg create --snapshot-id

Questions / Help Build debugging
1

So I tried to install TimescaleDB extension to my existing postgresql app by deploying a changed version of this repo. Of course, it didn’t work, because I am a Docker noob.
Then I decided that I will “rollback” it by removing all my changes and doing fly deploy from that repo. That didn’t work either.
Then I decided that I will create another postgresql app from a snapshot from time when my original postgresql app was still working.
I did LOG_LEVEL=debug fly pg create --verbose --snapshot-id <SNAPSHOT_ID> but the response was:

DEBUG <-- 200 https://api.fly.io/graphql (425.62ms) {"data":{"createPostgresCluster":null},"errors":[{"message":"Authentication required to access image \"docker.io/library/<MY_DB>:deployment-1647336333\"","locations":[{"line":1,"column":49}],"path":["createPostgresCluster"],"extensions":{"code":"UNAUTHORIZED"}}]}

Any idea what to do? I would like to have new postgresql app running with data from the old one, or fix the old one. :pray: Thanks a lot in advance.

2

Our PG restore feature doesn’t support restores for apps running a custom image. We should have a better error for this.

Do you know what version of postgres you were running before? You can revert to our standard postgres build by running fly deploy -i flyio/postgres:<version> --strategy immediate.

3

Thank you for your reply @kurt
After doing that I can see cmd/keeper.go:719 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"} in my logs.

4

I tried to to fly pg create --snapshot-id with snapshot from before my experiments but it got stuck with this:

runner[60ae6eef] fra [info] Starting instance
runner[60ae6eef] fra [info] Configuring virtual machine
runner[60ae6eef] fra [info] Pulling container image
runner[60ae6eef] fra [info] Unpacking image
runner[60ae6eef] fra [info] Preparing kernel init
runner[60ae6eef] fra [info] Setting up volume 'pg_data'
runner[60ae6eef] fra [info] Configuring firecracker
runner[60ae6eef] fra [info] Starting virtual machine
app[60ae6eef] fra [info] Starting init (commit: 0c50bff)...
app[60ae6eef] fra [info] Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
app[60ae6eef] fra [info] Error: Mount { source: "/dev/vdc", target: "/data", error: EINVAL }
app[60ae6eef] fra [info] [ 0.089493] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
app[60ae6eef] fra [info] [ 0.090452] CPU: 0 PID: 1 Comm: init Not tainted 5.12.2 #1
app[60ae6eef] fra [info] [ 0.091118] Call Trace:
app[60ae6eef] fra [info] [ 0.091401] show_stack+0x52/0x58
app[60ae6eef] fra [info] [ 0.091819] dump_stack+0x6b/0x86
app[60ae6eef] fra [info] [ 0.092229] panic+0xfb/0x2bc
app[60ae6eef] fra [info] [ 0.092625] do_exit.cold+0x60/0xb0
app[60ae6eef] fra [info] [ 0.093053] do_group_exit+0x3b/0xb0
app[60ae6eef] fra [info] [ 0.093507] __x64_sys_exit_group+0x18/0x20
app[60ae6eef] fra [info] [ 0.094105] do_syscall_64+0x38/0x50
app[60ae6eef] fra [info] [ 0.094549] entry_SYSCALL_64_after_hwframe+0x44/0xae
app[60ae6eef] fra [info] [ 0.095183] RIP: 0033:0x6f0675
5

I am not sure what changed, because I tried quite a lot of things, but now it’s working. Even the original DB app is OK, so I didn’t lost any data. :person_shrugging:
I am bit confused, but happy. :smiley:

6

@stefan.luptak Sorry for not getting back to you earlier! The issue with your original DB was that the PRIMARY_REGION environment variable was set to the incorrect region. I tweaked that for you and things came back to life. However, i’m still a little puzzled by the panic… I’ll keep investigating and let you know if I find anything.

7

Thanks a lot for your help @shaun. Probably better error message would help me to solve it by myself. I know you know there’s a room for improvement on the Fly.io side, I just had to mention it. :smile_cat: