Machines are being destroyed by flyd immediately after being created

1

Check this one: https://fly.io/apps/foo-dev-lax/machines/32874599c00938

An HTTP POST operation is performed against /apps/foo-dev-lax/machines with data:

{
    "json": {
        "name": "primary-75520",
        "config": {
            "image": "registry.fly.io/foo-dev:...",
            "env": {},
            "services": [
                {
                    "protocol": "tcp",
                    "internal_port": 80,
                    "ports": [
                        {
                            "port": 80
                        }
                    ]
                },
                {
                    "protocol": "tcp",
                    "internal_port": 443,
                    "ports": [
                        {
                            "port": 443
                        }
                    ]
                },
                {
                    "protocol": "tcp",
                    "internal_port": 32345,
                    "ports": [
                        {
                            "port": 5432,
                            "handlers": [
                                "proxy_proto"
                            ]
                        }
                    ]
                }
            ],
            "guest": {
                "cpu_kind": "performance",
                "cpus": 2,
                "memory_mb": 8192,
                "kernel_args": []
            },
            "mounts": [
                {
                    "volume": "vol_od56vjp8ydovny30",
                    "path": "/data"
                }
            ]
        },
        "region": "lax"
    }
}

The operation results on an HTTP 200 response with the following content:

{
    "id": "32874599c00938",
    "name": "primary-75520",
    "state": "created",
    "region": "lax",
    "instance_id": "01GXY02RGPAF36YSH7RVXSSZCF",
    "private_ip": "fdaa:1:3791:a7b:113:f10d:20f6:2",
    "config": {
        "env": {...},
        "init": {},
        "image": "registry.fly.io/foo-dev:deployment-01GXXP03GTQSDPGG0B0XMS0J2Y",
        "mounts": [
            {
                "encrypted": true,
                "path": "/data",
                "size_gb": 50,
                "volume": "vol_od56vjp8ydovny30",
                "name": "primary_75520"
            }
        ],
        "restart": {},
        "services": [
            {
                "protocol": "tcp",
                "internal_port": 80,
                "ports": [
                    {
                        "port": 80
                    }
                ]
            },
            {
                "protocol": "tcp",
                "internal_port": 443,
                "ports": [
                    {
                        "port": 443
                    }
                ]
            },
            {
                "protocol": "tcp",
                "internal_port": 32345,
                "ports": [
                    {
                        "port": 5432,
                        "handlers": [
                            "proxy_proto"
                        ]
                    }
                ]
            }
        ],
        "guest": {
            "cpu_kind": "performance",
            "cpus": 2,
            "memory_mb": 8192
        }
    },
    "image_ref": {
        "registry": "registry.fly.io",
        "repository": "foo-dev",
        "tag": "...",
        "digest": "...",
        "labels": null
    },
    "created_at": "2023-04-13T19:04:38Z",
    "updated_at": "2023-04-13T19:04:38Z",
    "events": [
        {
            "id": "01GXY02RM9VHBW5NNQN5SRAVFX",
            "type": "launch",
            "status": "created",
            "source": "user",
            "timestamp": 1681412678281
        }
    ]
}

At this point all seems right, but when I go to check the machine either through fly m list or UI it is not there… then looking at machine details in the UI it shows it was destroyed by flyd

Screen Shot 2023-04-13 at 1.20.42 PM
Screen Shot 2023-04-13 at 1.20.42 PM2798×624 34.8 KB

I was able to create and use those machines earlier today, but since about 10:00am PT I’m not able to see them. Tried on iad, mia and lax… all the same.

Any idea whats going on?

2

It looks to be failing when going to pull the image from our registry:

failed to resolve reference "registry.fly.io/bit-dev:deployment-01GXXP03GTQSDPGG0B0XMS0J2Y": registry.fly.io/bit-dev:deployment-01GXXP03GTQSDPGG0B0XMS0J2Y: not found

Which would make sense as we don’t allow pulling an image across apps.

3

Thanks for quick response @JP_Phillips , I’m just wondering if this is something new because I was definitely able to use images from other apps in my organization

4

Ah, right… sorry. Do you happen to be using the new deploy tokens feature?

5

I’m using the token from fly auth token to do the deployments of my fly machines… is that what you are referring to?

FWIW given your previous message suggests image is not found I updated the reference to a newer version and it works… so perhaps the old image was removed from the registry?

6

We recently announced a new type of token specifically for deploying, Deploy Tokens

Yes, we do purge old versions from the registry.

7

I’ll check that now, thanks for pointing me to it!

may I know how frequently?

the reason I’m cross-referencing images between apps is due to the lack of support of private registries.

8

Looking at the history of the bit-dev App, it was created and deleted several times and the image tag reference I pasted above deployment-01GXXP03GTQSDPGG0B0XMS0J2Y is not associated with the current bit-dev App so my guess is it was from a previous one with the same name.

9

That is right, but it was created and deleted because I started experiencing the problem.

FWIW I’m not blocked anymore, though learning more about image persistency will be important for the success of my project.

10

Thanks again!

11

Glad things are working again. When an App is deleted, we purge all images from the registry so that would explain what you experienced.

12

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.