App deployment getting stuck at pushing image to fly

[castulo]

I have been trying to deploy my Rails app for 3 days now, and the deployment keeps getting stuck at “pushing image to fly”.

$ fly deploy -c fly.staging.toml
==> Verifying app config
Validating fly.staging.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
==> Building image
Remote builder fly-builder-hidden-thunder-2933 ready
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 14.6s (23/23) FINISHED
 => [internal] load build definition from Dockerfile                                                                           0.1s
 => => transferring dockerfile: 69B                                                                                            0.1s
 => [internal] load .dockerignore                                                                                              0.1s
 => => transferring context: 72B                                                                                               0.1s
 => resolve image config for docker.io/docker/dockerfile:1                                                                     0.7s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be00  0.0s
 => [internal] load build definition from Dockerfile                                                                           0.0s
 => [internal] load metadata for docker.io/library/ruby:2.7.6-slim                                                             0.2s
 => [internal] load .dockerignore                                                                                              0.0s
 => [base 1/5] FROM docker.io/library/ruby:2.7.6-slim@sha256:4ffb3fa8d4e1edbffa7ce27d57fd005d9b4f0730ea57aaaf50e0c557f5466454  0.0s
 => [internal] load build context                                                                                             13.3s
 => => transferring context: 6.13MB                                                                                           12.8s
 => CACHED [base 2/5] WORKDIR /rails                                                                                           0.0s
 => CACHED [base 3/5] RUN gem update --system --no-document &&     gem install -N bundler                                      0.0s
 => CACHED [base 4/5] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y curl &&     rm -rf /var/lib/ap  0.0s
 => CACHED [base 5/5] RUN curl -sL https://github.com/nodenv/node-build/archive/master.tar.gz | tar xz -C /tmp/ &&     /tmp/n  0.0s
 => CACHED [stage-2 1/4] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y postgresql-client &&     rm  0.0s
 => CACHED [build 1/5] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y build-essential libpq-dev      0.0s
 => CACHED [build 2/5] COPY --link Gemfile Gemfile.lock ./                                                                     0.0s
 => CACHED [build 3/5] RUN bundle install &&     bundle exec bootsnap precompile --gemfile &&     rm -rf ~/.bundle/ $BUNDLE_P  0.0s
 => CACHED [build 4/5] COPY --link . .                                                                                         0.0s
 => CACHED [build 5/5] RUN bundle exec bootsnap precompile app/ lib/                                                           0.0s
 => CACHED [stage-2 2/4] COPY --from=build /usr/local/bundle /usr/local/bundle                                                 0.0s
 => CACHED [stage-2 3/4] COPY --from=build /rails /rails                                                                       0.0s
 => CACHED [stage-2 4/4] RUN useradd rails --create-home --shell /bin/bash &&     chown -R rails:rails db log storage tmp      0.0s
 => exporting to image                                                                                                         0.0s
 => => exporting layers                                                                                                        0.0s
 => => writing image sha256:e1f4bf1f3ba22325dadf9d1269b31382f6ef3011fc732a3d2b06fcf2dd8f650e                                   0.0s
 => => naming to registry.fly.io/petcom-api-staging:deployment-01H96M3G9GM4ZD7GPPMPHAHP7K                                      0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/petcom-api-staging]
57e0202b0f9a: Pushing [=======================>                           ]  2.602GB/5.555GB
03283981f813: Pushing [======================>                            ]   2.54GB/5.631GB
413480dbf8d7: Layer already exists
d18f903b00f4: Layer already exists
be88f50c7ef7: Layer already exists
134fa1c68972: Layer already exists
48c879ffdab3: Layer already exists
eb4ff19f34ef: Layer already exists
38dd6771cb6d: Layer already exists
93840f9e3e37: Layer already exists
c3e35b2afb23: Layer already exists
b7fcac299347: Layer already exists
ec4a38999118: Layer already exists

The pushing always gets to around 3.3GB and then suddenly goes to retrying:

...
==> Pushing image to fly
The push refers to repository [registry.fly.io/petcom-api-staging]
57e0202b0f9a: Retrying in 5 seconds
03283981f813: Retrying in 4 seconds
413480dbf8d7: Layer already exists
d18f903b00f4: Layer already exists
...

It keeps doing the same behavior until eventually it just freezes and doesn’t do anything anymore (the builder machine probably gets suspended after a while).

I have no idea what is going on or how to fix this. Does anybody have a clue?

Thanks

[kaelynH]

Hey there,

Sorry to hear you are having some issues with deploying your app. Would you mind running LOG_LEVEL=debug fly deploy and providing us with the output? I also wonder if running fly deploy --remote-only would work? Here is the doc for reference- fly deploy · Fly Docs

[castulo]

Hi Kaelyn,

This is the log using LOG_LEVEL=debug as requested.

LOG_LEVEL=debug fly deploy -c fly.production.toml
DEBUG Loaded flyctl config from/Users/castulo/.fly/config.yml
DEBUG determined hostname: "Castulos-MBP"
DEBUG determined working directory: "/Users/castulo/workspace/petcom/petcom-api"
DEBUG determined user home directory: "/Users/castulo"
DEBUG determined config directory: "/Users/castulo/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG skipped querying for new release
DEBUG client initialized.
DEBUG app config loaded from fly.production.toml
DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query ($appName: String!) { appbasic:app(name: $appName) { id name platformVersion organization { id slug paidPlan } } }",
  "variables": {
    "appName": "petcom"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (205.39ms)

DEBUG {
  "data": {
    "appbasic": {
      "id": "petcom",
      "name": "petcom",
      "platformVersion": "machines",
      "organization": {
        "id": "896mQD3oYX6ANTBln3aOD7GN5Oc1wVVZy",
        "slug": "madcoyote",
        "paidPlan": false
      }
    }
  }
}

==> Verifying app config
DEBUG Starting task manager
DEBUG Config has metrics token

--> Verified app config
Validating fly.production.toml
Platform: machines
✓ Configuration is valid
DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query ($appName: String!) { appcompact:app(name: $appName) { id name hostname deployed status appUrl platformVersion organization { id slug paidPlan } postgresAppRole: role { name } imageDetails { repository version } } }",
  "variables": {
    "appName": "petcom"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (192.98ms)

DEBUG {
  "data": {
    "appcompact": {
      "id": "petcom",
      "name": "petcom",
      "hostname": "petcom.fly.dev",
      "deployed": true,
      "status": "suspended",
      "appUrl": "https://2a09:8280:1::24:23c4",
      "platformVersion": "machines",
      "organization": {
        "id": "896mQD3oYX6ANTBln3aOD7GN5Oc1wVVZy",
        "slug": "madcoyote",
        "paidPlan": false
      },
      "postgresAppRole": null,
      "imageDetails": {
        "repository": "unknown",
        "version": "unknown"
      }
    }
  }
}

==> Building image
DEBUG --> POST https://api.fly.io/graphql

DEBUG trying remote docker daemon
DEBUG {
  "query": "mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { id slug } } } }",
  "variables": {
    "input": {
      "appName": "petcom",
      "organizationId": null
    }
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (334.76ms)

DEBUG {
  "data": {
    "ensureMachineRemoteBuilder": {
      "machine": {
        "id": "3d8d9296ce4e48",
        "state": "started",
        "ips": {
          "nodes": [
            {
              "family": "v6",
              "kind": "public",
              "ip": "2605:4c40:93:b7a6:0:f3bb:494:1"
            },
            {
              "family": "v4",
              "kind": "private",
              "ip": "172.19.1.242"
            },
            {
              "family": "v6",
              "kind": "privatenet",
              "ip": "fdaa:2:4ac1:a7b:1a8:f3bb:494:2"
            }
          ]
        }
      },
      "app": {
        "name": "fly-builder-young-wildflower-8362",
        "organization": {
          "id": "896mQD3oYX6ANTBln3aOD7GN5Oc1wVVZy",
          "slug": "madcoyote"
        }
      }
    }
  }
}

Waiting for remote builder fly-builder-young-wildflower-8362...
DEBUG checking ip &{Family:v6 Kind:public IP:2605:4c40:93:b7a6:0:f3bb:494:1 MaskSize:0}

DEBUG checking ip &{Family:v4 Kind:private IP:172.19.1.242 MaskSize:0}

DEBUG checking ip &{Family:v6 Kind:privatenet IP:fdaa:2:4ac1:a7b:1a8:f3bb:494:2 MaskSize:0}

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query ($appName: String!) { appbasic:app(name: $appName) { id name platformVersion organization { id slug paidPlan } } }",
  "variables": {
    "appName": "petcom"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (139.83ms)

DEBUG {
  "data": {
    "appbasic": {
      "id": "petcom",
      "name": "petcom",
      "platformVersion": "machines",
      "organization": {
        "id": "896mQD3oYX6ANTBln3aOD7GN5Oc1wVVZy",
        "slug": "madcoyote",
        "paidPlan": false
      }
    }
  }
}

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }",
  "variables": {
    "input": {
      "peerIps": [
        "fdaa:2:4abb:a7b:177d:0:a:2",
        "fdaa:0:e2d1:a7b:177d:0:a:202",
        "fdaa:2:4ac1:a7b:177d:0:a:2"
      ]
    }
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (135.43ms)

DEBUG {
  "data": {
    "validateWireGuardPeers": {
      "invalidPeerIps": []
    }
  }
}

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

Remote builder fly-builder-young-wildflower-8362 ready
DEBUG Remote builder is ready to build!
DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "\n# @genqlient\nmutation ResolverCreateBuild ($input: CreateBuildInput!) {\n\tcreateBuild(input: $input) {\n\t\tid\n\t\tstatus\n\t}\n}\n",
  "variables": {
    "input": {
      "appName": "petcom",
      "builderType": "remote",
      "clientMutationId": "",
      "imageOpts": {
        "buildArgs": {},
        "buildPacks": null,
        "builder": "",
        "builtIn": "",
        "builtInSettings": null,
        "dockerfilePath": "",
        "extraBuildArgs": null,
        "imageLabel": "",
        "imageRef": "",
        "noCache": false,
        "publish": true,
        "tag": "registry.fly.io/petcom:deployment-01H972ND8ZTTXAE08M29ZPMJ4C",
        "target": ""
      },
      "machineId": "",
      "strategiesAvailable": [
        "Buildpacks",
        "Dockerfile",
        "Builtin"
      ]
    }
  },
  "operationName": "ResolverCreateBuild"
}

DEBUG {0x14000cbd020}
DEBUG <-- 200 https://api.fly.io/graphql (138.24ms)

DEBUG {
  "data": {
    "createBuild": {
      "id": "3276978",
      "status": "started"
    }
  }
}

DEBUG Trying 'Buildpacks' strategy

DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>

DEBUG Trying 'Dockerfile' strategy

DEBUG buildkitEnabled true
DEBUG fetching docker server info
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 29.5s (16/22)
 => [internal] load build definition from Dockerfile                                                                             0.1s
[+] Building 59.5s (20/22)
 => [internal] load build definition from Dockerfile                                                                             0.1s
[+] Building 89.5s (21/22)
 => [internal] load build definition from Dockerfile                                                                             0.1s
[+] Building 119.5s (22/23)
 => [internal] load build definition from Dockerfile                                                                             0.1s
[+] Building 136.6s (22/23)
 => [internal] load build definition from Dockerfile                                                                             0.1s
[+] Building 149.4s (23/23) FINISHED
 => [internal] load build definition from Dockerfile                                                                             0.1s
 => => transferring dockerfile: 69B                                                                                              0.1s
 => [internal] load .dockerignore                                                                                                0.1s
 => => transferring context: 72B                                                                                                 0.1s
 => resolve image config for docker.io/docker/dockerfile:1                                                                       0.8s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021  0.0s
 => [internal] load build definition from Dockerfile                                                                             0.0s
 => [internal] load metadata for docker.io/library/ruby:2.7.6-slim                                                               0.2s
 => [internal] load .dockerignore                                                                                                0.0s
 => [base 1/5] FROM docker.io/library/ruby:2.7.6-slim@sha256:4ffb3fa8d4e1edbffa7ce27d57fd005d9b4f0730ea57aaaf50e0c557f5466454    0.0s
 => [internal] load build context                                                                                               12.1s
 => => transferring context: 6.13MB                                                                                             11.6s
 => CACHED [base 2/5] WORKDIR /rails                                                                                             0.0s
 => CACHED [base 3/5] RUN gem update --system --no-document &&     gem install -N bundler                                        0.0s
 => CACHED [base 4/5] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y curl &&     rm -rf /var/lib/apt/  0.0s
 => CACHED [base 5/5] RUN curl -sL https://github.com/nodenv/node-build/archive/master.tar.gz | tar xz -C /tmp/ &&     /tmp/nod  0.0s
 => CACHED [build 1/5] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y build-essential libpq-dev        0.0s
 => CACHED [build 2/5] COPY --link Gemfile Gemfile.lock ./                                                                       0.0s
 => CACHED [build 3/5] RUN bundle install &&     bundle exec bootsnap precompile --gemfile &&     rm -rf ~/.bundle/ $BUNDLE_PAT  0.0s
 => [build 4/5] COPY --link . .                                                                                                 20.3s
 => [build 5/5] RUN bundle exec bootsnap precompile app/ lib/                                                                    0.9s
 => CACHED [stage-2 1/4] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y postgresql-client &&     rm -  0.0s
 => CACHED [stage-2 2/4] COPY --from=build /usr/local/bundle /usr/local/bundle                                                   0.0s
 => [stage-2 3/4] COPY --from=build /rails /rails                                                                               18.1s
 => [stage-2 4/4] RUN useradd rails --create-home --shell /bin/bash &&     chown -R rails:rails db log storage tmp              39.7s
 => exporting to image                                                                                                          33.1s
 => => exporting layers                                                                                                         33.1s
 => => writing image sha256:1f3816299a88ac919a7d141677cf29cef6b4315c9584351aed7b6542f868702b                                     0.0s
 => => naming to registry.fly.io/petcom:deployment-01H972ND8ZTTXAE08M29ZPMJ4C                                                    0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/petcom]
DEBUG metrics error: write tcp [2600:100f:a102:8a14:701e:8aa4:7369:ea2e]:64802->[2a09:8280:1::1c:3475]:443: write: broken pipe

DEBUG metrics error: write tcp [2600:100f:a102:8a14:701e:8aa4:7369:ea2e]:64802->[2a09:8280:1::1c:3475]:443: write: broken pipe

f063e4bccc6f: Preparing
5157e617f8e6: Preparing
f063e4bccc6f: Pushing [===>                                               ]  380.8MB/5.555GB
5157e617f8e6: Pushing [===>                                               ]  379.1MB/5.631GB
f063e4bccc6f: Pushing [======>                                            ]  705.8MB/5.555GB
5157e617f8e6: Pushing [======>                                            ]  704.7MB/5.631GB
f063e4bccc6f: Pushing [=========>                                         ]  1.029GB/5.555GB
5157e617f8e6: Pushing [=========>                                         ]  1.019GB/5.631GB
f063e4bccc6f: Pushing [============>                                      ]  1.396GB/5.555GB
5157e617f8e6: Pushing [===========>                                       ]  1.347GB/5.631GB
f063e4bccc6f: Pushing [===============>                                   ]  1.732GB/5.555GB
5157e617f8e6: Pushing [===============>                                   ]  1.694GB/5.631GB
f063e4bccc6f: Pushing [==================>                                ]  2.071GB/5.555GB
5157e617f8e6: Pushing [==================>                                ]  2.034GB/5.631GB
f063e4bccc6f: Pushing [=====================>                             ]  2.408GB/5.555GB
5157e617f8e6: Pushing [=====================>                             ]  2.392GB/5.631GB
f063e4bccc6f: Pushing [========================>                          ]  2.758GB/5.555GB
5157e617f8e6: Pushing [========================>                          ]  2.744GB/5.631GB
f063e4bccc6f: Pushing [===========================>                       ]  3.096GB/5.555GB
5157e617f8e6: Pushing [===========================>                       ]  3.055GB/5.631GB
f063e4bccc6f: Pushing [==============================>                    ]  3.419GB/5.555GB
5157e617f8e6: Pushing [==============================>                    ]  3.409GB/5.631GB
f063e4bccc6f: Pushing [==>                                                ]  247.5MB/5.555GB
5157e617f8e6: Pushing [==>                                                ]  243.1MB/5.631GB
f063e4bccc6f: Pushing [=====>                                             ]  583.4MB/5.555GB
5157e617f8e6: Pushing [====>                                              ]  541.3MB/5.631GB
f063e4bccc6f: Pushing [========>                                          ]  920.4MB/5.555GB
5157e617f8e6: Pushing [=======>                                           ]  871.8MB/5.631GB
f063e4bccc6f: Pushing [===========>                                       ]  1.258GB/5.555GB
5157e617f8e6: Pushing [==========>                                        ]  1.191GB/5.631GB
f063e4bccc6f: Pushing [==============>                                    ]  1.569GB/5.555GB
5157e617f8e6: Pushing [=============>                                     ]  1.528GB/5.631GB
f063e4bccc6f: Pushing [=================>                                 ]  1.914GB/5.555GB
5157e617f8e6: Pushing [================>                                  ]  1.865GB/5.631GB
f063e4bccc6f: Pushing [====================>                              ]  2.286GB/5.555GB
5157e617f8e6: Pushing [====================>                              ]  2.271GB/5.631GB
f063e4bccc6f: Pushing [=======================>                           ]  2.612GB/5.555GB
5157e617f8e6: Pushing [=======================>                           ]  2.607GB/5.631GB
f063e4bccc6f: Pushing [==========================>                        ]  2.956GB/5.555GB
5157e617f8e6: Pushing [==========================>                        ]  2.943GB/5.631GB
f063e4bccc6f: Pushing [=============================>                     ]  3.296GB/5.555GB
5157e617f8e6: Pushing [=============================>                     ]  3.281GB/5.631GB
f063e4bccc6f: Pushing [>                                                  ]  55.25MB/5.555GB
5157e617f8e6: Pushing [>                                                  ]  88.02MB/5.631GB
f063e4bccc6f: Pushing [====>                                              ]  449.6MB/5.555GB
5157e617f8e6: Pushing [===>                                               ]  447.9MB/5.631GB
f063e4bccc6f: Pushing [=======>                                           ]  789.9MB/5.555GB
5157e617f8e6: Pushing [======>                                            ]  785.5MB/5.631GB
f063e4bccc6f: Pushing [==========>                                        ]  1.129GB/5.555GB
5157e617f8e6: Pushing [=========>                                         ]  1.124GB/5.631GB
f063e4bccc6f: Pushing [=============>                                     ]  1.465GB/5.555GB
5157e617f8e6: Pushing [============>                                      ]  1.459GB/5.631GB
f063e4bccc6f: Pushing [================>                                  ]  1.821GB/5.555GB
5157e617f8e6: Pushing [===============>                                   ]  1.799GB/5.631GB
f063e4bccc6f: Pushing [===================>                               ]  2.172GB/5.555GB
5157e617f8e6: Pushing [==================>                                ]  2.135GB/5.631GB
f063e4bccc6f: Pushing [======================>                            ]  2.543GB/5.555GB
5157e617f8e6: Pushing [=====================>                             ]  2.471GB/5.631GB
f063e4bccc6f: Pushing [=========================>                         ]  2.814GB/5.555GB
5157e617f8e6: Pushing [========================>                          ]  2.809GB/5.631GB
f063e4bccc6f: Pushing [============================>                      ]  3.183GB/5.555GB
5157e617f8e6: Pushing [===========================>                       ]  3.147GB/5.631GB
f063e4bccc6f: Pushing [===============================>                   ]   3.49GB/5.555GB
5157e617f8e6: Retrying in 15 seconds
f063e4bccc6f: Pushing [=>                                                 ]  178.1MB/5.555GB
5157e617f8e6: Pushing [=>                                                 ]  213.1MB/5.631GB
f063e4bccc6f: Pushing [====>                                              ]  515.1MB/5.555GB
5157e617f8e6: Pushing [====>                                              ]  547.9MB/5.631GB
f063e4bccc6f: Pushing [=======>                                           ]  852.7MB/5.555GB
5157e617f8e6: Pushing [=======>                                           ]  886.6MB/5.631GB
f063e4bccc6f: Pushing [==========>                                        ]   1.19GB/5.555GB
5157e617f8e6: Pushing [==========>                                        ]  1.232GB/5.631GB
f063e4bccc6f: Pushing [=============>                                     ]  1.539GB/5.555GB
5157e617f8e6: Pushing [==============>                                    ]  1.578GB/5.631GB
f063e4bccc6f: Pushing [================>                                  ]  1.873GB/5.555GB
5157e617f8e6: Pushing [================>                                  ]  1.901GB/5.631GB
f063e4bccc6f: Pushing [===================>                               ]  2.216GB/5.555GB
5157e617f8e6: Pushing [====================>                              ]  2.266GB/5.631GB
f063e4bccc6f: Pushing [======================>                            ]   2.54GB/5.555GB
5157e617f8e6: Pushing [=======================>                           ]  2.608GB/5.631GB
f063e4bccc6f: Pushing [=========================>                         ]   2.88GB/5.555GB
5157e617f8e6: Pushing [==========================>                        ]  2.943GB/5.631GB
f063e4bccc6f: Pushing [============================>                      ]  3.174GB/5.555GB
5157e617f8e6: Pushing [=============================>                     ]  3.282GB/5.631GB
f063e4bccc6f: Retrying in 13 seconds
5157e617f8e6: Retrying in 11 seconds
f063e4bccc6f: Pushing [==>                                                ]  243.1MB/5.555GB
5157e617f8e6: Pushing [==>                                                ]    242MB/5.631GB
f063e4bccc6f: Pushing [====>                                              ]  554.5MB/5.555GB
5157e617f8e6: Pushing [====>                                              ]  553.9MB/5.631GB
f063e4bccc6f: Pushing [=======>                                           ]  869.6MB/5.555GB
5157e617f8e6: Pushing [=======>                                           ]  868.5MB/5.631GB
f063e4bccc6f: Pushing [==========>                                        ]  1.192GB/5.555GB
5157e617f8e6: Pushing [==========>                                        ]  1.191GB/5.631GB
f063e4bccc6f: Pushing [=============>                                     ]  1.529GB/5.555GB
5157e617f8e6: Pushing [=============>                                     ]  1.528GB/5.631GB
f063e4bccc6f: Pushing [================>                                  ]  1.866GB/5.555GB
5157e617f8e6: Pushing [================>                                  ]  1.866GB/5.631GB
f063e4bccc6f: Pushing [===================>                               ]  2.193GB/5.555GB
5157e617f8e6: Pushing [===================>                               ]  2.185GB/5.631GB
f063e4bccc6f: Pushing [======================>                            ]  2.541GB/5.555GB
5157e617f8e6: Pushing [======================>                            ]  2.525GB/5.631GB
f063e4bccc6f: Pushing [=========================>                         ]  2.877GB/5.555GB
5157e617f8e6: Pushing [=========================>                         ]  2.848GB/5.631GB
f063e4bccc6f: Pushing [============================>                      ]  3.185GB/5.555GB
5157e617f8e6: Pushing [============================>                      ]  3.176GB/5.631GB
f063e4bccc6f: Pushing [==================================================>]  5.555GB/5.555GB
5157e617f8e6: Pushing [==================================================>]  5.631GB/5.631GB

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG --> POST https://api.fly.io/graphql

DEBUG result image:<nil> error:error rendering push status stream: unknown: unknown error

DEBUG {
  "query": "\n# @genqlient\nmutation ResolverFinishBuild ($input: FinishBuildInput!) {\n\tfinishBuild(input: $input) {\n\t\tid\n\t\tstatus\n\t\twallclockTimeMs\n\t}\n}\n",
  "variables": {
    "input": {
      "appName": "petcom",
      "buildId": "3276978",
      "builderMeta": {
        "builderType": "",
        "buildkitEnabled": true,
        "dockerVersion": "20.10.12",
        "platform": "linux/x86_64/3.15.0",
        "remoteAppName": "",
        "remoteMachineId": ""
      },
      "clientMutationId": "",
      "finalImage": {
        "id": "",
        "sizeBytes": 0,
        "tag": ""
      },
      "logs": "error rendering push status stream: unknown: unknown error",
      "machineId": "",
      "status": "failed",
      "strategiesAttempted": [
        {
          "error": "",
          "note": "no buildpack builder configured, skipping",
          "result": "failed",
          "strategy": "Buildpacks"
        },
        {
          "error": "error rendering push status stream: unknown: unknown error",
          "note": "",
          "result": "failed",
          "strategy": "Dockerfile"
        }
      ],
      "timings": {
        "buildAndPushMs": 1734586,
        "buildMs": 149675,
        "builderInitMs": 37,
        "contextBuildMs": -1,
        "imageBuildMs": 149638,
        "pushMs": 1584911
      }
    }
  },
  "operationName": "ResolverFinishBuild"
}

DEBUG {0x14001afcde0}
DEBUG <-- 200 https://api.fly.io/graphql (225.31ms)

DEBUG {
  "data": {
    "finishBuild": {
      "id": "3276978",
      "status": "failed",
      "wallclockTimeMs": 1734821
    }
  }
}

DEBUG Task manager done
Error: failed to fetch an image or build from source: error rendering push status stream: unknown: unknown error
DEBUG metrics error: write tcp [2600:100f:a102:8a14:701e:8aa4:7369:ea2e]:64802->[2a09:8280:1::1c:3475]:443: write: broken pipe

DEBUG metrics error: write tcp [2600:100f:a102:8a14:701e:8aa4:7369:ea2e]:64802->[2a09:8280:1::1c:3475]:443: write: broken pipe

[castulo]

Using --remote-only doesn’t work either, it also hangs.
This is the log:

LOG_LEVEL=debug fly deploy --remote-only  -c fly.production.toml
DEBUG Loaded flyctl config from/Users/castulo/.fly/config.yml
DEBUG determined hostname: "Castulos-MBP"
DEBUG determined working directory: "/Users/castulo/workspace/petcom/petcom-api"
DEBUG determined user home directory: "/Users/castulo"
DEBUG determined config directory: "/Users/castulo/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG client initialized.
DEBUG app config loaded from fly.production.toml
DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query ($appName: String!) { appbasic:app(name: $appName) { id name platformVersion organization { id slug paidPlan } } }",
  "variables": {
    "appName": "petcom"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (315.31ms)

DEBUG {
  "data": {
    "appbasic": {
      "id": "petcom",
      "name": "petcom",
      "platformVersion": "machines",
      "organization": {
        "id": "896mQD3oYX6ANTBln3aOD7GN5Oc1wVVZy",
        "slug": "madcoyote",
        "paidPlan": false
      }
    }
  }
}

==> Verifying app config
DEBUG Starting task manager
DEBUG started querying for new release
Validating fly.production.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
DEBUG Config has metrics token

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query ($appName: String!) { appcompact:app(name: $appName) { id name hostname deployed status appUrl platformVersion organization { id slug paidPlan } postgresAppRole: role { name } imageDetails { repository version } } }",
  "variables": {
    "appName": "petcom"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (140.15ms)

DEBUG {
  "data": {
    "appcompact": {
      "id": "petcom",
      "name": "petcom",
      "hostname": "petcom.fly.dev",
      "deployed": true,
      "status": "suspended",
      "appUrl": "https://2a09:8280:1::24:23c4",
      "platformVersion": "machines",
      "organization": {
        "id": "896mQD3oYX6ANTBln3aOD7GN5Oc1wVVZy",
        "slug": "madcoyote",
        "paidPlan": false
      },
      "postgresAppRole": null,
      "imageDetails": {
        "repository": "unknown",
        "version": "unknown"
      }
    }
  }
}

==> Building image
DEBUG trying remote docker daemon
DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { id slug } } } }",
  "variables": {
    "input": {
      "appName": "petcom",
      "organizationId": null
    }
  }
}


DEBUG {}
DEBUG querying for release resulted to 0.1.85
DEBUG <-- 200 https://api.fly.io/graphql (356.2ms)

DEBUG {
  "data": {
    "ensureMachineRemoteBuilder": {
      "machine": {
        "id": "3d8d9296ce4e48",
        "state": "started",
        "ips": {
          "nodes": [
            {
              "family": "v6",
              "kind": "public",
              "ip": "2605:4c40:93:b7a6:0:f3bb:494:1"
            },
            {
              "family": "v4",
              "kind": "private",
              "ip": "172.19.1.242"
            },
            {
              "family": "v6",
              "kind": "privatenet",
              "ip": "fdaa:2:4ac1:a7b:1a8:f3bb:494:2"
            }
          ]
        }
      },
      "app": {
        "name": "fly-builder-young-wildflower-8362",
        "organization": {
          "id": "896mQD3oYX6ANTBln3aOD7GN5Oc1wVVZy",
          "slug": "madcoyote"
        }
      }
    }
  }
}

DEBUG checking ip &{Family:v6 Kind:public IP:2605:4c40:93:b7a6:0:f3bb:494:1 MaskSize:0}

DEBUG checking ip &{Family:v4 Kind:private IP:172.19.1.242 MaskSize:0}

DEBUG checking ip &{Family:v6 Kind:privatenet IP:fdaa:2:4ac1:a7b:1a8:f3bb:494:2 MaskSize:0}

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query ($appName: String!) { appbasic:app(name: $appName) { id name platformVersion organization { id slug paidPlan } } }",
  "variables": {
    "appName": "petcom"
  }
}


DEBUG {}
Waiting for remote builder fly-builder-young-wildflower-8362... 🌍DEBUG <-- 200 https://api.fly.io/graphql (127.59ms)

DEBUG {
  "data": {
    "appbasic": {
      "id": "petcom",
      "name": "petcom",
      "platformVersion": "machines",
      "organization": {
        "id": "896mQD3oYX6ANTBln3aOD7GN5Oc1wVVZy",
        "slug": "madcoyote",
        "paidPlan": false
      }
    }
  }
}

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }",
  "variables": {
    "input": {
      "peerIps": [
        "fdaa:2:4ac1:a7b:177d:0:a:2",
        "fdaa:2:4abb:a7b:177d:0:a:2",
        "fdaa:0:e2d1:a7b:177d:0:a:202"
      ]
    }
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (122.13ms)

DEBUG {
  "data": {
    "validateWireGuardPeers": {
      "invalidPeerIps": []
    }
  }
}

Waiting for remote builder fly-builder-young-wildflower-8362... 🌏DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

Waiting for remote builder fly-builder-young-wildflower-8362... 🌍DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

Waiting for remote builder fly-builder-young-wildflower-8362... 🌎DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

Waiting for remote builder fly-builder-young-wildflower-8362... 🌏DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder available, but pinging again in 50ms to be sure

Waiting for remote builder fly-builder-young-wildflower-8362... 🌍DEBUG Remote builder available, but pinging again in 50ms to be sure

DEBUG Remote builder is ready to build!
Remote builder fly-builder-young-wildflower-8362 ready
DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "\n# @genqlient\nmutation ResolverCreateBuild ($input: CreateBuildInput!) {\n\tcreateBuild(input: $input) {\n\t\tid\n\t\tstatus\n\t}\n}\n",
  "variables": {
    "input": {
      "appName": "petcom",
      "builderType": "remote",
      "clientMutationId": "",
      "imageOpts": {
        "buildArgs": {},
        "buildPacks": null,
        "builder": "",
        "builtIn": "",
        "builtInSettings": null,
        "dockerfilePath": "",
        "extraBuildArgs": null,
        "imageLabel": "",
        "imageRef": "",
        "noCache": false,
        "publish": true,
        "tag": "registry.fly.io/petcom:deployment-01H974M5JFMGBF0CVH1Y7X0G74",
        "target": ""
      },
      "machineId": "",
      "strategiesAvailable": [
        "Buildpacks",
        "Dockerfile",
        "Builtin"
      ]
    }
  },
  "operationName": "ResolverCreateBuild"
}

DEBUG {0x14000628e10}
DEBUG <-- 200 https://api.fly.io/graphql (135.99ms)

DEBUG {
  "data": {
    "createBuild": {
      "id": "3277409",
      "status": "started"
    }
  }
}

DEBUG Trying 'Buildpacks' strategy

DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>

DEBUG Trying 'Dockerfile' strategy

DEBUG buildkitEnabled true
DEBUG fetching docker server info
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 29.4s (18/22)
 => [internal] load build definition from Dockerfile                                                                           0.1s
[+] Building 59.4s (20/22)
 => [internal] load build definition from Dockerfile                                                                           0.1s
[+] Building 89.5s (21/22)
 => [internal] load build definition from Dockerfile                                                                           0.1s
[+] Building 119.5s (22/23)
 => [internal] load build definition from Dockerfile                                                                           0.1s
[+] Building 136.7s (22/23)
 => [internal] load build definition from Dockerfile                                                                           0.1s
[+] Building 146.5s (23/23) FINISHED
 => [internal] load build definition from Dockerfile                                                                           0.1s
 => => transferring dockerfile: 69B                                                                                            0.1s
 => [internal] load .dockerignore                                                                                              0.1s
 => => transferring context: 72B                                                                                               0.1s
 => resolve image config for docker.io/docker/dockerfile:1                                                                     0.6s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be00  0.0s
 => [internal] load build definition from Dockerfile                                                                           0.0s
 => [internal] load metadata for docker.io/library/ruby:2.7.6-slim                                                             0.2s
 => [internal] load .dockerignore                                                                                              0.0s
 => [internal] load build context                                                                                              6.5s
 => => transferring context: 6.14MB                                                                                            6.5s
 => [base 1/5] FROM docker.io/library/ruby:2.7.6-slim@sha256:4ffb3fa8d4e1edbffa7ce27d57fd005d9b4f0730ea57aaaf50e0c557f5466454  0.0s
 => CACHED [base 2/5] WORKDIR /rails                                                                                           0.0s
 => CACHED [base 3/5] RUN gem update --system --no-document &&     gem install -N bundler                                      0.0s
 => CACHED [base 4/5] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y curl &&     rm -rf /var/lib/ap  0.0s
 => CACHED [base 5/5] RUN curl -sL https://github.com/nodenv/node-build/archive/master.tar.gz | tar xz -C /tmp/ &&     /tmp/n  0.0s
 => CACHED [build 1/5] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y build-essential libpq-dev      0.0s
 => CACHED [build 2/5] COPY --link Gemfile Gemfile.lock ./                                                                     0.0s
 => CACHED [build 3/5] RUN bundle install &&     bundle exec bootsnap precompile --gemfile &&     rm -rf ~/.bundle/ $BUNDLE_P  0.0s
 => [build 4/5] COPY --link . .                                                                                               19.2s
 => [build 5/5] RUN bundle exec bootsnap precompile app/ lib/                                                                  0.8s
 => CACHED [stage-2 1/4] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y postgresql-client &&     rm  0.0s
 => CACHED [stage-2 2/4] COPY --from=build /usr/local/bundle /usr/local/bundle                                                 0.0s
 => [stage-2 3/4] COPY --from=build /rails /rails                                                                             19.7s
 => [stage-2 4/4] RUN useradd rails --create-home --shell /bin/bash &&     chown -R rails:rails db log storage tmp            39.5s
 => exporting to image                                                                                                        34.4s
 => => exporting layers                                                                                                       34.4s
 => => writing image sha256:79842774a443aa57fee4f902dc611ba709c079d97134010547178c0dac755041                                   0.0s
 => => naming to registry.fly.io/petcom:deployment-01H974M5JFMGBF0CVH1Y7X0G74                                                  0.0s
--> Building image done
==> Pushing image to fly
DEBUG metrics error: write tcp [2600:100f:a102:8a14:701e:8aa4:7369:ea2e]:65120->[2a09:8280:1::1c:3475]:443: write: broken pipe

The push refers to repository [registry.fly.io/petcom]
DEBUG metrics error: write tcp [2600:100f:a102:8a14:701e:8aa4:7369:ea2e]:65120->[2a09:8280:1::1c:3475]:443: write: broken pipe

378162d442de: Pushing [>                                                  ]  29.13MB/5.555GB
4fe964d19552: Pushing [>                                                  ]  29.12MB/5.631GB
378162d442de: Pushing [===>                                               ]  398.3MB/5.555GB
4fe964d19552: Pushing [===>                                               ]  379.1MB/5.631GB
378162d442de: Pushing [======>                                            ]  725.4MB/5.555GB
4fe964d19552: Pushing [======>                                            ]  717.2MB/5.631GB
378162d442de: Pushing [=========>                                         ]  1.082GB/5.555GB
4fe964d19552: Pushing [=========>                                         ]  1.056GB/5.631GB
378162d442de: Pushing [============>                                      ]  1.439GB/5.555GB
4fe964d19552: Pushing [============>                                      ]  1.393GB/5.631GB
378162d442de: Pushing [===============>                                   ]  1.766GB/5.555GB
4fe964d19552: Pushing [===============>                                   ]  1.731GB/5.631GB
378162d442de: Pushing [===================>                               ]  2.136GB/5.555GB
4fe964d19552: Pushing [==================>                                ]  2.069GB/5.631GB
378162d442de: Pushing [======================>                            ]  2.474GB/5.555GB
4fe964d19552: Pushing [=====================>                             ]  2.413GB/5.631GB
378162d442de: Pushing [=========================>                         ]  2.811GB/5.555GB
4fe964d19552: Pushing [========================>                          ]  2.747GB/5.631GB
378162d442de: Pushing [============================>                      ]  3.148GB/5.555GB
4fe964d19552: Pushing [===========================>                       ]  3.109GB/5.631GB
378162d442de: Pushing [===============================>                   ]  3.486GB/5.555GB
4fe964d19552: Pushing [==============================>                    ]  3.446GB/5.631GB
378162d442de: Pushing [==>                                                ]  244.8MB/5.555GB
4fe964d19552: Pushing [==>                                                ]  283.6MB/5.631GB
378162d442de: Pushing [=====>                                             ]  581.2MB/5.555GB
4fe964d19552: Pushing [=====>                                             ]  597.6MB/5.631GB
378162d442de: Pushing [========>                                          ]  918.8MB/5.555GB
4fe964d19552: Pushing [========>                                          ]  920.4MB/5.631GB
378162d442de: Pushing [===========>                                       ]  1.245GB/5.555GB
4fe964d19552: Pushing [===========>                                       ]  1.257GB/5.631GB
378162d442de: Pushing [==============>                                    ]  1.584GB/5.555GB
4fe964d19552: Pushing [==============>                                    ]  1.595GB/5.631GB
378162d442de: Pushing [=================>                                 ]  1.909GB/5.555GB
4fe964d19552: Pushing [=================>                                 ]  1.946GB/5.631GB
378162d442de: Pushing [====================>                              ]  2.226GB/5.555GB
4fe964d19552: Pushing [====================>                              ]  2.271GB/5.631GB
378162d442de: Pushing [======================>                            ]  2.449GB/5.555GB
4fe964d19552: Pushing [=====================>                             ]  2.472GB/5.631GB

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Remote builder heartbeat pulse failed: Get "http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline": connect tcp [fdaa:2:4ac1:a7b:1a8:f3bb:494:2]:8080: operation timed out

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Remote builder heartbeat pulse failed: Get "http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline": connect tcp [fdaa:2:4ac1:a7b:1a8:f3bb:494:2]:8080: operation timed out

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Remote builder heartbeat pulse failed: Get "http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline": connect tcp [fdaa:2:4ac1:a7b:1a8:f3bb:494:2]:8080: operation timed out

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Remote builder heartbeat pulse failed: Get "http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline": connect tcp [fdaa:2:4ac1:a7b:1a8:f3bb:494:2]:8080: operation timed out

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Remote builder heartbeat pulse failed: Get "http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline": connect tcp [fdaa:2:4ac1:a7b:1a8:f3bb:494:2]:8080: operation timed out

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Remote builder heartbeat pulse failed: Get "http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline": connect tcp [fdaa:2:4ac1:a7b:1a8:f3bb:494:2]:8080: operation timed out

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Remote builder heartbeat pulse failed: Get "http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline": connect tcp [fdaa:2:4ac1:a7b:1a8:f3bb:494:2]:8080: operation timed out

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

DEBUG Remote builder heartbeat pulse failed: Get "http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline": connect tcp [fdaa:2:4ac1:a7b:1a8:f3bb:494:2]:8080: operation timed out

DEBUG Sending remote builder heartbeat pulse to http://fdaa:2:4ac1:a7b:1a8:f3bb:494:2:8080/flyio/v1/extendDeadline...

[castulo]

Any idea what could be happening?

[kaelynH]

Hey thank you so much for providing all of that. I appreciate it! I keep getting hung up on the tcp part of the error… I’m wondering if there is some sort of network error/issue. I did some digging and found this other forum post. Do you mind performing the recommended steps in the post? Here’s the link:

Eror "Failed to start remote builder heartbeat: failed building options: failed probing "personal": context deadline exceeded"

when I deploy the application it’s always like that. i use windows . This is my first time using fly.io is there a way to solve it [image]

[castulo]

@kaelynH

… I’m wondering if there

I tried using fly wireguard reset, and fly wireguard websockets enable and it didn’t work. The deployment keeps failing exactly the same, it starts pushing to fly, it gets to about 3.5 GB and resets the push, and starts again from 0, it retries in general like 3 times, and fails.

fly doctor also didn’t show any issue.

This is my first time using fly, I was using Heroku before, and never had this kind of issues, this makes me question if I really want to use fly for my production app, it seems very unreliable.

[kaelynH]

I totally get where you are coming from. I want to suggest a few more things if that’s alright. Do you mind deploying with --local-only and see if that gets you anywhere different? So fly deploy --local-only.

Also is there anyway that the image you are using/have could be slimmed down a bit? I’m curious if that could be causing any issues at all.

[castulo]

Thanks so much for trying to help out Kaelyn.

So I tried using --local-only as you suggested, but that failed too. With a different error. This is the log from that:

$ LOG_LEVEL=debug fly deploy --local-only -c fly.production.toml
DEBUG Loaded flyctl config from/Users/castulo/.fly/config.yml
DEBUG determined hostname: "Castulos-MBP"
DEBUG determined working directory: "/Users/castulo/workspace/petcom/petcom-api"
DEBUG determined user home directory: "/Users/castulo"
DEBUG determined config directory: "/Users/castulo/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG client initialized.
DEBUG app config loaded from fly.production.toml
DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query ($appName: String!) { appbasic:app(name: $appName) { id name platformVersion organization { id slug paidPlan } } }",
  "variables": {
    "appName": "petcom"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (256.05ms)

DEBUG {
  "data": {
    "appbasic": {
      "id": "petcom",
      "name": "petcom",
      "platformVersion": "machines",
      "organization": {
        "id": "896mQD3oYX6ANTBln3aOD7GN5Oc1wVVZy",
        "slug": "madcoyote",
        "paidPlan": false
      }
    }
  }
}

==> Verifying app config
DEBUG Starting task manager
DEBUG started querying for new release
Validating fly.production.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
DEBUG Config has metrics token

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query ($appName: String!) { appcompact:app(name: $appName) { id name hostname deployed status appUrl platformVersion organization { id slug paidPlan } postgresAppRole: role { name } imageDetails { repository version } } }",
  "variables": {
    "appName": "petcom"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (140.89ms)

DEBUG {
  "data": {
    "appcompact": {
      "id": "petcom",
      "name": "petcom",
      "hostname": "petcom.fly.dev",
      "deployed": true,
      "status": "suspended",
      "appUrl": "https://2a09:8280:1::24:23c4",
      "platformVersion": "machines",
      "organization": {
        "id": "896mQD3oYX6ANTBln3aOD7GN5Oc1wVVZy",
        "slug": "madcoyote",
        "paidPlan": false
      },
      "postgresAppRole": null,
      "imageDetails": {
        "repository": "unknown",
        "version": "unknown"
      }
    }
  }
}

==> Building image
DEBUG trying local docker daemon
DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "\n# @genqlient\nmutation ResolverCreateBuild ($input: CreateBuildInput!) {\n\tcreateBuild(input: $input) {\n\t\tid\n\t\tstatus\n\t}\n}\n",
  "variables": {
    "input": {
      "appName": "petcom",
      "builderType": "local",
      "clientMutationId": "",
      "imageOpts": {
        "buildArgs": {},
        "buildPacks": null,
        "builder": "",
        "builtIn": "",
        "builtInSettings": null,
        "dockerfilePath": "",
        "extraBuildArgs": null,
        "imageLabel": "",
        "imageRef": "",
        "noCache": false,
        "publish": true,
        "tag": "registry.fly.io/petcom:deployment-01H9KZ2N3F3YZW66P7QW2XGBRB",
        "target": ""
      },
      "machineId": "",
      "strategiesAvailable": [
        "Buildpacks",
        "Dockerfile",
        "Builtin"
      ]
    }
  },
  "operationName": "ResolverCreateBuild"
}

DEBUG {0x140006d8120}
DEBUG querying for release resulted to 0.1.87
DEBUG <-- 200 https://api.fly.io/graphql (199.75ms)

DEBUG {
  "data": {
    "createBuild": {
      "id": "3370686",
      "status": "started"
    }
  }
}

DEBUG Trying 'Buildpacks' strategy

DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>

DEBUG Trying 'Dockerfile' strategy

DEBUG buildkitEnabled true
DEBUG fetching docker server info
==> Building image with Docker
--> docker host: 24.0.5 linux aarch64
[+] Building 895.6s (14/20)
 => [internal] load .dockerignore                                                                                               0.0s
 => => transferring context: 793B                                                                                               0.0s
 => [internal] load build definition from Dockerfile                                                                            0.0s
 => => transferring dockerfile: 2.48kB                                                                                          0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                      1.6s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be002  0.0s
 => [internal] load metadata for docker.io/library/ruby:2.7.6-slim                                                              1.4s
 => [base 1/5] FROM docker.io/library/ruby:2.7.6-slim@sha256:4ffb3fa8d4e1edbffa7ce27d57fd005d9b4f0730ea57aaaf50e0c557f5466454   0.0s
 => [internal] load build context                                                                                               5.9s
 => => transferring context: 6.17MB                                                                                             5.6s
 => CACHED [base 2/5] WORKDIR /rails                                                                                            0.0s
 => CACHED [base 3/5] RUN gem update --system --no-document &&     gem install -N bundler                                       0.0s
 => CACHED [base 4/5] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y curl &&     rm -rf /var/lib/apt  0.0s
 => CACHED [base 5/5] RUN curl -sL https://github.com/nodenv/node-build/archive/master.tar.gz | tar xz -C /tmp/ &&     /tmp/no  0.0s
 => CACHED [stage-2 1/4] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y postgresql-client &&     rm   0.0s
 => CACHED [build 1/5] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y build-essential libpq-dev       0.0s
 => CACHED [build 2/5] COPY --link Gemfile Gemfile.lock ./                                                                      0.0s
 => [build 3/5] RUN bundle install &&     bundle exec bootsnap precompile --gemfile &&     rm -rf ~/.bundle/ $BUNDLE_PATH/ru  886.5s
 => => # Fetching timeout 0.3.2
 => => # Installing timeout 0.3.2
 => => # Fetching version_gem 1.1.1
 => => # Installing version_gem 1.1.1
 => => # Fetching pg 1.4.5
 => => # Installing pg 1.4.5 with native extensions
DEBUG result image:<nil> error:error building: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "\n# @genqlient\nmutation ResolverFinishBuild ($input: FinishBuildInput!) {\n\tfinishBuild(input: $input) {\n\t\tid\n\t\tstatus\n\t\twallclockTimeMs\n\t}\n}\n",
  "variables": {
    "input": {
      "appName": "petcom",
      "buildId": "3370686",
      "builderMeta": {
        "builderType": "local",
        "buildkitEnabled": true,
        "dockerVersion": "24.0.5",
        "platform": "linux/aarch64/",
        "remoteAppName": "",
        "remoteMachineId": ""
      },
      "clientMutationId": "",
      "finalImage": {
        "id": "",
        "sizeBytes": 0,
        "tag": ""
      },
      "logs": "error building: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF",
      "machineId": "",
      "status": "failed",
      "strategiesAttempted": [
        {
          "error": "",
          "note": "no buildpack builder configured, skipping",
          "result": "failed",
          "strategy": "Buildpacks"
        },
        {
          "error": "error building: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF",
          "note": "",
          "result": "failed",
          "strategy": "Dockerfile"
        }
      ],
      "timings": {
        "buildAndPushMs": 900722,
        "buildMs": 895600,
        "builderInitMs": 6,
        "contextBuildMs": -1,
        "imageBuildMs": 895594,
        "pushMs": -1
      }
    }
  },
  "operationName": "ResolverFinishBuild"
}

DEBUG {0x1400163c990}
DEBUG <-- 200 https://api.fly.io/graphql (211.32ms)

DEBUG {
  "data": {
    "finishBuild": {
      "id": "3370686",
      "status": "failed",
      "wallclockTimeMs": 900941
    }
  }
}

DEBUG Task manager done
Error: failed to fetch an image or build from source: error building: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF

[kaelynH]

You’re welcome, sorry it’s been a hassle trying to get this app deployed. I looked up that error you received outside of fly. To see if its something to do with your application, instead of the issue relating to fly. I wonder if it has something to do with Docker? I found this here- Error: failed to receive status: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = "error reading from server: io: read/write on closed pipe", received prior goaway: code: NO_ERROR · Issue #1079 · docker/buildx · GitHub. Take a look at this and see if it helps, specifically, this section of that page- Error: failed to receive status: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = "error reading from server: io: read/write on closed pipe", received prior goaway: code: NO_ERROR · Issue #1079 · docker/buildx · GitHub.

If you look up what defaultKeepStorage is actually used for it states:

Garbage collection runs in the BuildKit daemon. The daemon clears the build cache when the cache size becomes too big, or when the cache age expires

Here’s the link to that as well- Build garbage collection | Docker Docs

I wonder if this has anything to do with your app, because of the size of it?

[castulo]

I tried increasing the size of the defaultKeepStorage and the virtual disk limit as suggested in that post and also didn’t help.

If I try building the docker image using just docker with:
docker build -t my-app .

The image gets built correctly, so I don’t think the issue is with docker, I think the problem is with whatever else fly does during the fly deploy command.

[kaelynH]

In all the attempts to deploy, was there ever a fly.toml file that was generated for you? If so do you mind sharing that?

[castulo]

Sure Kaelyn, this is my toml file:

cat fly.production.toml
# fly.toml app configuration file generated for petcom on 2023-06-01T18:29:17-07:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = "petcom"
primary_region = "gdl"
console_command = "/rails/bin/rails console"

[http_service]
  internal_port = 3000
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0

[[statics]]
  guest_path = "/rails/public"
  url_prefix = "/"

[castulo]

I think something must have happened with the VM where my app is being deployed. I just tried cloning my repo again and deploying a new app there and everything worked fine. And actually, the deployment was very fast, and the image size seems to be very small (about 490 MB).

I had been noticing that every time I would deploy a new version of my app it would get slower and slower in the deployment, and as time went by, the deployment would even start failing and I would need to attempt the deploy 2 or 3 times before it succeeded until it got to the point where it just won’t deploy anymore. Also, the image is very big, at ~5.5 GB, I don’t know how it got that big.

I guess I could just destroy my current fly app and deploy it from scratch since that seems to work, and it is still in development. But that makes me wonder if fly is really a reliable option for deploying a production app, if this happened when my app is in production and I’m stuck not being able to deploy new versions of my app, that would really suck.

[kaelynH]

I appreciate you going back and forth with me trying to get this solved. I should have asked you sooner, what region you were deploying to. I should of asked for the fly toml sooner as well. That’s my bad. I had a feeling the issue was tied to the region you were deploying to. I want to just reassure you that we have a lot of people who deploy their app onto our platform. Apps that are in development and need to be in production. I know it wasn’t a smooth and simple process for you out the gate, but maybe it’s all for good reason?

From here on out because you had a harder “first deployment” you will now know what to do if something like this happens again, and also, know how to go about troubleshooting it from Fly’s end.

Thank you for coming back to the community forum and posting updates, as it not only helps us get a better sense of what’s going on, but helps others who might be going through the same problem. You have a lot of support here, not just from me but from others who work at Fly, and people within the community. I hope there’s smooth sailing from here on out, but you and I both know, that’s not how it typically goes when it comes to programming haha.

[castulo]

Thank you Kaelyn for taking the time to follow up on this and try to help me, I really appreciate it. You are the best.

I want to make a note however that I don’t think the problem I’m having is bound to the region I’m deploying to, and I’ll tell you why I don’t think this is the case.

Somewhere I read that it was recommended to have two different deployments for your app, one for staging and one for production, so you could test your app using the staging deployment that resembles the production deployment, and if everything works fine, then you can deploy to production.

So I created two different toml files, named fly.staging.toml and fly.production.toml, and deployed both apps. Since the app I’m working on is targeted to be used in Mexico, my production toml uses the gdl primary region so it is closer to the users. But since I’m based in Oregon, US, I chose sea for the primary region of my staging deployment, to allow for faster deployments while in development. And both deployments are failing with the same errors regardless of one being in Seattle, Washington and one being in Guadalajara, Mexico. Furthermore, the deployment that is supposed to be the “production” environment, hasn’t even been used yet, since I haven’t really released the app yet. I just have been deploying changes to it when deployments to the staging environment look good.

So, somehow looks like both apps degraded over time for no reason.

[castulo]

I finally found out what the problem was. Or at least part of the problem.

The thing was that the tmp/ directory within my local rails app had grown to be very big, about 5.5 GB in size. Files within the tmp/ directory are supposed to be ignored when sending the context to build the docker image. However, there seems to be a bug in the .dockerignore file that is autogenerated by fly during fly launch that is causing all the files in the tmp/ directory to be sent as context to the docker image, causing the image to become huge (5.5 GB). Why was this big image failing to be pushed to the fly registry? that, I don’t know, probably some size limit somewhere in fly since the push kept restarting after having pushed about 3.5 GB of the image to the registry.

This is the bug in the .dockerignore file generated by fly:

# Ignore pidfiles, but keep the directory.
/tmp/pids/*
!/tmp/pids/  # <-- bug: This line should be deleted, is basically making the previous line useless, all files inside /tmp/pids are being sent to the context
!/tmp/pids/.keep

# Ignore storage (uploaded files in development and any SQLite databases).
/storage/*
!/storage/.keep
/tmp/storage/*
!/tmp/storage/ # <-- bug: This line should be deleted, is also making the previous line useless, all files inside /tmp/storage are being sent to the context. This directory had 5.5 GB of data in my case.
!/tmp/storage/.keep

After removing those lines from the .dockerignore my app went down to 490 MB, and was finally deployed successfully.

[rubys]

Interesting. I’ll investigate further, but this matches the dockerignore file that Rails will be producing in 7.1:

rails/rails/blob/234eff4cbcbff283df887928142aa5189411178b/railties/lib/rails/generators/rails/app/templates/dockerignore.tt#L24-L27

[rubys]

Reproduced and opened as an issue with rails:

.dockerignore is not excluding pids, storage directories