Deploy timing out, but actually succeeding?

kentcdodds · July 24, 2023, 8:37pm

I have this app: https://epicweb-dev-data-modeling.fly.dev/

Here’s where it’s deployed via GitHub Actions: Actions · epicweb-dev/data-modeling · GitHub

If I check the logs, it seems to deploy without issue. But the logs in GitHub actions shows timeout failures:

deployment-01H60AYDNFXRGCH77M989V0M5K: digest: sha256:0cd2bcd64d697f86fdaf25d79ac4615eaed58f791090365f2b43a0c93126e025 size: 2212
--> Pushing image done

Watch your app at https://fly.io/apps/epicweb-dev-data-modeling/monitoring

image: registry.fly.io/epicweb-dev-data-modeling:deployment-01H60AYDNFXRGCH77M989V0M5K
image size: 3.8 GB
  [1/1] Updating 4d891262b6d158 [app]
Updating existing machines in 'epicweb-dev-data-modeling' with rolling strategy
  [1/1] Waiting for 4d891262b6d158 [app] to have state: started
Error: timeout reached waiting for machine to started failed to wait for VM 4d891262b6d158 in started state: Get "https://api.machines.dev/v1/apps/epicweb-dev-data-modeling/machines/4d891262b6d158/wait?instance_id=01H60BJGMRFZZFWMS4VKJZFB52&state=started&timeout=60": net/http: request canceled
note: you can change this timeout with the --wait-timeout flag

I’ve got other very similar apps that aren’t having this problem. Any ideas?

kentcdodds · July 24, 2023, 9:10pm

Actually, after posting this, I tried to push another update and now it’s not even finishing this step anymore:

==> Pushing image to fly

[147](https://github.com/epicweb-dev/data-modeling/actions/runs/5649557668/job/15304715048#step:5:147)The push refers to repository [registry.fly.io/epicweb-dev-data-modeling]

[148](https://github.com/epicweb-dev/data-modeling/actions/runs/5649557668/job/15304715048#step:5:148)

It’s hitting:

116c046ff166: Retrying in 5 seconds
116c046ff166: Retrying in 4 seconds
116c046ff166: Retrying in 3 seconds
116c046ff166: Retrying in 2 seconds
116c046ff166: Retrying in 1 second

Not sure what’s going on with it now.

I should mention it is a bit bigger of an image than I normally deploy (measured in GBs instead of MBs) so maybe that has something to do with it?

kentcdodds · July 29, 2023, 10:08pm

So this app started deploying fine, but now I’m experiencing the same problem in another (similar) app:

The app name is “epicweb-dev-web-auth”

It’s 3.9GB in size. Any help is appreciated. I have no idea what to do about this.

benbjohnson · July 29, 2023, 10:25pm

It looks like the program is exiting:

swapfile setup complete
🐨  Let's get learning!
Local:            http://localhost:8080
On Your Network:  ****
Press Ctrl+C to stop
Main child exited normally with code: 1

It also looks like there were some heath check issues so that could be related:

Health check on port 8080 has failed. Your app is not responding properly. Services exposed on ports [80, 443] will have intermittent failures until the health check passes.

You can try removing the health checks from fly.toml temporarily and deploying and see if you’re able to access port 8080 when you ssh in.

kentcdodds · July 29, 2023, 10:37pm

Ah, interesting. That’s either new or I somehow missed that bit I’ll see if I can figure out what changed with the other one that may have caused it to start working. Thanks!

musicetre · July 30, 2023, 1:31pm

I am having difficulties when pushing my image. Are there any workaround other than --remote-only?

kentcdodds · August 4, 2023, 6:43am

Question… Does the size of the docker image eat into the volume size? Because my image is 7.9GB and now I’m getting the following log output:

deployment-01H6ZJRZZ3DSG88C74KF0RRRT7: digest: sha256:ff1183b383597e01a63d45426cc7a81df340fb866e594d07a8e3fda84fd6275d size: 2213
--> Pushing image done
image: registry.fly.io/epicweb-dev-web-auth:deployment-01H6ZJRZZ3DSG88C74KF0RRRT7
image size: 7.9 GB

Watch your app at https://fly.io/apps/epicweb-dev-web-auth/monitoring

  [1/2] Updating 9185775c744383 [app]
Updating existing machines in 'epicweb-dev-web-auth' with rolling strategy
  [1/2] Waiting for 9185775c744383 [app] to have state: started
  Machine 9185775c744383 [app] has state: started
  [1/2] Checking that 9185775c744383 [app] is up and running
Smoke checks for 9185775c744383 failed: the app appears to be crashing
Check its logs: here's the last lines below, or run 'fly logs -i 9185775c744383':
  Successfully prepared image registry.fly.io/epicweb-dev-web-auth:deployment-01H6ZJRZZ3DSG88C74KF0RRRT7 (1m21.917155681s)
  [    0.035171] PCI: Fatal: No config space access function found
   INFO Starting init (commit: b572577)...
   INFO Preparing to run: `docker-entrypoint.sh node ./scripts/start.js` as root
   INFO [fly api proxy] listening at /.fly/api
  2023/08/04 06:31:31 listening on [fdaa:0:23df:a7b:ad0:f99f:856e:2]:22 (DNS: [fdaa::3]:53)
  setting up swapfile...
  file:///myapp/node_modules/execa/lib/error.js:60
  		error = new Error(message);
  		        ^
  Error: Command failed with exit code 1: fallocate -l 512M /swapfile
  fallocate: fallocate failed: No space left on device
      at makeError (file:///myapp/node_modules/execa/lib/error.js:60:11)
      at handlePromise (file:///myapp/node_modules/execa/index.js:124:26)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async file:///myapp/scripts/setup-swap.js:7:1 {
    shortMessage: 'Command failed with exit code 1: fallocate -l 512M /swapfile',
    command: 'fallocate -l 512M /swapfile',
    escapedCommand: 'fallocate -l 512M "/swapfile"',
    exitCode: 1,
    signal: undefined,
    signalDescription: undefined,
    stdout: '',
    stderr: 'fallocate: fallocate failed: No space left on device',
    cwd: '/myapp',
    failed: true,
    timedOut: false,
    isCanceled: false,
    killed: false
  }
  Node.js v18.17.0
   INFO Main child exited normally with code: 1
   INFO Starting clean up.
   WARN hallpass exited, pid: 256, status: signal: 15 (SIGTERM)
  2023/08/04 06:31:32 listening on [fdaa:0:23df:a7b:ad0:f99f:856e:2]:22 (DNS: [fdaa::3]:53)
  [    2.271260] reboot: Restarting system
  machine did not have a restart policy, defaulting to restart
  [    0.034286] PCI: Fatal: No config space access function found
   INFO Starting init (commit: b572577)...
   INFO Preparing to run: `docker-entrypoint.sh node ./scripts/start.js` as root
   INFO [fly api proxy] listening at /.fly/api
  2023/08/04 06:31:34 listening on [fdaa:0:23df:a7b:ad0:f99f:856e:2]:22 (DNS: [fdaa::3]:53)
  setting up swapfile...
  file:///myapp/node_modules/execa/lib/error.js:60
  		error = new Error(message);
  		        ^
  Error: Command failed with exit code 1: fallocate -l 512M /swapfile
  fallocate: fallocate failed: No space left on device
      at makeError (file:///myapp/node_modules/execa/lib/error.js:60:11)
      at handlePromise (file:///myapp/node_modules/execa/index.js:124:26)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async file:///myapp/scripts/setup-swap.js:7:1 {
    shortMessage: 'Command failed with exit code 1: fallocate -l 512M /swapfile',
    command: 'fallocate -l 512M /swapfile',
    escapedCommand: 'fallocate -l 512M "/swapfile"',
    exitCode: 1,
    signal: undefined,
    signalDescription: undefined,
    stdout: '',
    stderr: 'fallocate: fallocate failed: No space left on device',
    cwd: '/myapp',
    failed: true,
    timedOut: false,
    isCanceled: false,
    killed: false
  }
  Node.js v18.17.0
   INFO Main child exited normally with code: 1
   INFO Starting clean up.
   WARN hallpass exited, pid: 256, status: signal: 15 (SIGTERM)
  2023/08/04 06:31:35 listening on [fdaa:0:23df:a7b:ad0:f99f:856e:2]:22 (DNS: [fdaa::3]:53)
Error: smoke checks for 9185775c744383 failed: the app appears to be crashing

system · August 11, 2023, 6:44am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Error: timed out waiting for machine to reach started state - Node.js app JavaScript postgres , nodejs	9	722	November 5, 2023
Deployment causes failure on CI workflow but actually succeeding Questions / Help	12	242	January 26, 2024
Cannot deploy app since yesterday	15	752	August 16, 2023
Reporting a failed deployment	2	261	November 17, 2021
Fly deploy failing executing release command Questions / Help	3	496	August 29, 2022

Deploy timing out, but actually succeeding?

Related Topics