cron-manager failed to provision machine error

janfly · April 27, 2024, 11:26am

Hello, I deployed the cron-manager, however the set up job is not being executed. I get this error:

2024-04-27T11:21:01.753 app[6e82992c129ed8] ord [info] api | INFO[0492] Preparing job... app-name=finohra job-id=1483 schedule=update-invoice-status

2024-04-27T11:21:01.844 app[6e82992c129ed8] ord [info] api | ERRO[0492] job processing failed app-name=finohra error="failed to launch machine: failed to launch VM: You must be authenticated to view this." job-id=1483 schedule=update-invoice-status

2024-04-27T11:21:01.844 app[6e82992c129ed8] ord [info] api | ERRO[0492] failed to process job error="failed to provision machine: failed to launch machine: failed to launch VM: You must be authenticated to view this."

Here is what my schedules.json looks like:

[
  {
    "name": "update-invoice-status",
    "app_name": "finohra",
    "schedule": "0 * * * *",
    "region": "fra",
    "command": "bundle exec rails runner 'UpdateInvoiceAndPaymentStatusJob.perform_now'",
    "command_timeout": 60,
    "enabled": true,
    "config": {
      "metadata": {
        "fly_process_group": "cron"
      },
      "auto_destroy": true,
      "disable_machine_autostart": true,
      "guest": {
        "cpu_kind": "shared",
        "cpus": 1,
        "memory_mb": 512
      },
      "image": "ruby:3.3.0-slim",
      "restart": {
        "max_retries": 1,
        "policy": "no"
      }
    }
  }
]

I tried resetting my token:

fly secrets set FLY_API_TOKEN=$(fly auth token),

and logout/login:

flyctl auth logout,
flyctl auth login,

but it did not help.

system · May 4, 2024, 11:27am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

roadmr · May 28, 2024, 3:59pm

Hi there!

When you create an API token with fly auth token, and then log the user out, the token is invalidated. This might explain why your token is not working. I would recommend trying to regenerate the token fly auth token and setting it up in your cron-manager app, see if that gets things working properly. Don’t fly auth logout before trying it!

Daniel

janfly · May 29, 2024, 9:07am

I was able to resolve the “authentication” error. It was necessary to create a deploy token:

fly tokens deploy -a finohra-cron-manager

and then set the secret like so:

fly secrets set DEPLOY_TOKEN="FlyV1 fm2_lJPECAAAAAAAAfS7xBBBQ3...60FeJY=" -a finohra-cron-manager

However the job is failing. Here is the error I have in the logs now:

2024-05-29T08:49:01.623 app[6e82992c129ed8] ord [info] api | INFO[0356] Preparing job... app-name=finohra job-id=3376 schedule=update-invoice-status

2024-05-29T08:49:02.296 app[6e82992c129ed8] ord [info] api | INFO[0357] Running job... app-name=finohra job-id=3376 machine-id=d891395f466928 schedule=update-invoice-status

2024-05-29T08:49:10.340 app[6e82992c129ed8] ord [info] monitor | INFO[0365] Job failed with exit code 10 app-name=finohra execution-time=1.16s job-id=3376 machine-id=d891395f466928 schedule=update-invoice-status

When I run the job directly in my main app, it DOES work:

flyctl ssh console -C "bundle exec rails runner 'UpdateInvoiceAndPaymentStatusJob.perform_now'" -a finohra

Does anybody know what “exit code 10” means? I did not find it in the documentation.

shaun · May 29, 2024, 2:09pm

Is there anything interesting in the Machine logs associated with the job?

E.G. fly logs -i d891395f466928 --app finohra

janfly · June 5, 2024, 8:38am

Hi Shaun,

I get following errors on the “finohra” machine:

2024-06-05T08:27:04.027 app[d8d9240b2e99d8] fra [info] INFO Starting init (commit: d772ddd9)...

2024-06-05T08:27:04.054 app[d8d9240b2e99d8] fra [info] INFO Preparing to run: `bundle exec rails runner UpdateInvoiceAndPaymentStatusJob.perform_now` as root

2024-06-05T08:27:04.061 app[d8d9240b2e99d8] fra [info] INFO [fly api proxy] listening at /.fly/api

2024-06-05T08:27:04.066 runner[d8d9240b2e99d8] fra [info] Machine created and started in 1.518s

2024-06-05T08:27:04.068 app[d8d9240b2e99d8] fra [info] 2024/06/05 08:27:04 INFO SSH listening listen_address=[fdaa:4:c11c:a7b:1c2:4976:1da5:2]:22 dns_server=[fdaa::3]:53

2024-06-05T08:27:04.326 app[d8d9240b2e99d8] fra [info] Could not locate Gemfile or .bundle/ directory

I’m not sure why it cannot locate Gemfile or bundle directory. It looks like “finohra-cron-manager” is trying to run the command on itself, instead of “finohra” machine?

I tried to change the command from:

"bundle exec rails runner 'UpdateInvoiceAndPaymentStatusJob.perform_now'"

To:

"cd /rails && bundle exec rails runner 'UpdateInvoiceAndPaymentStatusJob.perform_now'"

But then I get directory was not found errors:

2024-06-04T13:58:03.377 app[e2867429a904d8] fra [info] INFO Preparing to run: `cd /rails && bundle exec rails runner UpdateInvoiceAndPaymentStatusJob.perform_now` as root

2024-06-04T13:58:03.381 app[e2867429a904d8] fra [info] ERROR Error: failed to spawn command: cd /rails && bundle exec rails runner UpdateInvoiceAndPaymentStatusJob.perform_now: No such file or directory (os error 2)

I whish there was a simple way to run scheduled jobs like on Heroku

shaun · June 5, 2024, 2:14pm

[
  {
    "name": "update-invoice-status",
    "app_name": "finohra",
    "schedule": "0 * * * *",
    "region": "fra",
    "command": "bundle exec rails runner 'UpdateInvoiceAndPaymentStatusJob.perform_now'",
    "command_timeout": 60,
    "enabled": true,
    "config": {
      "metadata": {
        "fly_process_group": "cron"
      },
      "auto_destroy": true,
      "disable_machine_autostart": true,
      "guest": {
        "cpu_kind": "shared",
        "cpus": 1,
        "memory_mb": 512
      },
      "image": "ruby:3.3.0-slim",
      "restart": {
        "max_retries": 1,
        "policy": "no"
      }
    }
  }
]

So one thing that jumps out is that you’re using the ruby:3.3.0-slim image. This image would not contain rails or anything pertaining to your project, so I think the errors your seeing are expected.

I don’t know what your setup looks like, but you could explore using the same image your destination App is using. If your App exposes API endpoints, your cron-job could simply hit those. Otherwise, you may need to create a custom image that’s tailored for whatever you’re doing.

janfly · June 5, 2024, 2:54pm

I’m not sure what image I’m using in my main app (if any). The package.json contains following:

{
  "license": "UNLICENSED",
  "devDependencies": {
    "@babel/core": "^7.16.12",
    "@babel/preset-env": "^7.16.11",
    "@fullhuman/postcss-purgecss": "4.1.3",
    "babel-jest": "^27.4.6",
    "cypress": "^9.4.1",
    "jest": "^27.4.7",
    "postcss": "8.4.5",
    "sass": "1.45.1",
    "vite": "^5.0.0",
    "vite-plugin-full-reload": "0.2.2",
    "vite-plugin-ruby": "^5.0.0"
  },
  "dependencies": {
    "@hotwired/stimulus": "^3.2.1",
    "@hotwired/turbo-rails": "^7.3.0",
    "@popperjs/core": "2.11.0",
    "bootstrap": "5.2.1",
    "chart.js": "^4.3.0",
    "chartkick": "^5.0.1",
    "stimulus-use": "^0.52.0"
  },
  "scripts": {
    "test": "jest",
    "build": "vite build --outDir public"
  },
  "jest": {
    "testEnvironment": "jsdom",
    "roots": [
      "test/javascript"
    ],
    "moduleDirectories": [
      "node_modules",
      "app/frontend/javascripts"
    ],
    "collectCoverage": true,
    "coverageReporters": [
      "text",
      "html"
    ],
    "coverageDirectory": "<rootDir>/coverage/js/"
  },
  "packageManager": "yarn@1.22.19"
}

In the cron-manager the image is being set in the schedules.json, line: "image": "ghcr.io/livebook-dev/livebook:0.11.4". I don’t have such file in my main app or "image:" mentioned anywhere. Rails and Ruby are being set only in the Gemfile.

ChatGPT suggested to use "image": "ruby:3.3.0-slim" in cron-manager. When I change it back to "image": "ghcr.io/livebook-dev/livebook:0.11.4", as per the documentation, I get similar errors:

2024-06-05T14:33:03.153 app[6e825100f5e687] fra [info] INFO Preparing to run: `bundle exec rails runner UpdateInvoiceAndPaymentStatusJob.perform_now` as root

2024-06-05T14:33:03.158 app[6e825100f5e687] fra [info] ERROR Error: failed to spawn command: bundle exec rails runner UpdateInvoiceAndPaymentStatusJob.perform_now: No such file or directory (os error 2)

2024-06-05T14:33:03.158 app[6e825100f5e687] fra [info] does `bundle` exist and is it executable?

I thought the command "bundle exec rails runner 'UpdateInvoiceAndPaymentStatusJob.perform_now'" gets executed in the main app. Do I really have to set up Rails environment in the cron-manager? I’m confused about how this should work.

shaun · June 5, 2024, 3:55pm

You can see what image your app is running via fly image show --app <app-name>.

I thought the command "bundle exec rails runner 'UpdateInvoiceAndPaymentStatusJob.perform_now'" gets executed in the main app.

Each job runs within its own Machine under the App you specify. It does not, however, run the command within one of your existing Machines.

Do I really have to set up Rails environment in the cron-manager?

If you want to investigate what the environment looks like, you can adjust your command to sleep for some duration of time. Once the machine has been created, you can ssh into it and evaluate the state of things.

janfly · June 5, 2024, 10:28pm

Thanks for the effort, but I gave up. I set up an endpoint that triggers the job via easycron.com.

Topic		Replies	Views
Fly Deploy is failing	4	230	January 12, 2024
FLY deploys failing (again) - please take a look	1	203	December 22, 2023
Unable to deploy Rails app Questions / Help rails	4	673	June 14, 2023
Deployment started failing with "failed to release lease for machine" Questions / Help machines	5	107	October 24, 2024
Failed to update machine 328733df615118: failed to update VM: 504: upstream request timeout Questions / Help	4	55	March 7, 2025

cron-manager failed to provision machine error

Related topics